Microsoft Word - simpson.doc Australasian Journal of Educational Technology 2007, 23(2), 187-208 Electronic voting systems for lectures then and now: A comparison of research and practice Vicki Simpson University of Surrey Martin Oliver University of London Research and practice in the use of electronic voting systems has developed over the last five years. Electronic voting systems, also known as personal response systems, audience response systems or classroom communication systems, use handsets to elicit responses from students as part of structured teaching sessions, typically lectures. The use of this information has implications for pedagogy; they are associated with the introduction of interactive, discursive and more segmented approaches to teaching. The pedagogic and organisational implications of adopting such systems are summarised, along with the perceptions that staff and students hold. Comparisons are drawn between practice up to 2002 and between 2002 and 2006; these reveal how both practice and research on this topic has matured, highlighting (for example) the development of models that seek to abstract and share practice. The paper concludes by outlining the ways in which such tools can be used to improve lecturing, and identifies an agenda for future work in this area. Introduction This paper considers the influence of one technology on lecturing: electronic voting systems (EVS), also known as personal response systems, audience response systems or classroom communication systems. It does not focus on developments in the technology itself but on the changing applications and impact of the technology on attitudes and practices. This focus on change is achieved by providing two historically specific reviews of research. In 2002, the authors were commissioned to undertake a literature review of this area to inform internal institutional policy (Simpson & Oliver, 2002) and a separate review by Judson and Sawada (2002) was also published. With the increasing interest in this area, boosted by the growth in use and expertise of these systems, a comparison is drawn regarding the attitudes and understanding of such systems since 2002. 188 Australasian Journal of Educational Technology, 2007, 23(2) Background There has been a growing trend to focus on the potential of technology to engage learners in activities: E-learning has been criticised for being technology led, with a focus on providing materials, but has relatively recently focused more on the learner and enabling students and other users. (Higher Education Fund Council for England, 2005: p. 4) The extent to which universities have already adopted web based instruction… suggests there is a need to explore a number of pedagogical issues. Among these is the question of whether the teaching model in higher education should shift from traditional didactic methods to a more discovery-based methodology, through the application of technology. (Bell, Bush, Nicholson, O’Brien & Tran, 2002: p.29) However, this emphasis seems more prevalent in online practice than face to face. Lectures are still seen as the dominant form of teaching, and are associated with the tendency to emphasise content transmission over student engagement (e.g. Laurillard, 2002; Bligh, 1998). Arguably, this is because actual practice (as opposed to perceived practice) is under-researched. In 1999, Jones noted, “much attention has been on the use of computer technology for delivering education on the desktop, but not much attention is paid to its use in the lecture”. However, this situation has been slowly changing with the growing interest in using mobile technologies. With the focus on active e-learning, such systems are seen to offer an opportunity to improve the often passive lecture format. The engagement and interaction with students that these systems encourage can prompt lecturers to reflect on qualities believed to characterise good teaching (Simpson & Oliver, 2002). In order to appreciate the pedagogic developments in this area, it is necessary to understand the practical process of using these systems. A typical pattern of use is as follows. During the lecture, the lecturer poses a question. Each student has a handset that allows him or her to select the preferred option for the answer. The handsets transmit this information to a receiver, which in turn transmits it to the voting software on a computer in the theatre. After the allotted time, the software produces a histogram or bar chart of the results, which is displayed to the students using a data projector connected to the computer. The lecturer then chooses a course of action to respond to the results. The software also allows the data to be recorded so that results can be analysed later. Most handsets allow multiple choice responses (and sometimes multiple selections), with up to ten answers available. Simpson and Oliver 189 As well as transmitting the preferred option, some handsets have buttons where the student can record low, medium or high confidence in the accuracy of their answer. This is helpful in analysing whether correct answers were chosen through luck or knowledge (Gardner-Medwin, 1995). The handsets can be used anonymously. However, handsets can be mapped to a student’s name. This allows the lecturer to see an individual’s answer, either within the lecture or when reviewing responses at a later stage. The handsets transmit to the receiver using wireless technologies, depending on the particular system used. Radio is becoming more widely used than infrared since radio tends to be faster in operation, operates over a greater range and is more accurate in receiving signals. Free text entry allows the extension of feedback from the group and is becoming more prevalent: We can at last address a fundamental problem of discussion groups (such as research seminars) where many people want to ask a question: which is the best question to take for the group as a whole? Using only voice, we cannot know what the set of candidate questions is without having them asked. With textual group responses, everyone's questions could appear in front of the speaker and/or facilitator, and could then be grouped, sequenced, and sorted by priority. (Draper, Cargill & Cutts, 2002). Having established what EVS are, the remainder of this paper will review the available evidence about how they have been used. Methodology The purpose of this paper is not simply to draw together practice to date on the use of EVS. Research and practice have developed over the last few years, to the point where simple aggregation might hide developments and new directions in the work. In order to draw these points out more clearly, a two stage systematic review was undertaken. Each part of this process is described below. Review 1 The first review took place in 2002. Voting systems are used globally, and thus it was necessary to identify and review international literature. However, the terminology used to describe such systems varied considerably, indicating the newness of this area and complicating the identification of relevant research. The search involved a range of indexes and search engines, including the Web of Science (Arts & Humanities 190 Australasian Journal of Educational Technology, 2007, 23(2) Citation Index and Social Sciences Citation Index), BIDS (ERIC, British Education Index and International Bibliography of Social Sciences), Zetoc (British Library resources), Education-line, EdResearch Online and Google. These were iterative, with databases being re-searched as new descriptors were identified. The final search terms used to identify voting systems from various perspectives were: • Technological (e.g. ‘key-pad’, ‘hand-set’) • Pedagogical (e.g. ‘interactive’) • Physical action (e.g. ‘group response’, ‘audience response’, ‘personal response’, ‘electronic voting’) • Brand name (e.g. ‘Varitronix’) These searches identified around thirty potentially relevant papers. The abstracts were inspected to select those relevant to teaching and learning, focusing on lectures. Sixteen were classified as being of direct relevance and formed the basis of the first review. Review 2 The follow on review took place in 2006. The search indicated that a consistent terminology has still not emerged. However, there was greater breadth of citation, including reference to papers using different terminology. This led to the identification of two pre-2002 papers not found during the previous review, which suggests a bringing together of previously separate strands of work. In addition, seventeen new papers and four newsletter articles were identified and used in the follow on review. This lack of coherence is echoed in the motivation behind the publication of the book Audience Response Systems in Higher Education (Banks, 2006): “I found it frustrating that although there was a considerable body of literature relating to these systems, it was scattered throughout a wide range of journals, books and collections of conference papers”. The book includes accounts by some practitioners who have not previously published, and thus draws in additional experiences and reflections not documented elsewhere. Themes For review 1, the content of each paper was coded to identify the different concerns and aspects of using voting systems. This identified six themes: context in which voting systems are used; reasons for use; pedagogical practice; student perceptions; impact on staff and organisational issues. For review 2, the same themes were used in order to look at the development and changes in practice and research since then. Simpson and Oliver 191 Comparison of research and practice between the reviews Contexts In review 1, electronic voting systems were used in a variety of contexts. Use is more prevalent in the science and engineering disciplines (Burnstein & Lederman, 2001; Cue, 1998; Poulis, Massen, Roberts & Gilbert, 1998; Van Dijk, Van den Berg & Van Keulen, 2001), but also in other subject areas such as economics (Elliott, 2002); management (Cue, 1998) and psychology, philosophy and medicine (Draper, 2002). The range of disciplines continues to grow and in review 2, voting systems were also being used in subjects such as biology and statistics (Draper & Brown, 2004) and law (Burton, 2004). In review 1, voting systems were aimed primarily at managing interaction with large groups, with class sizes ranging from 50 to 300 (Burnstein & Lederman, 2001; Poulis et al, 1998; Van Dijk et al., 2001). There were some exceptions: Draper (2002) mentions use of voting systems with class sizes of 20 and for formative assessment ‘sessions’ (rather than lectures) and McCabe, Heal and White (2001a) have used voting systems for “revision classes, small group tutorials, peer assessment sessions”. In review 2, while there are mentions of use with smaller groups (Draper & Brown, 2004), most of the focus continues to be on large groups with class sizes ranging from 100 to nearly 400 (Burton, 2004; Nicol & Boyle, 2003; Kennedy & Cutts, 2005; Sharma, Khachan, Chan & O’Byrne, 2005). It could be argued that interaction is most difficult to encourage in larger groups and thus voting systems are being used where it is perceived there will be the greatest need and benefit. There is more discussion about how often a voting system is used within a course (Kennedy & Cutts, 2005) and the importance of sequencing of activities rather than merely whether to use the system or not (Draper & Brown, 2004; Sharma et al., 2005). This also implies a maturing of the application of these systems beyond the initial idea of just ‘solving big group problems’, and a development of the underlying pedagogies suitable for different contexts. Reasons for use In review one, the following uses of voting systems in lectures were identified: • Preliminary checking of existing understanding or preparation (e.g. Herr, 1994; Burnstein & Lederman, 2001) – a diagnostic, informing teaching. 192 Australasian Journal of Educational Technology, 2007, 23(2) • Verifying comprehension (Elliott, 2002) – confirmatory; can people apply the concept? • Assessment and revision (Draper et al., 2002; Burnstein & Lederman, 2001). • Creating a community (Draper et al., 2002) – e.g. by asking questions about students’ backgrounds as scene setting for a course. • Initiating discussion (e.g. Draper et al., 2002; Poulis et al, 1998; Burnstein & Lederman, 2001). • Peer assessments (Draper et al., 2002). • Focus and direction – to guide teaching priorities and time allocation. • Experiments using human responses (Draper et al., 2002) – suitable primarily within a psychology context. • Administrative uses (Poulis et al., 1998), e.g. student feedback on the lecture or course. • Checking attendance (in named mode) – (Draper et al., 2002). However, these uses could be viewed as a list of specific and seemingly unrelated tactics. Practice in Review 2 is discussed more strategically, identifying the reasoning behind use: Increasing the lecturer’s knowledge of students’ understanding; Increasing the students’ knowledge of their own understanding; Increasing the students’ knowledge of lecturer’s expectations; Increasing the students’ understanding of difficult material (Purchase, Mitchell & Ounis, 2004) Current practice is also elaborated upon in review 2. Whereas earlier case studies focused on solving small educational tasks, current research focuses on a wider educational dilemma or aspect and appears to take a more holistic view of the role of voting systems within that context. For example, ‘initiating discussion’, as identified in the first review, is now considered in more depth. Nicol and Boyle (2003) explore the merits of different discussion sequences, drawing on earlier teaching approaches, and Cutts, Carbone and van Haaster (2004) explore the relationship between revealing the voting responses and discussion. There are also more detailed reports about what works, such as, “teachers reported that if the pattern of correct responses was between 40% and 60% this would normally provide a good trigger for discussion” (Boyle & Nicol, 2003). This discussion of specific aspects, rather than just use versus non-use, indicates maturing practice. In both reviews, many papers propose that anonymity encourages student contribution. In review 1, the reasons given tend to focus on privacy (Draper et al., 2002), reduction of peer pressure and fear of embarrassment (McCabe et al., 2001a). However in review 2, anonymity is considered in a more nuanced manner. There is more depth about the positive contribution Simpson and Oliver 193 that anonymity has to learning, which has implications for understanding when it will be of most benefit: Anonymity seems to function to induce people to pick a definite answer even when they are quite uncertain; and this in turn seems useful in getting people both to think in order to produce and answer, and then to take this (if they get it wrong) as a reason for working on the point later. (Draper & Brown, 2004: p. 89) Banks (2003) suggests that anonymity helps students from different cultures where expressing criticism of others’ views is problematic. An additional study (Freeman, Blayney & Ginns, 2006) focuses explicitly on a range of response methods and the role of anonymity. As well as concluding that anonymity has a positive affect on students’ willingness to participate, “an unexpected finding was that students’ perceptions of their level of interaction in lectures, their understanding and their ability to gauge their own understanding was significantly higher” when a voting system was used. As practice has developed, there is greater discernment about the unique contribution of the technology to the classroom environment. Nicol and Boyle (2003) identify aspects of the quality of the feedback: it is immediate and public; more efficient dealing for large numbers than a show of hands; and shows how personal response relates to overall responses. Responses can also be accurately recalled after the lecture has ended (Elliott, 2003). Pedagogy Whilst voting systems can support teaching and learning within lectures, any benefits will depend on how effectively they are used on each occasion. In order to judge whether the system does indeed enhance the lecture format, it is first necessary to identify the assumptions made about what counts as ‘good’ learning. From review one, three key principles were identified: • Content transmission is not the most effective way of learning. Jones (1999), for example, cites Dearn who proposed that “universities require changing from the view that teaching consists of organising and transmitting content, and that learning is the accurate recall of factual information to one of promoting active learning.” • Students’ active engagement with ideas and applications supports learning (e.g. Burnstein & Lederman, 2001; Cue, 1998; McCabe et al., 2001a; Draper et al., 2002). This view is most clearly advocated by Laurillard (2002), whose conversational framework is drawn upon as a rationale by many researchers in this area, but is also visible in the majority of 194 Australasian Journal of Educational Technology, 2007, 23(2) studies reviewed (e.g. Cooper et al., 1997). The principle underlying this framework is that learning results from the process of ongoing and adaptive dialogue between teacher and learner, supplementing the application of ideas or practical skills. Importantly, researchers identified the way the system was used to encourage all students, not just one or two, to engage. However, anonymity was important to this; people felt encouraged because they could not be picked out if their answer was incorrect (Draper & Brown, 2002b; Jones, 1999). • Quality feedback should be provided to students. Cue (1998) argues that “timely feedback and reinforcement are vital to the synthesis and integration processes” of learning (cf. Jones, 1999). A key feature of voting system use was to allow students to compare their performance with their peers – for example, if they had made a common error, or if everyone had got it right (Draper, 2002). These principles are summarised by this model: The biggest learning gains […] are likely to come from the much better and quicker feedback from learners to teachers, allowing better atunement of the delivery; and from the method of teaching by questions, i.e. of discussions in class (whether in small groups, plenaries, or a combination) initiated by well designed questions and by getting each individual to start by committing to an initial position. (Draper et al., 2002). Although lectures have been seen as the teaching activity where there is the greatest tendency to emphasise content transmission, it is important to recognise that there is a tendency to over simplify this issue. There is a distinction between student activity and student cognitive experiences: It is possible that interactive teaching will not automatically result in active cognitive experiences. A student may, for example, choose not to think deeply about the questions the lecturer asks. Similarly, traditional lecturing behaviour will not automatically result in passive students who are accepting the information without thinking critically about it. [… However,] even though it is possible for students to be mentally active during lectures while listening to a teacher’s exposition, the chances are that more students will be mentally engaged when involved in learning activities like reading, writing, discussion and problem-solving. (Van Dijk et al., 2001: p. 25) Thus although interactive teaching does not guarantee an active learner, it is more likely to encourage active learning and positively influence student motivation. Working such activities into teaching will also help to create an atmosphere in which true engagement is encouraged and supported (Martin, 1999). Thus some benefits associated with voting systems are likely to result from the changes in teaching practice that accompany their introduction. Simpson and Oliver 195 Use of voting systems to stimulate discussion starts to emerge in review one, with Dufresne, Gerace, Leonard, Mestre and Wenk (1996) highlighting the use of question responses as “the springboard for a class-wide discussion”. However, the relationship of voting systems and discussion is more prevalent in review 2, with both Dufresne et al.’s model and Mazur’s peer instruction method (1997) often being cited (Nicol & Boyle, 2003; Draper & Brown, 2004; Cutts et al. 2004a; Beatty, Gerace, Leonard & Dufresne, 2006). Papers from review two continue the theme of effective practice by observing that technology works best in response to specific weaknesses or needs in current delivery (Wit, 2003) and that voting systems in themselves cannot solve problems: It is important not to lapse into thinking of it as a magic bullet but as more similar to central heating in contrast to individual fires. It is not essential to achieving learning or interactivity, which for millennia have been frequently achieved by other means; but nevertheless it makes the desired effect […] easier to achieve more often, in more contexts, and with much less effort and attention. (Draper & Brown, 2004: p. 93) With this broader view also comes greater discussion of particular pedagogical approaches that justify specific uses of the voting system (e.g. peer instruction for Draper & Brown, 2004). There is a clearer sense of historical context, as opposed to a focus on novelty, such as references to underlying pedagogies (e.g. Bligh, 1972, cited in Nicol & Boyle, 2003) and to related pre-EVS pedagogies such as flashcards, show of hands or voting on one student’s response (Purchase et al., 2004; Elliott, 2003; Stuart & Brown, 2003). Use is also justified in terms of previous psychological work on attention spans (e.g. Elliott, 2003). This demonstrates a desire to base or rationalise use of the voting system on understood pedagogical values, whereas earlier use tended to be more speculative. Several later papers also propose models that distil practice and experience in order to address issues. One paper includes a ‘pedagogical script’, which takes the lecturer through the sequence of events, including refinements and annotations made in light of extended use (Cutts et al., 2004a). Reay, Bao, Li, Warnakululasooriya and Baugh (2005) describe a model of conceptual development focusing on a three-question sequence. Another paper uses Laurillard’s conversational model to propose a rationale for use of voting systems (Cutts & Kennedy, 2005), and Sharma et al. (2005) outline a schema for the learning cycle they used to guide their voting system use. Draper and Brown (2004) also propose a sequential model of enabling factors which gives a pedagogical rationale of “improved learning, because it is adapted to learners’ current need”. 196 Australasian Journal of Educational Technology, 2007, 23(2) Discussion is now rationalised in terms of social constructivism; models of discussion are explored and the merits of each evaluated according to context, most notably by Nicol and Boyle (2003). Discussion after a response means that votes are not swayed and that discussion builds on individual thinking; moreover, students seemed more willing to discuss their response, once committed (Crouch & Mazur, 2001). Initiating class- wide discussion by selecting a person or group and requiring an answer was disliked, because it prevented anonymity (which could be threatening), could be less focused, took too much time, and hearing other students’ explanation via a microphone could be confusing for others (Dufresne et al., 1996). However, it could be of value where there are likely to be two points of view rather than total conformity or lots of different positions (Nicol & Boyle, 2003). More subtle educational issues are also identified. For example, Wit (2003) proposes including an ‘I don’t know choice’ as one of the response options (rather than forcing people to commit to an answer), which is pedagogically useful in helping eradicate successful guessing when diagnosing. Another consideration was fit with the wider pedagogic context. Some teachers give credit for right answers during the course which undercuts advocating the use of systems for voting formative purposes, especially where students might try out understanding (e.g. Lopez-Herrejon & Schulman, 2004). It is also observed that teachers themselves have had their own assumptions challenged when a supposedly easy question or concept was not understood by large numbers of the class, revealing a different conceptual model of the topic (Draper & Brown, 2004, Stuart & Brown, 2003). To summarise, discussions of pedagogy have become notably more sophisticated. This is visible both in terms of the level of detail considered, but also in attempts to create representations of practice (such as models) that abstract principles in a form that can be shared and used to improve other peoples’ practices. Student reactions and perceptions Many of the papers in review 1 included informal comment that students were broadly positive (e.g. Cue, 1998; Poulis et al., 1998). The few papers that included more formal analysis said that students were surprisingly tolerant of the system (Burnstein & Lederman, 2001), were generally but not universally enthusiastic (Draper et al., 2002a) and did not seem tempted to steal or ‘fiddle’ with the handsets (Elliott, 2002). In Van Dijk et al.’s study (2001), the lecturers’ attempts to engage the students using techniques such as humour proved to be just as effective at motivating students as using new technology or peer interaction. Similarly, the Simpson and Oliver 197 comparison by Halloran (1995) showed no significant difference between lectures with and without keypads, although observation suggested that students’ learning improved as the novelty (and thus intrusiveness) of the new technology wore off. The perceived benefits were closely aligned to the uses and advantages identified in the previous sections, largely concerning engagement and stimulation, feedback (allowing the lecturer to adapt their lesson plan), anonymity and the ability to compare oneself with peers. They also echo the findings of Halloran (1995), who found that students valued the way that use of the system made classes more interesting and better organised, and allowed them to focus on areas of weakness without exposing these to the group. Perceived disadvantages were also identified. These focus on the intrusiveness of the technology, doubts about how seriously other students take the exercise, and concern that technology is used for its own sake (Draper, 2002). In review 2, the benefits identified appear to have remained stable. For example, Williams (2003) reports that students felt they experienced deeper learning and the system facilitated critical thinking, whereas disadvantages have changed (Draper & Brown, 2004). This may suggest that, whereas previously studies were about celebrating an experimental success and informing basic practice, later studies have become more critical as they seek to improve established practice. It also to be expected that as these systems are used in more contexts, different problems will be identified. There is now evidence that students are not universally supportive of such systems, in spite of the many positive class evaluations reported in the literature. For example, d’Inverno (2003: p.19) reported a polarised response to the system in the class evaluation – those who were in favour were very keen, but there was a subset who felt that the teacher should “stop messing around with technology and get back to good basic teaching”, suggesting that there may be issues with the introduction of such systems. However, Draper and Brown (2004) suggest that “on first introduction, learners are ready to be sceptical but in subsequent years take it for granted and rate it higher”. D’Inverno conjectures that some students have come to “expect to switch off in lectures” and thus reject the more interactive format but this is contradicted by others (Elliott, 2003; Boyle & Nicol, 2003, Barnett, 2006) where students were positive about interactivity. In analysing the responses after the lecture, Cutts and Kennedy (2005) noted that many students did not answer all the questions. Experience of this is not documented elsewhere, although Banks conjectures that if such systems are “over-used or inappropriately used they could quickly lose impact and lead to uncritical ‘button pushing’ and student detachment” 198 Australasian Journal of Educational Technology, 2007, 23(2) (2003, p.45). Either this issue has not been explored or this is an infrequent occurrence. Cutts and Kennedy’s analysis also “showed that students who used the EVS more often did better in assessments, although level of correctness of responses did not correlate.” This might support the theory that learning through misunderstanding is educationally valuable and that engagement, whether correct or incorrect, is the key – although it could also be that students who would perform well anyway are also conscientious about using the EVS. Impact on staff As noted repeatedly in the literature, the potential of these systems depends on the skills and approach taken by teachers. It was clear in review 1 that the lecture format will need at the very least some minor adaptation (Elliott, 2002; Poulis et al., 1998) to cover less material and ‘make room’ for questions. Up front design before the session is required to develop good questions and consider their place in the pedagogy of the lecture (Burnstein & Lederman, 2001). Creating questions that are pitched at the appropriate level is another challenge (Draper & Brown, 2002). During the lecture, staff need to be able to react to the instant feedback and alter the course of the lecture in mid-flow to respond to identified students’ needs; consequently, instant feedback is presented as a positive (e.g. Poulis et al., 1998; Burnstein & Lederman., 2001; Draper, 2002) although it could place high demands on teaching skills. Now that papers in review 2 propose different teaching approaches, the impact on staff may be even greater. For example, Nicol and Boyle (2003), drawing on Crouch and Mazur’s and Dufresne et al.’s work, propose that a more radical approach is taken which involves making extensive changes to teaching practice, and moving away from the familiar traditional lecture format, towards a more discursive, segmented, tutorial style approach. Feedback may also lead to extensive changes across the curriculum (e.g. adapting future lessons) rather than just within sessions (Elliott, 2003). Earlier literature highlighted the need for development of staff skills for using a voting system, but now the emphasis seems to have gone a stage further by looking at development of teaching per se. The emphasis is on the design of contingent teaching so that voting systems can be used most effectively (Draper & Brown, 2004). Voting systems provide fast feedback; if staff are to respond, they cannot be wedded to scripted and planned lectures. For example, if adopting Dufresne et al.’s model in conjunction with a voting system, “these situations clearly point to the need for the instructor to manage the class-wide discussion and bring it to closure when it seems that all views have been presented and further discussion is not fruitful” (Dufresne et al., 1996, p.20). However, there can be benefits: for Simpson and Oliver 199 example Dufresne notes that the lecturer can use the students’ discussion time to pause and think about a response. This ‘thinking time’ is not available in a more didactic teaching format. By contrast, Cutts and Kennedy (2005) and Kennedy and Cutts (2005) describe a more pragmatic approach that involves using the same lecture format with some small changes, and argue against rethinking the whole teaching format. Stuart and Brown (2004) argue that whilst wholesale change is possible it is not always successful, although as Draper and Brown (2004) point out, feedback in spite of massification is valued by all and “this allows even relatively uninspired handset use to be valued by students”. This confirms Judson and Sawada’s earlier finding that “students will favour the use of electronic response systems no matter what the underlying pedagogy” (2002: p177). As has been seen with the adoption of other e-learning technologies, taking a simple approach can be a valid first step in developing new pedagogic practices. When considering the impact on staff, it is also important to recognise factors that stop people developing, such as a tendency to teach as we were taught, pressure to focus on research, fear of losing control over content being covered, and previous experience of failed teaching innovations through inadequate support (Dufresne et al., 1996). Some level of technical confidence and skills are needed to develop questions and operate the system (for example, Burton provides some practical tips in a section titled “Words of Advice and Caution”, 2004), although there is also the issue of having the skills and being able to use these while teaching. Even in the review 2 literature, there are still constant references to technical problems which can limit success if they intrude (Draper & Brown, 2004; Cutts et al., 2004a; Stuart & Brown, 2004; Reay et al., 2005). Interestingly, teaching assistants were sometimes used to provide support in early accounts of use (Draper & Brown, 2002); this approach is less evident in recent papers. Organisational impact A new theme has become evident in later papers: organisational impact. Until recently, voting system adoption has not been on a scale large enough to merit considering this; now there are sufficient numbers of lecturers using EVS in one institution (e.g. Glasgow, UK) to look beyond one lecturer’s practice and towards institutional support needs (Draper & Brown, 2004). Although it is useful to see results and conclusions synthesised from many instances of use, rather than just one case study (from which it is unclear what could be generalised), it must be accepted that this paper can only come up with local theories, suitable for that institution’s context (Draper 200 Australasian Journal of Educational Technology, 2007, 23(2) & Brown, 2004). Draper also points out that it can only be a limited report on organisational change because it was working with enthusiasts who wanted this initiative to work. However, this focus is likely to be increasingly common, echoing developments in the use of virtual learning environments and e-learning implementation. Use of voting systems no longer just concerns changing the curriculum plan, but changes to the way space is arranged to reflect this particular pedagogic approach, as well as the time required (away from the traditional one hour lecture) and the relationship of teacher to the curriculum (Draper & Brown, 2004). Whereas, in the first review, voting systems were seen as a way of enhancing an existing resource (the lecture theatre), now its use is driving - or at least asking questions - about the design of that resource (Boyle & Nicol, 2003). The NATALIE project, for example, is exploring how physical space needs to be adapted to encourage both group working and use of technology, so that students could sit in groups of four and use a voting system collaboratively (Nicol & Boyle, 2003). In review 1, some papers mention the practicalities of handset distribution and management. Different approaches are suggested: issuing in class (which takes time and requires assistance) (Draper, 2002), issuing by system (e.g. swap for ID card) (Burnstein & Lederman, 2001), and loan from the library (Cue, 1998). In review 2, new ideas such as student purchase (Barnett, 2006) are suggested but no single satisfactory solution is offered. In some studies, handsets were distributed for longer periods but it is reported that a substantial minority (for example, 25-35%) of students forget to bring them into class (Draper et al., 2004, Reay et al. 2005). As use of voting systems increases within institutions, it will be interesting to see how this practical issue might best be resolved. Discussion In both early and recent papers, the motivation to use a voting system effectively arises from the view that active learning and engagement of students is worthwhile. However, more recent papers have interpreted this in a more sophisticated and holistic way, looking beyond the assumption that using handsets equates to meaningful interaction and also more broadly than just using the system for one teaching goal, such as ‘checking understanding’. More specific and contextualised pedagogical approaches and theories are being used in developing practice; an indication of this increased expertise and the greater reference to pedagogy is the emergence of models that distil this practice and experience. Simpson and Oliver 201 References to the technology, making better use of space, rethinking teaching spaces and improving teaching suggest that the whole teaching environment needs to be taken into consideration when introducing such an innovation, and to look at one aspect in isolation is to miss opportunities. “A consideration of the use of voting systems in lectures alone appears to be insufficient to improve learning, and so this section proposes an integrated learning environment that makes use of EVS response data to bind lectures, self study time and tutor led sessions” (Cutts & Kennedy, 2005). The impact goes beyond merely developing the skills required to use a voting system and affects teaching practice and management in general. Although technology driven adoption appears less common now, there are variances within educationally driven practices. Boyle and Nicol start with an educational dilemma and seek solutions; in doing so, technology is selected as part of the solution and used to support and facilitate that development. These gains were the result of the application in class of teaching and learning principles centred on active engagement and dialogue which were supported by… technology. Many of the advantages of this style of learning could therefore be retained (but perhaps less easily) even without [technology]. (Boyle & Nicol, 2003). A different view is offered by Draper and Brown (2004). They are also driven by educational goals, but appear to see technology as a definite solution rather than a possible option: [We] should look at the teaching practices around us, identify the weakest points, and try to discover how ICT could address these. (Draper & Brown, 2004: p.82) Thus we need to ask if the framing question should be ‘how can this teaching problem be addressed?’ or ‘how can ICT be used to address this teaching problem?’ The field is definitely maturing but is still evolving around enthusiasts, even if it is becoming a wider group of enthusiasts (Draper & Brown, 2004). Practice may be becoming more sophisticated but there are still new case studies as people discover these systems for the first time in their own context (e.g. Stuart & Brown, 2004). Extending good practice in the use of EVS could be problematic. As Van Dijk et al. (2001) observe, A lot of lecturers are reluctant to accept claims on the merits of activating instruction, which can be found in educational theory. Activating students requires time, which lecturers would normally devote to lecturing. They often voice the concern that they will not get enough material across when 202 Australasian Journal of Educational Technology, 2007, 23(2) giving interactive lectures and that this consequently will negatively affect the student learning. (Van Dijk et al., 2001: p.16) It is worth noting that many of the uses identified can be (and sometimes already are) currently undertaken in lectures using non-technological methods. Thus these systems are not necessarily introducing new practices – instead, they enable existing practice to be refined and enhanced. However, their presence in lecture theatres may serve to raise awareness about a range of pedagogic approaches. Nonetheless, if a lecturer holds the conception that lectures are best used to present content, they will be resistant to changing their teaching style to incorporate interactivity. Judson observes that “It is imperative for the instructor to understand the tenets of constructivism and to have struggled with his/her own epistemological beliefs about instruction as he/she experiments with such a method of teaching” (Judson & Sawada, 2002: p179). If they do, they are unlikely to be convinced by the potential of a voting system, or else may adopt the system but use it ineffectively rather than supporting interactive teaching (Simpson, 2002). From the student perspective, the majority appear in favour of these systems and perceive that their use is of benefit to their learning. Most of these evaluations are based on self report of perceptions and qualitative methodologies rather than the impact on learning itself. This is understandable, given that “education is a complex process, and the introduction of technology only serves to exacerbate this situation… it can be difficult to know where to look for impact or to recognise it when it happens” (Oliver & Harvey, 2002). However efforts to explore these complex issues are developing: Kennedy and Cutts (2005) propose a methodology to investigate the connection between an individual’s use of a voting system and their learning outcomes. This area needs further development if the community is to discover more about the benefit and impact of voting systems on student learning. Overall, a more critical approach to evaluation of the contribution and value of voting systems is emerging. One study (Draper et al., 2004) asks about the net benefit rather than merely identifying any positive responses and using those as a rationale to continue. This shift in evaluation indicates that these systems are no longer being thought of as an innovative gadget for the few, but as a technology that has significant potential. If implemented across the institution, it has to be considered more rigorously because of the implied costs and risks. It is also interesting to note the significant rise of the “Who wants to be a millionaire?” references as a way of uniting papers in the otherwise fragmented field; this pop culture reference lending a degree of coherence Simpson and Oliver 203 absent theoretically (e.g. McCabe, Heal & White, 2001; Elliott, 2003; Wit, 2003; Stuart & Brown, 2004). This has been picked up by the popular press (e.g. Williams, 2002, in the Guardian article, “As seen on TV”). This has led, in some cases, to the adoption of the game show terminology to describe pedagogic practice (e.g. “a 50:50 technique”, and options being “eliminated”; Wit, 2003). Similarly, there has been a shift to commercialising and mainstreaming knowledge about EVS use via books, for example Clickers in the Classroom (Duncan, 2005) and Audience Response Systems in Higher Education (Banks, 2006) rather than publishing solely in journals. Methodologically, this review has illustrated the partiality of evidence based processes of systematic review (cf. Oliver & Conole, 2003). The second review revealed papers from before 2002 that had previously been undiscovered because the authors and their audience used different terms to describe very similar practices and resources. The existence of multiple communities researching the same topic, each with its own terms and theories, emphasises that even systematic reviews are situated, historical acts – especially in e-learning where there is no single database of research that serves to define and bound the field of study. Conclusions This paper has offered an overview of the themes and issues in the literature of electronic voting systems focusing on the developments and changes in this area. From this review, a number of conclusions can be drawn. Firstly, voting systems are best understood as a tool rather than a teaching approach. Throughout the literature reviewed, there appears to be consensus that they do not ‘cause’ good learning; however when used as part of a wider effort to support active engagement with learning there is evidence that they can support increased motivation and attainment, at least in part as a result of their ability to provide rapid feedback on the learning process. It is interesting to note that although there have been advances in the tools themselves (such as the shift from infrared to radio communication; some handsets offering text entry), exploration of this area is very much pedagogically focused rather than led by technical innovation and development. Secondly, it is generally agreed that these systems represent an opportunity to improve lecturing. The emphasis on engagement and interaction can prompt staff to rethink their conception of teaching, revising widely held opinions that the primary purpose of lectures should be broad coverage of a topic. Their presence can facilitate a variety of 204 Australasian Journal of Educational Technology, 2007, 23(2) teaching practices that promote interaction and engagement, although it should be noted that many of these can be replicated (albeit with greater effort) without such a system. Thirdly, we are seeing the emergence of a specific and defined field of educational interest. Although there are papers that still report on first use and do not necessarily extend debate around this field, the later literature tends to move beyond investigating merely if voting systems have a role in teaching, thoughtfully exploring how they can be best used. This has been supported by a greater analysis of student perceptions (not just looking at satisfaction) and attempts to investigate the impact on learning and achievement. Models of use are now being proposed and refined, indicating that the innovation is now becoming embedded as part of understood practice. The wider impact (beyond the earlier focus of making the lecture format more interactive) is also explored, as illustrated by the debates concerning contingent teaching and the re-design of physical teaching spaces. This maturity is typified by the greater shared understanding about these systems and use of terminology which was less evident in the earlier literature. Fourthly, there are still some issues that are either not included or only occasionally raised. To use the tools effectively requires an understanding and belief in active learning, and hence there are staff development implications not just for the use and application of the technology but for the underlying and supporting concepts. There are also institutional and organisational issues, such as the practical management and administration of these systems which have not been wholly resolved. This field is developing and maturing but it may remain in the preserve of the enthusiast and not part of the accepted array of institutional teaching tools - or used widely but possibly inappropriately - unless the practical issues and context in which voting systems are used are supported. It is interesting to compare the adoption of virtual learning environments, which moved from being the tool of enthusiasts to an institutional ‘must have’: perhaps it can be projected that voting systems will be embraced by institutions in a similar fashion and that evaluation and report on these aspects may become part of the literature in this field in the future. References Banks, D. (2003). Using keypad-based group process support systems to facilitate student reflection. In G. Crisp, D. Thiele, I. Scholten, S. Barker and J. Barron (Eds), Interact, Integrate, Impact: Proceedings 20th ASCILITE Conference. (pp. 37- 46). http://www.ascilite.org.au/conferences/adelaide03/docs/pdf/37.pdf Simpson and Oliver 205 Banks, D. (Ed) (2006). Audience response systems in higher education: Applications and cases. London: Information Science Publishing Barnett, J. (2006). Implementation of personal response units in very large lecture classes: Student perceptions. Australasian Journal of Educational Technology, 22(4), 474-494. http://www.ascilite.org.au/ajet/ajet22/barnett.html Beatty, I., Gerace, W., Leonard, W. & Dufresne, R. (2006). Designing effective questions for classroom response system teaching. American Journal of Physics, 74(1), 31-39. Bell, M., Bush, D., Nicholson, P., O’Brien, D. & Tran, T. (2002) A survey of online education and services in Australia. Commonwealth Department of Education, Science and Training. [viewed 3 Aug 2006, verified 8 Apr 2007] http://www.dest.gov.au/sectors/higher_education/publications_resources/ profiles/online_education_services_in_ustralia.htm Bligh, D. (1998). What’s the use of lectures? Exeter: Intellect. Boyle, J. & Nicol, D. (2003). Using classroom communication systems to support interaction and discussion in large class settings. Association for Learning Technology Journal, 11(3), 43-57. Burnstein, R. & Lederman, L. (2001). Using wireless keypads in lecture classes. The Physics Teacher, 39, 8-11. http://www.replysystems.com/pdfs/benefits/24.pdf [viewed 3 Aug 2006] Burton, K. (2004). Interactive Powerpoints: Is there any point in giving power to students? Murdoch University Electronic Journal of Law, 11(4). [viewed 3 Aug 2006] http://www.murdoch.edu.au/elaw/ Cooper, C., Gerth, S., Lewis, K., Maxwell, D. & Tymchysyn, P. (1997). Computerized classroom presentation with keypad questions as compared to traditional classroom lecture. In Gerdin, U., Tällberg, M. & Wainwright, P. (Eds), The Impact of Nursing Knowledge on Health Care Informatics: Proceedings of the 6th International Congress on Nursing Informatics, 559. Stockholm: Sweden. Crouch, C. & Mazur, E. (2001). Peer instruction: Ten years of experience and results. American Journal of Physics, 69, 970-977. Cue, N. (1998). A universal learning tool for classrooms? Proceedings of the First Quality in Teaching and Learning Conference. Hong Kong International Trade and Exhibition Center (HITEC), Hong Kong SAR, China 10-12 December 1998. http://celt.ust.hk/ideas/prs/pdf/Nelsoncue.pdf [viewed 3 Aug 2006] Cutts, Q., Carbone, A. & van Haaster, K. (2004). Using an electronic voting system to promote active reflection on coursework feedback. Proceedings of the International Conference on Computers in Education 2004, Australia. [verified 8 Apr 2007] http://www.dcs.gla.ac.uk/~quintin/papers/ICCE04QC.pdf Cutts, Q., Kennedy, G., Mitchell, C. & Draper, S. (2004). Maximising dialogue in lectures using group response systems. 7th IASTED International Conference on Computers and Advanced Technology in Education. Hawaii, August 2004. [verified 8 Apr 2007] http://www.dcs.gla.ac.uk/~quintin/papers/cate2004.pdf 206 Australasian Journal of Educational Technology, 2007, 23(2) Cutts, Q. & Kennedy, G. (2005). Connecting learning environments using electronic voting systems. In A. Young & D. Tolhurst (Eds), Australasian Computing Education Conference 2005, Australia. Conferences in Research and Practice in Information Technology, Vol 42. [verified 8 Apr 2007] http://www.dcs.gla.ac.uk/~quintin/papers/CRPITV42Cutts.pdf D’Inverno, R. (2003). Making lectures interactive. MSOR Connections, 3(1), 18-19. [verified 8 Apr 2007] http://www.maths.soton.ac.uk/staff/d'Inverno/zappers/msor.htm Draper, S. (2002). Using handsets at Glasgow University. [viewed 3 Aug 2006] http://www.psy.gla.ac.uk/%7Esteve/ilig/local.html Draper, S. & Brown, M. (2002). Use of the PRS (Personal Response System) handsets at Glasgow University, Interim Report. [viewed 3 Aug 2006] http://www.psy.gla.ac.uk/%7Esteve/ilig/interim.html Draper, S. & Brown, M. (2004). Increasing interactivity in lectures using an electronic voting system. Journal of Computer Assisted Learning, 20, 81-94. Draper, S., Cargill, J. & Cutts, Q. (2002). Electronically enhanced classroom interaction. Australian Journal of Educational Technology, 18(1), 13-23. http://www.ascilite.org.au/ajet/ajet18/draper.html Dufresne, R., Gerace, W., Leonard, W., Mestre, J. & Wenk, L. (1996). Classtalk: A classroom communication system for active learning. Journal of Computing in Higher Education, 7, 3-47. Duncan, D. (2005). Clickers in the Classroom. San Francisco: Pearson Education. Elliott, C. (2002). Case study: Economics lectures using a personal response system. The Economics Centre of the Learning and Teaching Support Network. [viewed 3 Aug 2006] http://www.economics.ltsn.ac.uk/showcase/elliott_prs.htm Elliott, C. (2003). Using a personal response system in economics teaching. International Review of Economics Education, 1(1). [viewed 3 Aug 2006] http://www.economics.ltsn.ac.uk/iree/i1/elliott.htm Freeman, M., Blayney, P. & Ginns, P. (2006). Anonymity and in class learning: The case for electronic response systems. Australasian Journal of Educational Technology, 22(4), 568-580. http://www.ascilite.org.au/ajet/ajet22/freeman.html Gardner-Medwin, A. (1995). Confidence assessment in the teaching of basic science. Association for Learning Technology Journal, 3(1), 80-85. Halloran, L. (1995). A comparison of two methods of teaching: Computer managed instruction and keypad questions versus traditional classroom lecture. Computers in Nursing, 13(6), 285-288. Herr, R. B. (1994). Computer assisted communication within the classroom: Interactive lecturing. Newark: Delaware University. ERIC Document ED416821. [verified 8 Apr 2007] http://eric.ed.gov/ERICWebPortal/contentdelivery/servlet/ERICServlet? accno=ED416821 Simpson and Oliver 207 Higher Education Fund Council for England (2005). HEFCE strategy for e-learning. [viewed 3 Aug 2006] http://www.hefce.ac.uk/pubs/hefce/2005/05_12/05_12.pdf Jones, P. (1999). Improving learning in lectures using keypad-response units. Proceedings of the 8th Annual Teaching Learning Forum. The University of Western Australia, 3-4 February. http://lsn.curtin.edu.au/tlf/tlf1999/jones.html Judson, E. & Sawada, D. (2002). Learning from past and present: Electronic response systems in college lecture halls. Journal of Computers in Mathematics and Science Teaching, 21(2), 167-181. Kennedy, G. & Cutts, Q. (2005). The association between students’ use of an electronic voting system and their learning outcomes. Journal of Computer Assisted Learning, 21, 260-268. Laurillard, D. (2002). Rethinking university teaching: A conversational framework for the effective use of learning technologies (2nd Edition). London: RoutledgeFalmer. Lopez-Herrejon, R. & Schulman, M. (2004). Using interactive technology in a short Java course: An experience report. The 9th Annual Conference on Innovation and Technology in Computer Science Education 2004 (pp. 203-207) Leeds, 28-30 June. Martin, E. (1999). Changing academic work: Developing the learning university. Buckingham: SRHE/Open University Press. Mazur, E. (1997). Peer instruction: A user’s manual. Englewood Cliffs NJ: Prentice Hall. McCabe, M., Heal, A. & White, A. (2001a). Integration of group response systems into teaching. In M. Danson & C. Eabry (Eds), Fifth International CAA Conference Proceedings. Loughborough University. [verified 8 Apr 2007] http://www.caaconference.co.uk/pastConferences/2001/proceedings/d2.pdf McCabe, M., Heal, A. & White, A. (2001b). New approaches to computer assessment for higher level learning. Proceedings of The Fifth International Conference on Technology in Mathematics Teaching. University of Klagenfurt, Austria, 6-9 August. http://ltsn.mathstore.ac.uk/articles/maths-caa- series/sep2001/index.shtml [viewed 3 Aug 2006, verified 9 Apr 2007] Nicol, D. & Boyle, J. (2003). Peer instruction versus class-wide discussion in large classes: A comparison of two interaction methods in the wired classroom. Studies in Higher Education, 28(4), 457-473. Oliver, M. & Conole, G. (2003). Evidence-based practice and e-learning in higher education: Can we and should we? Research Papers in Education, 18(4), 385-397. Oliver, M. & Harvey, J. (2002). What does ‘impact’ mean in the evaluation of learning technology? Educational Technology & Society, 5(3), 18-26. http://ifets.ieee.org/periodical/vol_3_2002/oliver.html Poulis, J., Massen, C., Roberts, E. & Gilbert, M. (1998). Physics lecturing with audience paced feedback. American Journal of Physics 66 (5), 439-441. [viewed 3 Aug 2006] http://www.bedu.com/Publications/PhysLectAPF.pdf 208 Australasian Journal of Educational Technology, 2007, 23(2) Purchase, H., Mitchell, C. & Ounis, I. (2004). Gauging students’ understanding through interactive lectures. In H. Williams & L. McKinnon (Eds), Proceedings of BNCOD 21. Lecture Notes in Computer Science, 3112: 234-243. [verified 8 Apr 2007] http://www.psy.gla.ac.uk/~steve/ilig/papers/hcp1.pdf Reay, N., Bao, L., Li, P., Warnakululasooriya, R. & Baugh, G. (2005). Towards the effective use of voting machines in physics lectures. American Journal of Physics 73(6), 554-558. Sharma, M., Khachan, J., Chan, B. & O’Byrne, J. (2005). An investigation of the effectiveness of electronic classroom communication systems in large lecture classes. Australasian Journal of Educational Technology, 21(2), 137-154. http://www.ascilite.org.au/ajet/ajet21/sharma.html Simpson, V. (2002). The effect on staff perceptions of online learning when using a non-traditional approach to staff development. In S. Banks, P. Goodyear, V. Hodgson & D. McConnell (Eds), Proceedings of Networked Learning 2002, pp. 550- 557. School of Education, University of Sheffield. [verified 8 Apr 2007] http://www.networkedlearningconference.org.uk/past/nlc2002/proceedings/ papers/37.htm Simpson, V. & Oliver, M (2002). Using electronic voting systems in lectures. UCL Internal report. [viewed 3 August 2006] http://www.ucl.ac.uk/ learningtechnology/examples/ElectronicVotingSystems.pdf Stuart, S. & Brown, M. (2004). An evaluation of learning resources in the teaching of formal philosophical methods. Association for Learning Technology Journal, 11(3), 58-68. Williams, D. (2002). As seen on TV. The Guardian Education. [viewed 3 August 2006] http://education.guardian.co.uk/higher/news/story/0,9830,719051,00.html Williams, J. (2003). ‘Learning by remote control’: Exploring the use of an audience response system as a vehicle for content delivery. In G. Crisp, D. Thiele, I. Scholten, S. Barker and J.Barron (Eds), Interact, Integrate, Impact: Proceedings ASCILITE Conference. (pp 739-838) Adelaide, 7-10 December. http://www.ascilite.org.au/conferences/adelaide03/docs/pdf/739.pdf Wit, E. (2003). Who wants to be…. The use of a personal response system in statistics teaching. MSOR Connections 3(2), 14-20. [verified 8 Apr 2007] http://www.psy.gla.ac.uk/~steve/ilig/papers/ernst.pdf Van Dijk, L. A., Van den Berg, G. C. & Van Keulen, H. (2001). Interactive lectures in engineering education. European Journal of Engineering Education, 26(1), 15-28. Vicki Simpson, Head of E-Learning George Edwards Building, University of Surrey Guildford, Surrey GU2 7XH, United Kingdom Email: V.Simpson@surrey.ac.uk Dr Martin Oliver, Senior Lecturer in ICT in Education London Knowledge Lab 23-29 Emerald Street, London WC1 3QS, United Kingdom Email: M.Oliver@ioe.ac.uk