Microsoft Word - freeman.doc Australasian Journal of Educational Technology 2006, 22(4), 568-580 Anonymity and in class learning: The case for electronic response systems Mark Freeman, Paul Blayney and Paul Ginns The University of Sydney This research presents the results of a study of alternative response methods for in class formative questioning. Students’ anonymity from their peers and instructor was studied through a research design that maintained a constant interactive teaching strategy in a large lecture hall, in all respects except for the method used by students to respond to the in class questions. A handheld electronic response keypad was the only approach affording complete anonymity. Student perceptions of the benefits of anonymity were obtained from a survey conducted at the end of the course. The results suggest that anonymity is a critical factor affecting student willingness to participate with in class exercises. Furthermore, the results indicate that students’ propensity to engage with in class questions increases with the degree of anonymity provided to the student in revealing their response. The benefit of anonymity, combined with the increased availability and affordability of electronic response systems, will be of interest to academics keen to design engaging learning environments. Introduction This paper reports upon the relative benefits of alternative response methods used to promote learner engagement in class, and in particular, benefits related to the anonymity afforded by handheld electronic response systems. Electronic response systems (ERS) are also known as personal response systems, audience response systems, electronic voting systems, classroom communication systems and classroom performance systems. The motivation for this paper arises from recent research on ERS by Draper & Brown (2004), Draper, Cargill & Cutts (2002), Crouch & Mazur (2001) and Poulis, Massen, Robens & Gilbert (1998), who provide anecdotal evidence that the anonymity provided by ERS appears to play a major role in explaining students’ propensity to engage, to join a class discussion, and to reveal their own response. Sharma, Khachan, Chan & O'Byrne (2005) find students more comfortable using keypads to respond to in class questions than doing so orally. The motivation for this study is to conduct a more systematic exploration of anonymity in explaining this preference. Freeman, Blayney and Ginns 569 It is well accepted that quality learning outcomes are more likely to occur when students adopt a deep approach to learning (Ramsden, 2003). While there are various factors that determine a student’s approach to learning, such as prior learning experiences, the student’s perception of the learning context is a strong determinant over which academics have influence. Assessment is the most obvious way academics can influence perceptions and behaviour. Ramsden (2003, p.182) takes this further by concluding that “from our students’ point of view, assessment always defines the actual curriculum”. Academics can also impact the student’s perception of the learning context by the in class experience. In a large research project with physics classes, Hake (1998) shows that interactive classes were correlated with higher student achievement. While Biggs (2003) reiterates that it is what the student does that directly affects learning outcomes, achieving interaction and engagement in large classes is more difficult. Promoting interaction in larger lecture halls can be achieved with students attempting questions individually and/or with peers (Boud, Cohen & Sampson, 1999). Student responses can be revealed by asking for or selecting one or more students to share their thoughts. In large classes either of these can be intimidating (e.g. unsure students may feel uncomfortable being selected, and shy students may feel uncomfortable being the focus of attention even if they know the correct response). Such alternatives are possibly ineffective and time consuming as well (e.g. hearing responses from students at the back of the lecture hall). Asking for a show of hands (i.e. ‘hands up if you think the answer is A?’) is somewhat less intimidating because some anonymity is possible amidst a crowd of hands (unless a further public verbal response might be required). While students can gauge their relative understanding from viewing the distribution of responses and the confirmation of the correct response from the instructor, it is limited by the class participation. While relative understanding can only be approximated by the hands up method, it should be sufficient for the academic to decide if further discussion is needed. An alternative to a sequential show of hands is to provide students with different coloured paper for each choice (e.g. Apricot for A, Blue for B etc) and thus seek their responses simultaneously (Harden, Wayne & Donald, 1968). However, in addition to it being administratively cumbersome (e.g. distribution of the coloured paper), this method still fails to provide complete anonymity to the students. An alternative, more anonymous approach is to use a paper based formative quiz that is marked on completion in class. Dihoff, Brosvic and Epstein (2004) report a novel approach that can provide feedback even sooner. Feedback is available immediately following each quiz question when students use a pre-formatted response sheet (called an Immediate Feedback Assessment Test or IF-AT). Students scratch off their preferred 570 Australasian Journal of Educational Technology, 2006, 22(4) response as if they were scratching a lottery ticket. If the first response scratched is incorrect, further scratches are made which can earn partial credit. Dihoff, Brosvic & Epstein (2004) report improved student achievement in subsequent tests or exams from students receiving such immediate feedback. This is no surprise because students are ripe for learning at such ‘teachable moments’ (Dempsey, Driscoll & Swindell, 1993). The main drawback with this method is the loss of spontaneity for the instructor to ask new questions since all questions must be pre-ordered to fit the pattern of answers beneath the scratchable response sheet’s pre- formatted surface. The final approach for eliciting and gathering students’ responses is facilitated by electronic means. Quizzes and tests can be marked by portable electronic scanner in the classroom, interactive computer assisted quizzes in an online or lab context, mobile phone, or a handheld audience electronic response keypad. Michaelsen, Knight and Fink (2004) report positive benefits from the use of portable scanners, but with a loss of flexibility for questioning for formative purposes. While computer labs are a scarce resource and wireless enabled laptops and mobile phones are still emerging as solutions for electronic questioning (Prensky, 2005) handheld electronic response systems appear to be the most viable current solution. Mazur (1997) provides an overview of how ERS can be used to facilitate student learning in large lectures and particularly with peer discussion. The level of anonymity provided by these different in class response methods is represented in Figure 1. At one end of the spectrum handheld ERS and other electronic means for obtaining student responses provide a high level of anonymity. At the other extreme is a student being randomly selected in class to provide a response to an instructor’s question - many students, especially the shy, less confident or unprepared ones, may feel threatened in such circumstances. Large lecture halls exacerbate this problem. Low Anonymity High Instructor Instructor selects Show of hands as Electronic randomly selects from volunteer instructor cycles student students through choices Figure 1: Anonymity provided by different interactive teaching strategies Even with academics purposefully designing for student-instructor interaction in large lecture halls using ERS, there will still be barriers for some students to participate actively. This reluctance can be alleviated partially by giving space for student-student interaction to achieve peer Freeman, Blayney and Ginns 571 learning, sometimes combined with self reflection, prior to student- instructor interaction. However, Draper and Brown (2004) observed that even in small classes when students were uncertain of the most appropriate response, they were more reluctant to join a discussion. This occurred even where students knew each other well and previously had not hesitated to join an oral group discussion. Draper, Cargill and Cutts (2002, p.15) note the advantage over the hands up alternative, in that ERS “crucially offers more privacy (it's a secret ballot, and important for just the same reasons)”. Crouch and Mazur (2001) make a similar observation. Poulis et al. (1998) suggest ERS use is associated with a reduction in fear and potential peer ridicule. Banks (2003) suggests that cultural background may be significant with a preference for ERS from those for whom criticism is problematic. Draper and Brown (2004) propose that increased understanding arises from anonymity because it induces students to pick a definite answer even when they are quite uncertain, and it is the latter that encourages student effort to then engage to produce an answer. While this may be similar to the pressure on a juror to make a decision when they feel ill equipped to do so, it is different in that a student’s decision does not have major irreversible consequences. Academics can choose to provide instant feedback on wrong answers, and if in class discussion is insufficient, students are more likely to be motivated to clarify their understanding after the class. While new technology may have some novelty effect, Draper and Brown (2004; p.89) conclude in relation to ERS that “it seems to last only somewhere between 5 and 50 minutes”. However, notwithstanding these advantages of ERS, there is considerable research to show that technology in and of itself does make any significant difference to student learning achievements (Cuban, 1986; Russell, 1999; Alexander & McKenzie, 1998; Kirkwood & Price 2005). What matters is the pedagogical design within which technology is used to facilitate learning. Following their review of three decades of ERS research, Judson and Sawada (2002; p.167) agree that this principle also applies specifically to ERS educational technology, concluding that the …literature points to the pedagogical practices of the instructor, not the incorporation of the technology as being key to student comprehension. Electronic response systems are viewed as a tool that holds a promise of facilitating earnest discussion. This research explores whether the anonymity afforded by ERS is a key contribution to the learning environment, valued by students over alternative response methods. It follows up on a study (Freeman & Blayney, 2005) that compared the use of an ERS with a traditional response method (i.e. a show of hands), when students were exposed to an 572 Australasian Journal of Educational Technology, 2006, 22(4) equivalent learning context in all respects except response method, which alternated weekly between hands up and ERS. Students were encouraged to engage in the learning activity by interacting with their peers before providing a response and by strong instructor encouragement to take a stand. While the instructor successfully elicited close to 100% participation with the ERS a much lower response rate was achieved with the show of hands instances. An unexpected finding was that students’ perceptions of their level of interaction in lectures, their understanding and their ability to gauge their own understanding, was significantly higher when the ERS was used. Anonymity was one factor highlighted in the qualitative research responses with statements like “everyone gets to participate without coming out” and “I don’t know why I am afraid to ask the lecturer questions”. The current study has attempted to evaluate anonymity as provided by the ERS as the key factor behind these improved learning experiences afforded by ERS educational technology. Method Participants A second year introductory management accounting course with 139 students at the University of Sydney provided the subjects for this research. This cohort was more diverse than regular classes as it comprised students from other institutions seeking to accelerate or catch up on their study program over an intensive, six-week 2005 Summer school. Students were surveyed at the end of the course (n = 135) to obtain their perceptions of various aspects of learning and particularly the importance of anonymity when responding to in class questions. The majority of respondents were female (73%) and 20-22 years old (93%) with a non-English speaking background (82%). Materials Similar to the earlier research, the current study evaluated the student learning experience over the extent of the course with all aspects of the lecture hall teaching strategy held constant except for the method used by students to respond to in class questions. Table 1: Alternative teaching strategies Stage of in class learning environment Teaching strategy PowerPoint presentation Multiple choice questions Prior peer discussion Response method Post response discussion ERS Yes Yes Yes Electronic Yes Non-ERS Yes Yes Yes Show hands Yes Freeman, Blayney and Ginns 573 Table 1 summarises the two teaching strategies: • Presentation software (PowerPoint) was used in each class to display learning material and questions on an overhead screen. A software add in provided the functionality for aggregating and displaying the results of the electronic questioning. • Each of the three hour classes were interspersed with formative, mainly rules based, multiple choice questions (MCQ), approximately every 15 minutes. • Students were encouraged to engage in reciprocal peer learning with one or more peers prior to indicating their preferred response to the MCQ. • The method by which students provided their response to the MCQ was alternated. In one class responses were collected using the ERS wireless keypads and in the next class by a show of hands. When the ERS was used the instructor would choose to immediately provide a clear display of the aggregate responses to each MCQ choice with the correct response highlighted in green. In the manual hands up approach the academic would simply announce the correct option following the sequence of show of hands for each possible choice. • Participation and engagement in the exercise was strongly encouraged by the instructor with both methods. Students were strongly encouraged to take a position even if they were not 100% certain if their response was likely to be correct. However, similar to the earlier research, the lecturer’s encouragement was considerably more successful with the ERS than with the show of hands method. • Following the receipt of student responses, the instructor would display the aggregate distribution for the class, typically as a bar chart. This distribution would guide the direction and extent of this discussion and the subsequent classroom dialogue. For example if a large proportion of students had selected a response reflecting a common misconception, then an extensive discussion of the theory behind that incorrect response would occur. Figure 2 illustrates a typical in-class question and the subsequent immediate results of the aggregated response for the cohort attending the lecture and using the ERS. Only 43% of the class selected the correct response (answer B) for this management accounting question. Subsequent to the instant feedback being displayed to students, the instructor illustrated why answers A and C were incorrect and why these reflected common misunderstandings. Some further class discussion ensued. 574 Australasian Journal of Educational Technology, 2006, 22(4) Figure 2: A typical ERS enhanced learning activity Procedure A survey was used to collect students’ qualitative feedback comments as well as quantitative ratings. A paper based survey rather than the ERS was used, to allow students to provide responses at their own pace and to have an opportunity to reflect and change their entry if they wished to do so. A second reason for using a paper based survey was to obtain qualitative responses. Neither changing of earlier responses nor free response comments are possible with ERS. The survey sought student evaluations on seven rating statements using a five-point Likert scale (strongly disagree to strongly agree). Four of these statements sought ratings on various aspects of learning design and the remaining three statements directly addressed the role of anonymity. 1. “The use of non-assessable questions during class helped me to learn” 2. “I was more likely to prepare for class if I knew we would complete questions in class” 3. “Before answering such questions in class I learned more by first discussing my thoughts/ideas with other students” 4. “I was more likely to follow up gaps in my learning after class when I received feedback on my understanding from such in-class questions”. 5. “I preferred answering such questions when my answers were anonymous to the instructor” (NB. Anonymity was achieved with a keypad and the electronic response system; Anonymity to instructor could also have been achieved by written responses being marked by peers only) 6. “I preferred answering such questions when my answers were anonymous to other students” (NB. Anonymity was achieved with a Freeman, Blayney and Ginns 575 keypad and the electronic response system; Anonymity to peers could also have been achieved by use of a written quiz collected by the instructor) 7. “Anonymity was more important with such questions when I was uncertain about the answer”. Perceptions of anonymity were also elicited from a ranking question based on Figure 1. Students ranked their preference for the four response methods using a scale of 1 (most preferred) to 4 (least preferred). These ranged from the condition expected to provide high anonymity, namely “using a keypad to submit my answers with my identity anonymous to academic and peers”, to the other extreme “being selected by the academic to provide my answer”. In between these extremes were “raising my hand when asked how many students thought the answer was ‘a’ ‘b’ ‘c’ etc” and “raising my hand when I knew the answer and then being selected to respond”. To assist with the further analysis various demographic data was collected including gender, age, tuition basis (i.e. government subsidised or fully self funded), hours per week working for payment, and first language. Finally, the survey sought students’ evaluations using two free response items to obtain qualitative feedback on the best aspects of the course and those aspects needing improvement. Statistical analysis of the survey results was undertaken in several different forms. Students’ responses to the Likert scale items were analysed using both the “percentage agreement” metric (percentage endorsing either “agree” or “strongly agree”) and the “percentage disagreement” metric (percentage endorsing either “disagree” or “strongly disagree”). Students’ ranking of response preferences were analysed using the Friedman test (Conover, 1980) with a Bonferroni correction for follow up multiple comparison tests between specific preferences (Hsu, 1996). Mann-Whitney and t-tests were conducted to test for differences due to demographic variables. Results and discussion Table 2 below indicates students had a predominantly positive view of the use of formative questions to enhance interaction during lectures. 72% of students (strongly) agreed that non-assessable questions in class helped their learning. Only 11% (strongly) disagreed that greater preparation was more likely. Greater post-class follow up of learning gaps was also more likely. Lower agreement about the benefit of peer discussion was apparent with the majority unwilling to take a stand on this issue (i.e. 48% neutral). The finding that peer discussion was not highly valued in this usage, 576 Australasian Journal of Educational Technology, 2006, 22(4) contrary to previous research by Crouch and Mazur (2001) and Judson and Sawada (2002), might be due to the nature of our intensive course which contains a more diverse student cohort. There were no significant differences in these ratings for any of the demographic variables collected including gender, age, tuition basis, hours per week working for payment and first language. Table 2: Student perceptions of in-class questioning - learning aspects Use helped learning Greater class preparation Greater follow up of learning gaps Greater learn- ing by peer discussion No. % No. % No. % No. % Strongly disagree 2 1% 2 1% 1 1% 11 8% Disagree 4 3% 14 10% 8 6% 18 13% Neutral 31 23% 34 25% 31 23% 65 48% Agree 57 42% 69 51% 69 51% 38 28% Strongly agree 41 30% 16 12% 26 19% 3 2% Total 135 100% 135 100% 135 100% 135 100% The issue of anonymity was directly addressed in Table 3. Students generally prefer to not reveal their identity when they respond to in-class questions. Students value anonymity to their instructor (68%) more than anonymity to their peers (62%). Anonymity is important when students are uncertain of the answer (63%). Of course, there is always likely to be a proportion of students who are less shy and more self confident and thus for whom anonymity is not an issue. We surmise that it is those students who might be more likely to participate in a show of hands or even offer to respond if the instructor makes a general request to the class for an answer. Again there were no significant differences in these ratings for any of the demographic variables. Table 3: Student perceptions of in-class questioning – anonymity aspects Preference for anonymity from instructor Preference for anonymity from other students Preference for anonymity when uncertain No. % No. % No. % Strongly disagree 1 1% 2 1% 2 1% Disagree 8 6% 8 6% 10 7% Neutral 33 24% 42 31% 37 27% Agree 60 44% 52 39% 64 47% Strongly agree 33 24% 31 23% 22 16% Total 135 100% 135 100% 135 100% Table 4 indicates students' preferred response method as identified by the mean ranking per preference. The method affording the greatest anonymity, namely the ERS, had the highest ranking (mean ranking 1.4), Freeman, Blayney and Ginns 577 followed by show of hands (mean ranking 2.3), volunteering an answer (mean ranking 2.65) and being selected by the instructor as least preferred (mean ranking 3.66). A complete set of rankings was given by 124 students. The overall test was significant, Q(3) = 194.86, p < 0.001, indicating that statistically reliable differences existed between students’ rankings of preferences. The results of the Bonferroni-corrected pairwise comparisons between rankings, using a critical value for the difference of 53.42 are in the final three columns of Table 4. Table 4: Preferred response method Matrix of pairwise comparisons (difference) Response method Mean rank Response method 4 Response method 3 Response method 2 1. Using a keypad to submit my answers with my identity anonymous to academic and peers 1.40 277.50 * 154.00 * 110.50 * 2. Raising my hand when asked how many students thought the answer was “a”, “b”, “c”, etc. 2.30 167.00 * 43.50 ns 3. Raising my hand when I knew the answer and then being asked to respond 2.65 123.50 * 4. Being selected by the academic to provide my answer 3.66 * difference between preference rankings statistically significant ns difference between preference rankings not statistically significant The results in Table 4 concur with our expectations, namely that the level of anonymity afforded by a response method is a good predictor of students’ preferences. This preference is significant in all cases except between alternate variations of raising hands. Further analysis of the above ranking data using Mann-Whitney tests found no significant difference in preference ratings between any of the demographic variables. Interestingly, there was no significant difference in preference ratings between students whose first language was English or those for whom it was not. Using first language as a rough proxy for culture, and in particular openness to criticism, these results might be seen to contradict Banks (2003) who suggested that cultural background could impact preference for ERS usage. Future research should explore this with a better proxy. Conclusion This research builds on Sharma et al. (2005) who found students ‘more comfortable’ with ERS than orally answering questions, and on Freeman 578 Australasian Journal of Educational Technology, 2006, 22(4) and Blayney (2005) who found electronic response systems positively influenced student perceptions of the learning experience. Iterating between an ERS and non-ERS teaching strategy, Freeman & Blayney (2005) found students had a significantly higher preference for ERS in relation to their level of interaction in lectures, their understanding of material content gained during lectures and their ability to gauge understanding of material content in lectures. This research suggests that anonymity, as afforded by electronic response systems in this context, is a critical factor affecting student willingness to participate with in class exercises. The results of this study indicate that students’ propensity to engage with in class questions increases with the degree of anonymity provided to students in revealing their responses. No significant differences in preferences were found between major demographic variables. While open discussion is possible (and even the norm) in small post- graduate classes, this is not the case in a typical university lecture hall with a large class of undergraduate students. As class sizes increase due to funding constraints, the findings in this study are increasingly important for academics wishing to engage students in such challenging contexts. However, despite the benefits for student learning, the costs to instructors of adopting ERS are not necessarily small. While the monetary costs of such systems have declined with technological improvements in recent years, and even with textbook publishers strongly supporting such adoptions (with limited or no cost to the instructor), penetration into higher education contexts appears to be slow and quite limited. Physics appears to be the exception (Sharma et al., 2005; Crouch & Mazur, 2001). This may be due in part to strong cultural factors within disciplinary circles and the earlier influential work of Mazur (1997), specifically in physics. Indirect costs on instructors, such as the time and hassle of the distribution and collection of keypads, also need to be further explored. The instructor experience with ERS is the subject of other systematic research currently underway. The limitations of this study include sample size, the single site context, and focus on perceptions. These suggest profitable future opportunities for research relating to ERS and anonymity benefits to students that can be achieved by larger samples and other contexts. Experimental research methods that focus on increased understanding, and learning contexts designed to provide anonymity using ERS or other technologies are possible research avenues. Better proxies for cultural factors, such as openness to criticism to unpack the influence on anonymity, may also be useful to investigate. Finally, further research focussing on the instructor experience with ERS would be particularly useful. Notwithstanding these limitations, the finding of the benefit of anonymity in promoting in class engagement and interaction, combined with the Freeman, Blayney and Ginns 579 increased availability of and anonymity afforded by electronic response systems, will be of interest to instructors keen to design engaging learning environments. These findings will also be of interest to those investigating educational technologies more generally that afford anonymity since the latter appears to be an important aspect increasing engagement and interaction. References Alexander, S. & McKenzie, J. (1998). An evaluation of information technology projects for university learning. Canberra: AGPS. [verified 31 Oct 2006; Executive summary] http://www.dest.gov.au/archive/cutsd/publications/exsummary.html Banks, D. (2003). Using keypad-based group process support systems to facilitate student reflection. In G. Crisp, D. Thiele, I. Scholten, S. Barker & J. Baron (Eds), Interact, Integrate, Impact: Proceedings of the 20th ASCILITE Conference, Adelaide. http://www.ascilite.org.au/conferences/adelaide03/docs/pdf/37.pdf Biggs, J. (2003). Teaching for quality learning at university. 2nd ed. Buckingham: Open University Press. Boud, D., Cohen, R. & Sampson, J. (1999). Peer learning and assessment. Assessment and Evaluation in Higher Education, 24(4), 413-426. [verified 31 Oct 2006] http://www.education.uts.edu.au/ostaff/staff/publications/db_23_bcs_aehe99.pdf Conover, W. J. (1980). Practical nonparametric statistics. New York: John Wiley. Crouch, C. H. & Mazur, E. (2001). Peer instruction: Ten years of experience and results. American Journal of Physics, 69(9), 970-977. Cuban, L. (1986). Teachers and machines: The classroom use of technology since 1920. New York: Teachers College Press. Dempsey, J. V., Driscoll, M. P. & Swindell, L. K., (1993). Text-based feedback. In J.V. Dempsey & G. C. Sales (Eds), Interactive Instruction and Feedback (pp.21-54). Englewood Cliffs, NJ: Educational Technology Publications. Dihoff, R. E., Brosvic, G. M. & Epstein, M. L. (2004). Provision of feedback during preparation for academic testing: Learning is enhanced by immediate but not delayed feedback. The Psychological Record, 54(2), 207-231. [verified 31 Oct 2006] http://www.med.wright.edu/aa/facdev/_Files/PDFfiles/ImmediateFeedback.pdf Draper, S. W. & Brown, M. I. (2004). Increasing interactivity in lectures using an electronic voting system. Journal of Computer Assisted Learning, 20(2), 81-94. Draper, S. W., Cargill, J. & Cutts, Q. (2002). Electronically enhanced classroom interaction. Australian Journal of Educational Technology, 18(1), 13-23. http://www.ascilite.org.au/ajet/ajet18/draper.html Freeman, M. & Blayney, P. (2005). Promoting interactive in-class learning environments: A comparison of an electronic response system with a traditional alternative. In S. L. Cheung (Ed), Innovation for Student Engagement in Economics: Proceedings of the Eleventh Australasian Teaching Economics Conference, pp.23-34. http://hdl.handle.net/2123/199 [viewed 13 Oct 2006, verified 31 Oct 2006] 580 Australasian Journal of Educational Technology, 2006, 22(4) Hake, R. (1998). Interactive engagement versus traditional methods: a six thousand – student survey of mechanics test data for introductory physics course. American Journal of Physics, 66(1), 64-74. Harden, R. M., Wayne, E. & Donald G. (1968). An audio-visual technique for medical teaching. Journal of Medical and Biological Illustration, 18(1), 29-32. Hsu, J. C. (1996). Multiple comparisons: Theory and methods. London: Chapman & Hall. Judson, E. & Sawada, D. (2002). Learning from past and present: Electronic response systems in college lecture halls. Journal of Computers in Mathematics and Science Teaching, 21(2), 167-181. [verified 31 Oct 2006] http://www.aace.org/dl/files/JCMST/JCMST212167.pdf Kirkwood, A. & Price, L. (2005). Learners and learning in the twenty first century: What do we know about students’ attitudes towards and experiences of information and communication technologies that will help us design courses? Studies in Higher Education, 30(3), 257-274. Mazur, E. (1997). Peer instruction: Manual. New Jersey: Prentice Hall. Michaelsen, L., Knight A. & Fink, L. (2004). Team-based learning: A transformative use of small groups. Stylus Publishing LLC. Poulis, J., Massen, C., Robens, E. & Gilbert, M. (1998). Physics lecturing with audience paced feedback. American Journal of Physics, 99(5), 439-441. Prensky, M. (2005). What can you learn from a cell phone? Almost anything!. Innovate 1(5). http://www.innovateonline.info/index.php?view=article&id=83 [viewed 15 June 2005, verified 31 Oct 2006] Ramsden, P. (2003). Learning to Teach in Higher Education. 2nd ed. London: Routledge. Russell, T. (1999). The 'No Significant Difference' Phenomenon. http://www.nosignificantdifference.org/ Sharma, M. D., Khachan, J., Chan, B. & O'Byrne, J. (2005). An investigation of the effectiveness of electronic classroom communication systems in large lecture classes. Australasian Journal of Educational Technology, 21(2), 137-154. [viewed 21 Aug 2006] http://www.ascilite.org.au/ajet/ajet21/sharma.html Mark Freeman Office of Learning and Teaching in Economics and Business Faculty of Economics and Business, The University of Sydney Email: m.freeman@econ.usyd.edu.au Paul Blayney, Discipline of Accounting Faculty of Economics and Business, The University of Sydney Email: p.blayney@econ.usyd.edu.au Paul Ginns, Institute for Teaching and Learning, The University of Sydney. Email: p.ginns@itl.usyd.edu.au