Assessment in the Modern Large High-Tech Classroom ASSESSMENT IN THE MODERN LARGE HIGH-TECH CLASSROOM Peter M. Markulis SUNY Geneseo Markulis@Geneseo.Edu Daniel R. Strang SUNY Geneseo ABSTRACT This paper offers a way for instructors in large, high-tech classrooms to meet the challenge of simultaneously teaching and assessing. The paper explains how it is possible for an instructor in a large class to use hand-held clickers to assess student learning and to modify his/her lectures “on- the-fly.” The paper also presents the results of an experiment that the researchers conducted to determine the impact that clickers have on the classroom. Some of the results, particularly those involving attendance and student involvement, are very encouraging. Other results are less encouraging and call for additional attention and, perhaps, additional research. INTRODUCTION The most recent wave to hit higher education and education at all levels is the push for meaningful assessment. In addition to the traditional demands placed upon college instructors, they are now facing pressures to provide concrete evidence that learning is occurring in their classrooms. Colleges across the county are scrambling to develop mechanisms and procedures to provide an assurance that learning is occurring in the courses that are being taught. Since AACSB has embraced the assessment mantra, Schools of Business have devised various strategies and programs to comply with the demands of assessment. For example, the AACSB has sponsored a series of seminars to assist business schools with assessment activities. Their Basic Seminar Series focuses on the goals and purposes of assessment while the Applied Assessment Series provides practical suggestions for the design and implementation of assessment protocols. A two-volume series called, “Assessment of Student Learning in Business Schools: Best Practices Each Step of the Way (Martell & Calderon, 2005) has been co-published by the AACSB to assist deans and assessment coordinators in their efforts to develop meaningful assessment programs for the member schools. Coincident with this emphasis on assessment has been the growing use of computer technology in the classroom (Peluchette & Rust, 2005). Some of the preliminary research done in this new direction of pedagogy has focused on pedagogy in the field of economics. Clearly, a key question is “how does the use of computer technology affect student performance.” Sosin et al. (2004) use the post- and pre-course scores on the Test of Understanding College Economics (TUCE) to compare the effects of different technologies including PowerPoint, email, courseware, and web browsing. They find that using extensive technology in class causes significant, but small, improvement in student performance. Interestingly, they find individual technologies have different effects. For example, PowerPoint has a negative effect on student performance, courseware has a positive effect but only in macroeconomics courses, and e- mailing materials has a positive effect only in microeconomics courses. Agarwal and Day (1998) use both TUCE scores and final grades to analyze the impact of the Internet tools such as email, class mailing list discussion, and relevant web pages on economic education. They find positive correlation between using the Internet tools and exam grades. Manning (1996) reports that e-mail is useful as a teaching tool as it improves students’ communications with the instructor and other students. As Goffe and Sosin (2005) point out, there is a trend that more technological innovations will be used in teaching. The Personal Response System (PRS) is one of the newest technologies being introduced to the classroom. A number of instructors report how the PRS enhances teaching and learning experiences (e.g., Wood, 2004; Briggs, 2006; Elliott, 2003). The existing literature reports that using the PRS enhances student-instructor interaction and student concentration in class. Siegel (2004) also reports that using a technology similar to the PRS increases class attendance when five percent of the final grade is associated with student participation. This study reveals how one of the newest advances in technology, hand-held student response units, also more commonly referred to as clicker, were used in large lectures to enhance instruction and simultaneously address that question of how to do assessment. After researching the several clickers available in the market, the instructor adopted the radio frequency (RF) Interwrite PRS sold by Interwrite Learning shown in Figure 1. Developments in Business Simulation and Experiential Learning, Volume 35, 2008 191 mailto:Markulis@Geneseo.Edu Interwrite RF PRS Figure 1 Readers may wish to consult http://www.interwritelearning.com/products/prs/radio/de tail.html . This study is based upon the experience that one professor has had in two sections of principles of microeconomics over the course of a year. The two sections include a section of 110 students in the fall of 2005, and a section of 113 in the fall of 2006. In the fall of 2006, the instructor initiated the use of clickers as a device to instantaneously record student responses to quiz questions. Each student was required to have a clicker and responded to quiz questions that the professor administered during the lecture sessions. In this study the instructor restricted the testing format to that of multiple-choice questions. The clickers that currently are available have many additional abilities including the ability of the student to submit answers to short open-ended questions. The students were informed that 20% of their course grade was based upon their performance on “clicker quizzes.” The course was taught at 8:30 a.m. on Tuesdays and Thursdays and quizzes were administered on most days during which a major exam was not being administered or discussed. The quizzes were administered at various points during the class sessions--at the beginning, middle and end of the class. On some occasions, two quizzes were given during a single class session. Although clickers had not been used in the section of microeconomics in the fall of 2005, many of other potentially significant factors were the same as those for the fall of 2006 section. The classes were taught at 8:30 a.m. on Tuesdays and Thursdays, the number of students was virtually same (110 versus 113) and testing procedures, with the exception of the quizzes, were the same. There is no reason to believe that the composition of students in terms of major and non-major and class year was different in any significant way. The same textbook was used in both sections. Thus, the only major difference was that students in the fall 2005 section achieved 20% of their grade based upon quizzes provided by the textbook publisher that they took online and students in the fall 2006 section achieved 20% of their grade based upon clicker quizzes. The use of clickers does offer the advantage of ease of maintenance of attendance records. So, attendance information was recorded for the section of microeconomics in the fall 2006. Since it was not practicable, attendance data was not collected for the section of microeconomics in the fall 2005. The grades for the two sections of microeconomics included in this study were based upon the same formula. In both sections, 20% of the grade was based upon quizzes—online quizzes in fall 2005 and clicker quizzes for fall 2006. The remainder of the grade was based upon three preliminary exams that were spaced out throughout the semester and weighted at 50% and a comprehensive final at 30%. All of the preliminary exams and the final were multiple-choice exams. In order to get a measure of the impact of clicker quizzes, the same final examination was administered to the sections of microeconomics in the fall of 2005 and the fall of 2006. Given that virtually every other key variable except for the use of clickers was the same for the two sections, differences in the results on final examinations could be attributed to the impact of the clicker quizzes versus online quizzes. A little further explanation of the specifics of the use of quizzes may be appropriate. In the section in which online quizzes were used, the class was divided into groups of three-person teams and the team members worked together on the quizzes. The quiz sessions were conducted by the three-person teams outside the classroom and no restrictions were imposed on resources used by students during quiz taking sessions. Additionally, the three-person teams were allowed to make multiple attempts at the quizzes and to submit the highest quiz scores for inclusion in the course grade. In contrast, the clicker quizzes were all administered during the lecture periods. Students were encouraged to confer on their answers and awarded points for the answers based on the following scheme—they received two points for each correct answer, one point for each incorrect answer and zero points for failure to answer. Duncan (2005) recommends this type of point allocation scheme in his booklet, Clickers in the Classroom. Anecdotally, the students seemed to like the award system. Students were encouraged to confer for two reasons—to Developments in Business Simulation and Experiential Learning, Volume 35, 2008 192 http://www.interwritelearning.com/products/prs/radio/detail.html http://www.interwritelearning.com/products/prs/radio/detail.html Two-sample difference in means test for final grades fall ’05 and fall ‘06 Table 1 Two-sample T for Finals grade 05 vs Finals grade 06 N Mean StDev SE Mean Finals grade ‘05 110 74.7 16.0 1.5 Finals grade ‘06 113 73.4 14.4 1.4 H0: :05 - :06 = 0 H1: :05 - :06 ≠ 0 T-Test of difference = 0 (vs ≠): T-Value = 0.63 P-Value = 0.532 DF = 217 promote the natural synergism of students working together, and more pragmatically, it would very difficult to enforce a no-collaboration restriction. A notable difference between the online quiz approach and the clicker quiz approach is that the online approach provided students the opportunity to be virtually unlimited in terms of the number of quizzes that the students could attempt. RESULTS The instructor initiated the use of clickers with high expectations and a great deal of enthusiasm. Unfortunately, the results were mixed. As described below, some results were favorable while some were not. The research design that was established would seem to be ideal to determine if the use of clickers had any direct impact on learning. To the extent that it was possible, every variable that could be controlled was controlled. To be specific, the class sizes were comparable. The classroom was the same. The time of day was the same. The format for testing, with the exception of the quizzes, was the same. The student audience was roughly the same in terms students who are majors versus non-majors, year in school, etc. Although the preliminary examinations were different, the same multiple-choice final exam was given in both sections. Since virtually every other factor that might impact the performance on the final exams was controlled, the only significant treatment that one would expect to show up would be the difference associated with in the administration of quizzes—online quizzes versus clicker quizzes. It was expected that the class that utilized the clickers would demonstrate a statistically significant difference in mean grade on the final exam from the class that utilized online quizzes. Much to the consternation of the researchers, the mean grade for the 110 students that took the final in the class that used online quizzes was 74.7 and the mean grade for the 113 students that took the final in the class that used the clickers for quizzes was 73.4. The simple difference in means test shown in Table 1 leads to the conclusion that there was no statistically significant difference in means. Careful scrutiny reveals a slight, although not statistically significant decline in mean of finals associated with the use of clickers. Although the researchers were initially perplexed and disappointed by these results, on further reflection, these results should not necessarily have been unexpected. Under both treatments (with online quizzes and with clicker quizzes) students were given the opportunity to hone the skills and understanding of microeconomic concepts via quizzes. Additionally, the manner in which both online quizzes and clicker quizzes were administered encouraged collaborative work and collaborative answers by students. Under these circumstances, perhaps a significant difference in means (hopefully in the positive direction) is too much to expect. Be that as it may, there were a number of other benefits that were associated with the use of clickers that are worth discussing, and some of these were germane for assessment purposes. ATTENDANCE Research findings by Marburger (2006) and Romer (1993) suggest the importance of attendance in college classes and Marburger additionally suggests that absenteeism interferes with learning. They are not earthshaking discoveries but buttress any argument or scheme that has the impact of bolstering attendance. The use of clickers makes the process of maintaining attendance records trivial. The mean attendance for the entire semester for the class that used clickers was 88%. Incidentally, that mean includes the Tuesday of the week of Thanksgiving with a class for which the attendance dropped to 65%. Although the authors were not able to find meaningful comparable numbers for large class (+100 students) attendance in the literature, the occasional references that they found seem to suggest attendance numbers in the 60% to 70% are fairly “typical.” Without the use of clickers, attendance records were not maintained for the comparison class in this study, but the instructor reported his impression that the typical attendance would be in the 70% range and, without a doubt, not at the level of 88% found in the “clicker class.” Another interesting dimension of attendance is the performance by each individual. If clickers are used every day (with the exception of days during which major exams were given or being discussed) then one can easily establish Developments in Business Simulation and Experiential Learning, Volume 35, 2008 193 Missed classes per student Table 2 Missed Classes Number of students Percent Cumulative Percent 0 31 27.4 27.4 1 29 25.7 53.1 2 20 17.7 70.8 3 14 12.4 83.2 4 5 4.4 87.6 5 3 2.7 90.3 6 7 6.2 96.5 7 1 0.9 97.3 8 3 2.7 100.0 Withdrawal Rates Table 3 Class Students that Withdrew Students Initially in course Percent withdrawal Clicker class 12 125 9.6% One year earlier 6 116 5.2% Two years earlier 5 111 4.5% Three years earlier 3 75 4.0% individual attendance patterns. Table 2 shows the pattern of missed clicker classes for the students for an entire semester. An impressive 27.4% of the students missed no classes, 25.7 missed one class and 70.8% of the students missed 2 or less classes. This is compelling evidence that the use of clickers is a strong incentive for students to attend class. Do the students like the clickers? The researchers have surveyed students in subsequent classes to find out student attitudes about clickers. At this point, the results are too tentative for inclusion in this paper. One crude measure of the student attitude about a course is their disposition to complete or withdraw from a course. Recognizing the imprecision of this measure, the authors did record the rates of withdrawal from the section of the course that used clickers and from roughly equal sections in previous years. Table 3 presents the percent withdrawal rates for the section of the class in which clickers were used and for comparable sections of the class one, two and three years earlier during which clickers were not used. ASSESSMENT Perhaps the most compelling reason for instructors to consider adopting clicker technology is the pedagogical implications in terms of assessment. As noted earlier, accrediting bodies are encouraging--often requiring--that assessment of learning be conducted at both the program and course levels in universities. A particularly nettlesome issue for assessment purposes is that of class size (Buchanan & Rogers, 1990). Little research exists on how to conduct assessment for large-scale (+100 students) classes (Martell, 2007). While many argue that smaller class size improves learning (Robertson, 2005), the evidence is not clear-cut (Ehrenberg, et. al, 2001). Despite the controversy over class size, the trend in many colleges is to have introductory level classes at the 100 students plus mark. Efficiency is often cited as the principle reason (Surry, 2000). Class size ipso facto seems to have some affect on the choice of technology, although as Peluchette and Rust note, some instructors see technology as facilitating learning in large classes, while others see it as a detriment (2005, 202). In terms of assessment, clickers provide the opportunity for everyone in an educational setting--both the students and the instructors--to know instantaneously what is being learned and what is not being learned. According to a study by Epstein, et al., (2002) which used the immediate assessment feedback technique (IFAT), “Active involvement in the assessment process plays a crucial role in the acquisition of information, the incorporation of accurate information into cognitive processing mechanisms, and the retrieval of correct answers during retention tests…studies indicate that the IFAT method actively engages learners in the discovery process and that this engagement promotes retention and the correction of initially inaccurate response strategies.” (2002, Developments in Business Simulation and Experiential Learning, Volume 35, 2008 194 187). Assessment, when used formatively, can be used immediately to see what students know before proceeding with additional material. To illustrate the point and to demonstrate how clickers were used in the microeconomics class, consider the following question from microeconomics. 1. Refer to the above diagram. At output level Q total fixed cost is: A) 0BEQ. B) BCDE. C) 0BEQ-0AFQ. D) 0CDQ. To avoid embarrassment on the part of readers who may not have thought about microeconomics for a number of years, the correct answer is B and marked in bold. When this question was presented using clickers in the class, the instructor and the class knew in seconds that 83% of the class got it correct. IMMEDIATE FEEDBACK— IMMEDIATE ASSESSMENT At that point, the instructor had to make several important pedagogical decisions. Was 83% high enough to justify moving to the next question? What about the 17% that got it wrong? The instructor has established his decision rules as to how to proceed. It is an interesting exercise to involve the students in the process of establishing these decision rules (What is the threshold percent correct for the class for which the instructor should decide to go back and review or proceed?). At this point, the instructor moved to the next question utilizing the same diagram. The second question reads: 2. Refer to the above diagram. At output level Q average fixed cost: A) is equal to EF. B) is equal to QE. C) is measured by both QF and ED. D) cannot be determined from the information given. Again the correct answer has been presented in bold typeface. For the second question, 53% of the class got the correct answer. Based on this immediate assessment information, the instructor realized that he had not been successful in explaining how to determine fixed cost from a cost diagram. The instructor held additional clicker questions in abeyance and branched the lecture to review the coverage of determination of costs in a cost diagram. After a brief review, the instructor returned to clicker mode and asked the following question that addressed a “nuanced issue” of cost determination based upon the same diagram as question 2. 3. Refer to the above diagram. The vertical distance between ATC and AVC reflects: A) the law of diminishing returns. B) the average fixed cost at each level of output. C) marginal cost at each level of output. D) the average total cost at each level of output. In this instance, 93% of the students responded with the correct answer. This response indicated that student understanding reached a point to justify moving ahead in the topical coverage of the lecture. Along with immediate assessment, Hoffman and Goodwin (2006) in their study of librarians using clickers to teach information literary, suggest the following benefits from using ‘clicker’ technology: • ensures interactions. The clickers provided an unexpected way to chat with students prior to class. .. students are curious about the clickers and what the instructors were about to teach… • keeps students focused. Incorporating clicker questions into the lecture noticeably helped keep students focused by restarting the attention span with each new questions • increases participation. As experience with the Aggie Honors classes, the ability to make student feedback anonymous helped reduce fears and increased students’ willingness to participate • Promotes discussion. …more questions (came from) from students than compared to classes taught without clickers. • Increases retention. All the benefits ultimately help improves student retention which is the goal of active learning. • Its fun! CONCLUSIONS The authors’ experience with the use of clickers provides some mixed messages. Although, it was hoped that this study would demonstrate that the use of clickers had a direct impact on the student performance on the final exam, that was not the result since the students were provided a mechanism to determine their learning through quizzes in the control class (i.e., online quizzes) and the experimental class (i.e., clicker quizzes). While student performance, as measured by final test Developments in Business Simulation and Experiential Learning, Volume 35, 2008 195 scores, did not increase due to their use of clicker technology as compared to more traditional methods, clicker usage did unquestionably increase attendance. Further, clickers were an invaluable—and perhaps the only way, in which to engage in meaningful classroom assessment for large-size classes. This finding is supported by additional anecdotal evidence as reported by Carnevale in the Chronicle of Higher Education (2005). Carnevale reports that when biology professor Mark Coykendall, at the College of Lake County, poses a question to the class, every one of his students' hands spring into the air. The students hold up translucent blue remote control devices and press buttons to register their answers to multiple-choice questions throughout each class session. The students' responses get tallied by a computer, and within seconds, a graphic is displayed on a large pull-down screen showing how many in the class grasped the concept and how many had no clue… It's every instructor's conundrum. How do you know when the class truly gets what you're teaching, or when you need to try a different approach…For years I've always asked, 'Does everybody understand this concept or this point?' and they nod their heads," Mr. Coykendall says. "Now I'll know right away whether or not they really understands." Clearly, students were more engaged in the learning process in the class that utilized clicker technology. This result is predictable if for no other reason than students sat in class knowing that the instructor could go into “quiz mode” at any point during the lecture. REFERENCES Agarwal, R., & Day, A. E. (1998). The impact of the internet on economics education. Journal of Economic Education 2. (Spring): 99-110. Briggs, L. (2006). Response devices keep FSU students focused. Campus Technology, Retrieved 11/27/2006, from http://www.campustechnology.com.article.aspx?aid=41 307. Buchanan, R. W. & Rogers, M. (1990). Innovative assessment in large classes. College Teaching, (Spring) Vol. 38, Issue 2, pp. 69-74. Carnevale, D. (2005). Run a class like a game show: ‘Clickers’ keep students involved. The Chronicle of Higher Education. Washington: June 24, 2005. Vol. 51, Issue 42; p. 3. Duncan, D. (2005). Clickers in the Classroom, Addison- Wesley, San Francisco. Ehrenberg, R. G. et al., (2001). Does class size matter? Scientific American, (November) Vol. 285, Issue 5, pp. 78-86. Elliott, C. (2003). Using a personal response system in economics teaching. International Review of Economics Education 1:80-86. Available online at http://www.economicsnetwork.ac.uk/iree/il. Epstein, M.L., et al,. (2002). Immediate feedback assessment technique promotes learning and corrects inaccurate first responses. The Psychological Record. (Spring) Vol. 52, Issue. 2: 187-102. Goffe, W., & Sosin, K.. (2005). Teaching with technology: may you live in interesting times. Journal of Economics Education 36.” (Summer): 278-292. Hoffman, C., & Goodwin, S. (2006) A clicker for your thoughts: Technology for active learning. New Library World, Vol. 107, Issue 9/10, p. 422. Marburger, D. (2006). Does mandatory attendance improve student performance? Journal of Economic Education 37.” (Spring): 148-156. Martell, K. (2007). E-mail correspondence, Assessment in large classes, Mon, 17 Sep 2007. Martell, K. & Calderon, T. (eds) . (2005). Assessment of Student Learning in Business Schools: Best Practices Each Step of the Way. The Association for Institutional Research and the AACSB International, Tallahassee, Florida: Florida State University Manning, L. (1996). Economics on the internet: Electronic mail in the classroom. Journal of Economic Education 27 (Summer); 201-204. Peluchette, J. V. & Rust K. A., (2005). Technology use in the classroom: Preferences of management faculty members. Journal of Education for Business. Robertson, H. J. (2005). Does size matter? Phi Delta Kappan, (November) Vol. 87, Issue 3: 251-254. Romer, D. (1993). Do students go to class? Should they? Journal of Economic Perspectives. (Summer), Vol. 7, No. 3: 167-174. Siegel, J. A. (2004). INTICE - Interactive Technology to Improve the Classroom Experience. Retrieved January 20,2007, from http://www.ph.utexas.edu/~ctalk/bulletin/intice.htm. Sosin, K., Blecha, B. J., Agarwal, R., Bartlett R. L., & Daniel, J. I. (2004). Efficiency in the use of technology in economic education: Some preliminary results. American Economic Review 94 (May): 253-58. Surry, D. W. (2000). Strategies for motivating higher education faculty to use technology. Innovations in Education and Training International, (May) Vol. 37, Issue 2, 145-154. Wood, W. (2004). Clickers: A Teaching Gimmick that Works. Developmental Cell. 7, 796-798. Developments in Business Simulation and Experiential Learning, Volume 35, 2008 196 http://www.ph.utexas.edu/%7Ectalk/bulletin/intice.htm Table of Contents Volume 35, 2008 Linking Team Covenants To Peer Assessment Of Simulation And Experiential Performance Using the Balance Scorecard Approach: A Group Exercise IndoAmerican Enterprises Class Size and Game Design Implementation Of Effective Experiential Learning Environments Simulation Sensemaking: The BusinessWeek Approach To Effective Debriefing A Lesson in Hide-And-Go-Seek: A Team Building Game Thoughts On How To Motivate Students Experientially Experiential Learning Is Not Just Experiential Teaching: Measurement of Student Skill Acquisition via Assessment Centers ABSEL Redux: Reflections after a 25 Year Hiatus College Student's Expectations of Technology- Enhanced Classrooms: Comparing 1996 and 2006 The Business Student Satisfaction Inventory (BSSI): Development and Validation of a Global Measure of Student Satisfaction The "Big Picture Question" Project: Explorations in Teaching Creativity within a Force-Field Research Framework "Viva Voce": Oral Exams as a Teaching & Learning Experience Internships And Occupational Socialization: What Are Students Learning? Does Learning Occur in One-Shot, Non-Cooperative Games? Beliefs and Behavior, an Ancient Perspective and Modern Application Developing Enterprise Culture Among the Students Through Intercollegiate Competitions: A Case of Student Enterprise Competition (SEC) 2007 Student Views of Management Skills and Their Future Careers after Using Business Simulations Back to the Future: Gender Differences In Self-Ratings of Team Performance Criteria A Case for Experiential Learning: Using Central Europe as a Classroom Modeling Strategic Opportunities in Product-Mix Strategy: A Customer- Versus Product-Oriented Perspective Assessment in the Modern Large High-Tech Classroom Target Profit Pricing With the Web-Based Breakeven Analysis Package Are the Business Simulations We Play Too Complex? Shared Experience as Incentive for Horizontal Integration in Business Simulations Affinity Propagation: A Clustering Algorithm for Computer-Assisted Business Simulations and Experiential Exercises Marketing Simulation Results as Embedded Forms of Program Assessment Human and Agent Playing the "Beer Game" Issues in Porting a LAN-based Total Enterprise Simulation Game to a Web-based Environment Partners or Competitors? A B2B Simulation Evaluation Model of the Global Performance of a Management Simulation for the Academic Environment Applying Bloom's Revised Taxonomy In Business Games Do Price Strategies Work in Business Simulations? Early Japanese Gaming Simulation Efforts Using Par Players to Enhance Learning in Business Simulations Corporate Cartooning: The Art of Computerized Business Simulation Design Should Business Game Players Choose Their Teammates: A Study with Pedagogical Implications Goal Orientation and Simulation Performance Design and Demonstration of an Online Managerial Economics Game with Automated Coaching For Learning and Graded Exercises for Assessment Are Good Strategy Decisions Consistently Good? A Real-Time Investigation Active Learning 2.0 or Wiki is not a 4-letter Word Comparing Student Learning in Online and Classroom Formats of the Same Course How Do We Get To Tomorrow? The Path to Online Learning Similar Media Attributes Lead to Similar Learning Outcomes Facilitating Business Gaming Simulation Modeling