503 service temporarily unavailable 503 service temporarily unavailable nginx/1.14.1 12746-rosales journal of the scholarship of teaching and learning, vol. 14, no. 5, december 2014, pp. 1 11. doi: 10.14434/josotlv14i5.12746   enhancing the impact of quality points in interteaching rocío rosales12, james l. soldner3, and william crimando4 abstract: interteaching is a classroom instruction approach based on behavioral principles that offers increased flexibility to instructors. there are several components of interteaching that may contribute to its demonstrated efficacy. in a prior analysis of one of these components, the quality points contingency, no significant difference was reported in student exam scores when quality points were made available. the purpose of the present study was to further evaluate the impact of the quality points component of interteaching, and to enhance the immediacy of feedback provided to students on this contingency via delivery of an answer key upon submission of post-discussion quizzes with the opportunity to review the answer key with their interteaching partner during class. we hypothesized that student quiz scores would be higher when this quality points contingency was in place. an alternating treatments design was employed to compare student performance on post-discussion quizzes during two conditions: quality points vs. no quality points with all other components of interteaching in effect. eleven undergraduate students enrolled in an introduction to applied behavior analysis course served as participants. results indicate average quiz performance was higher following class sessions with the quality points contingency. discussion is focused on the implications of delivering immediate feedback to students during classroom instruction and future directions of interteaching research. keywords: interteaching, quality points, feedback, college teaching interteaching is a deviation from traditional lecture, with one of its main components, the “interteach” defined by boyce and hineline (2002) as a “mutually probing, mutually informing conversation…” (p. 220). components of interteaching include the following: (1) preparation guides or “prep guides” which consist of 10-12 questions that cover roughly 10-15 pages of reading material, (2) pair discussions, conducted in class following the completion of each prep guide, (3) record sheets completed by students to provide feedback to the instructor and rate the overall quality of the pair discussion, (4) clarifying lectures designed to cover the most challenging topics based on feedback provided on the record sheets, (5) frequent test probes based on material from the prep guides to assess student learning, and (6) quality points, an explicit cooperative contingency whereby additional points are added to a student’s grade only if both students in a pair perform to a certain pre-determined criterion on a selected test probe. empirical support for the efficacy of interteaching in the college classroom, when compared to the traditional lecture, has been reported in previous studies. for example, saville, zinn, neef, van norman, and ferreri (2006) conducted two experiments to compare student                                                                                                                           1 department of psychology, university of massachusetts lowell, 113 wilder street, suite 300, lowell ma 01854-3059, rocio_rosales@uml.edu 2 author’s note: the first and second author’s contributions should be considered equal. 3 school for global inclusion and social development, university of massachusetts boston, 100 morrissey boulevard, boston, ma 02125 4 rehabilitation institute, southern illinois university carbondale, rehn hall, mail code 4609, carbondale, il 62901   rosales, r., soldner, j.l., & crimando, w. journal of the scholarship of teaching and learning, vol. 14, no. 5, december 2014. josotl.iu.edu 2 performance on short-answer quizzes following interteaching or traditional lecture. a total of 60 undergraduate students were recruited across the two studies. results showed consistently higher quiz scores following interteaching sessions when compared to lecture. arntzen and hoium (2010) reported similar results with a group of undergraduate students that self-rated their level of understanding and knowledge when interteaching was implemented compared to traditional lecture. overall, studies that have integrated interteaching into the classroom have consistently reported high academic achievement on homework and participation (filipiak, rehfeldt, heal, & baker, 2010; saville, pope, lovaas, & williams, 2012); or quiz and end-of-semester grades (cannella-malone, axe, & parker, 2009; lambert & saville, 2012; saville, et al., 2006; saville, pope, truelove, & williams, 2012). to date, only two published studies have included component analyses to identify the relative impact of each element of interteaching. for example, saville, cox, o’brien, and vanderbelt (2011) evaluated the impact of the clarifying lecture on the efficacy of interteaching. results of this study indicated that students who received lectures had consistently higher exam scores, but these differences were only statistically significant on two exams delivered throughout the semester. saville and zinn (2009) examined the extent to which quality points influenced test scores in an undergraduate research methods course. quality points are intended to enhance the value of pair discussions, overall student learning, and retention of material. when interteaching is incorporated, quality points are designed as a cooperative contingency to earn additional points (e.g., 8-10% of a final grade) if both students in a dyad score at or above a pre-determined level on test probes. saville and zinn (2009) implemented this contingency on a randomly assigned counterbalanced schedule across two sections of the same course. three quality points were awarded if both students scored at least 80% on a pre-selected essay question for each exam. results of this study indicated no statistically significant differences in exam scores. one potential factor impacting these results may be the delay in the delivery of feedback. specifically, students did not learn if quality points had been awarded until at least one week after completion of the exam. prior research has shown that the immediacy of feedback is an important indicator of performance across a variety of behaviors and settings (alvero, bucklin, & austin, 2001; codding, feinberg, dunn, & pace, 2005; renner, 1964). for example, solomon, klein, and politylo (2012) conducted a meta-analysis of the single-case literature and found that immediate feedback was effective at increasing teachers’ treatment integrity when a new intervention was introduced in the classroom. as additional evidence, scheeler, mcafee, ruhl, and lee (2006) found that the use of corrective, immediate feedback delivered via wireless technology was an effective way to increase completion of three-term contingency trials by teachers. these findings are consistent with past research that has shown a negative correlation between a delay of a contingency and its impact on performance (lattal, 1993). as a general rule: the sooner the delivery of feedback following desired behavior, the better (daniels & bailey, 2014). the immediacy of feedback may also impact student academic performance in the classroom. thus, the purpose of the present study was to further evaluate the impact of the quality points component of interteaching; and to enhance the immediacy of feedback provided to students on this contingency via delivery of an answer key immediately upon submission of postdiscussion quizzes with the opportunity to review the answer key with their interteaching partner. rosales, r., soldner, j.l., & crimando, w. journal of the scholarship of teaching and learning, vol. 14, no. 5, december 2014. josotl.iu.edu 3 method participants and materials eleven undergraduate students (1 male, 10 female; m age = 21) enrolled in one section of an introductory course to applied behavior analysis served as participants. students were of sophomore (3), junior (4) or senior (4) standing. students were not informed of the purpose of the study until the last day of the semester, at which time they were asked to review and sign a consent form that allowed use of their data. students were given the assurance that the instructor would not review the consent forms until final grades were submitted for the term. the class met face-to-face on mondays and wednesdays for 75 min. and was webenhanced (i.e., incorporated the use of blackboard learn to distribute materials to the students throughout the semester). the textbook adopted for the course was principles of everyday behavior analysis (miller, 2006). preparation guides, created by the instructor for the course (second author), were made available to students at least one week before they were expected to discuss the material with a classmate during class time. the prep guides reviewed 5 to 10 pages of material and included definition, conceptual, and applied questions (see appendix a for an example of questions included in the prep guides). dependent measure and experimental design an alternating treatments design was implemented whereby a quality points contingency was in effect via quasi-random assignment throughout the semester (quality points were not available during every class meeting). the availability of quality points was determined prior to the start of the semester by a coin flip with the constraint that each condition (quality points vs. no quality points) could occur for no more than two consecutive sessions. students were informed of the condition in place at the beginning of each pair discussion. that is, students were not aware if the contingency was in place until they began a pair discussion in class. the primary dependent variable for the study was average performance on 10-pt. weekly quizzes. quiz questions were based on the information presented in each prep guide and were created by adopting questions included as part of the textbook resources (see appendix b for a sample quiz). questions on each quiz required students to provide or apply a definition, recall information from the readings, or apply knowledge to novel examples. question format included fill-in-the-blank and short-answer. the short answer questions were objective and based on examples provided in the instructor resources of the text (e.g., correct responses consisted of writing numbers or 1-2 word verbatim responses from the text, see appendix b). a total of 17 quizzes were included as part of the data analysis, with a total of 34 possible quality points available throughout the semester. these points were considered extra credit and were added to the students’ overall grade at the end of the term. overall grades were determined by accumulating total quiz points and quality points, as well as various other required assignments (i.e., research article critique and book summaries) with objective point totals. procedure the general procedure for this study was similar to that reported by saville and zinn (2009). pair discussions were held for each prep guide followed by a post-discussion quiz; record rosales, r., soldner, j.l., & crimando, w. journal of the scholarship of teaching and learning, vol. 14, no. 5, december 2014. josotl.iu.edu 4 sheets were completed following each pair discussion; and a clarifying lecture was delivered based on feedback provided by the students during the next class session. students were allowed to select and work with one partner of their choosing, but were instructed to work with a different classmate during each pair discussion. if an odd number of students was present in class on any given day, one group was comprised of three students. during the time allotted for discussion (25-30 min.), the instructor walked around the room to answer questions and monitor discussions to ensure students stayed on topic. students received participation points for being present and prepared to discuss their prep guide with a classmate. participation points comprised 8% of the students’ final grade. following each pair discussion, students were provided with approximately 5 min. to complete a record sheet intended to provide feedback to the instructor on the quality of the discussion, and to list the topics that presented the most difficulty. quizzes were administered following completion of the record sheets. the instructor used the information from the record sheets to create a clarifying lecture that was presented at the beginning of the next class session and lasted approximately 25 min. the cooperative contingency in place for the post-discussion quizzes in this study was as follows: if both students in a dyad received a quiz score of 80% (i.e., eight out of ten correct responses) or higher on their respective quiz, then two points were awarded for that particular quiz to each student. if either student received a score below 80%, then neither received quality points for that quiz. quality points were calculated into the students’ overall grade as bonus points. no other extra credit point opportunities were made available as part of the course. the quality points contingency was described in the course syllabus and the instructor reviewed this with the students at the beginning of the semester. immediate feedback was provided on the students’ performance following every quiz regardless of the contingency in place. specifically, the instructor provided the student with an answer sheet upon submission of each quiz. upon receipt of the answer key, students had immediate information on their own performance and also had the opportunity to meet with their discussion partner (either in the classroom or in the hallway outside of the classroom) to discuss individual quiz performance and determine if the contingency for that respective quiz had been met. results the data analysis includes only scores for quizzes that were administered during class time immediately following pair discussions as described above. that is, if students missed class and subsequently received a score of ‘0’ for the quiz, these grades were omitted for the purpose of the present study. if students missed class, but took a make-up quiz at a later time, these scores were also omitted. the quality points contingency was in effect for a total of 10 in-class quizzes administered throughout the semester, while the absence of quality points was in effect during seven quizzes. results of average quiz performance for the 11 students in the course are depicted in figure 1. these results indicate overall average student performance was better when the cooperative contingency was in effect (m = 7.75, sd = 0.52 with quality points, compared to m = 7.05, sd = 0.33 with no quality points available). overall, average scores were indicative of a difference in letter grade (e.g., ‘c+’ with quality points contingency compared to ‘c-’ without quality points contingency). overall student performance tended to decrease as the semester rosales, r., soldner, j.l., & crimando, w. journal of the scholarship of teaching and learning, vol. 14, no. 5, december 2014. josotl.iu.edu 5 progressed. this may be accounted for by the fact that the information presented in the course was cumulative in nature. that is, after the basic principles of behavior analysis were presented and discussed, students were required to have a general understanding of these principles in order to perform well on subsequent chapter probes (e.g., differential reinforcement of behavior or shaping). although the quizzes were not cumulative per se, the information covered in each subsequent chapter may have required students to retain information from previous chapters, resulting in slightly more difficult quizzes as the semester progressed. figure 1. average performance across all post-discussion quizzes. to investigate whether the differences between students' performance under the quality points contingency and absent the contingency were significant, a paired t-test was computed, using the means of all students under both conditions as scores (m = 7.75, with quality points, m = 7.05 with no quality points). since means comparisons are sensitive to extreme outliers, mean differences were converted to z scores in a preliminary analysis were and then examined to determine if any of them exceeded 3 standard deviations either side of the grand mean. the largest z score was -2.4, indicating the absence of extreme outliers. thus we proceeded with the paired t test. the difference was significant, t10 = 3.29, p = .008. individual average quiz scores are presented in figure 2. these results reveal that 10 of the 11 participants scored higher on quizzes when quality points were available. the distribution of final course grades resulted in 6 students (55%) with a final course grade of a, 3 students (27%) with a final course grade of b, 1 student (9%) with a final course grade of c, and 1 student (9%) with a final course grade of d. 0 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 a ve ra ge  s co re quiz quality  points no  quality  points m =  7.75 sd =  0.52 m =  7.05 sd =  0.33 rosales, r., soldner, j.l., & crimando, w. journal of the scholarship of teaching and learning, vol. 14, no. 5, december 2014. josotl.iu.edu 6 figure 2. individual mean scores across participants for quality vs. no-quality points condition. discussion the present results provide support for the use of quality points when interteaching is implemented in the classroom. although the term ‘quality points’ was coined by boyce and hineline (2001), the use of a cooperative contingencies have been applied in classrooms through the use of strategies such as reciprocal peer tutoring (bowman-perrot et al., 2013) and other related paradigms. students in the present study received immediate feedback on all quizzes. previous research evaluating the overall effectiveness of performance feedback has revealed that delayed feedback has less impact on performance than immediate feedback (codding, feinberg, dunn, & pace, 2005; daniels & bailey, 2014; lattal, 1993). the immediacy of feedback in the present study may have enhanced the effect of quality points on quiz performance and subsequently, student learning. alternatively, immediate feedback may have enhanced performance throughout the semester, and not just when quality points were available. although results indicate average quiz scores were higher when the quality points contingency was in effect, these results should be interpreted with caution given some of the limitations of the study. first, the availability of quality points was not counterbalanced with a second class section. therefore, it is possible that the content of the quizzes was not equal across sessions. this presents a potential extraneous variable since students may have found material more difficult in class sessions when the quality points contingency was not available. for this reason, systematic replications of this procedure are needed. second, although we provided students with the opportunity to review answer keys with their interteaching partner, this was not a requirement of the course. therefore, it is unclear if the immediacy of feedback for cooperative performance made a significant difference only during the quality points contingency. future research on both the availability of quality points and immediacy of feedback for performance are needed before any conclusive statements are made with respect to this component of interteaching. 0 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10 11 a ve ra ge  q ui z   sc or e participant quality  points no  quality  points 11                                  2                                  7                                    6                                      1                                  4                                  9                     10                                8                                    3                                    5 rosales, r., soldner, j.l., & crimando, w. journal of the scholarship of teaching and learning, vol. 14, no. 5, december 2014. josotl.iu.edu 7 third, quizzes in the present study covered relatively little information (e.g., 10-15 pages of reading) compared to the exam probes that have covered more than one topic in previous studies (saville & zinn, 2009). questions included in the probes for the present study were primarily informational and may not have required higher level comprehension or critical thinking. these types of questions were generally reserved for review exams that were administered four times throughout the semester for students in the present study. fourth, the quality points in the present study were provided as extra credit. boyce and hineline (2002) recommend that quality points account for 10% of students’ overall course grade. the decision to incorporate quality points only as bonus was made based on anecdotal reports from students indicating they experienced less stress and anxiety when final course grades were not adversely impacted by a peer’s performance (e.g., when the quality points contingency was not in effect). future studies should evaluate the relative impact of quality points when the contingency is incorporated as part of students’ final grade or as an opportunity for extra credit; and also determine if the value of quality points impacts performance during probes (e.g., by making the overall percentage of quality points higher on each probe).   fifth, class size in the present study was relatively small. although some support exists for the effectiveness of interteaching with large class sections (scoboria & pascual-leone, 2009), comparison studies of small vs. large class size have yet to be conducted. students in smaller class sections typically have the advantage of receiving more individual time from the instructor. individual attention from the instructor may have impacted student performance in the present study, but this variable was held constant (in effect during both conditions) and should not have impacted the results. finally, considering the relatively high distribution of final course grades (9 of 11 students [82%] received a grade of either a or b), it is possible that the decline in average quiz performance for the last quiz of the semester (#17) was due in part to student knowledge of high overall course grades. as a result, the perceived value of individual and overall quiz performance and quality points may have decreased at the end of the semester. given the limited data reported to date on the impact of each individual element of interteaching, future studies should continue to conduct component analyses (e.g., pair discussion, record sheets, etc.). future research should manipulate the value of quality points to determine if there is a difference in performance when these points are offered as extra credit or as part of the overall course grade. the quality points contingency could also be set at a higher level for each test probe, or offered for a single question that involves the use of critical thinking. in addition, future research may enhance the notification of feedback and attainment of quality points in an immediate way through on-line notification or delivery of quizzes on-line with immediate feedback on performance. finally, direct and systematic replications of these procedures in distance education and on-line courses will help to further provide evidence for the robustness of this teaching paradigm. rosales, r., soldner, j.l., & crimando, w. journal of the scholarship of teaching and learning, vol. 14, no. 5, december 2014. josotl.iu.edu 8 appendices appendix a. sample quiz from principles of everyday behavior analysis (4th ed.) by l. keith miller, pp. 507. © 2006 toronto: thomson wadsworth. reprinted with permission. rosales, r., soldner, j.l., & crimando, w. journal of the scholarship of teaching and learning, vol. 14, no. 5, december 2014. josotl.iu.edu 9 appendix b. sample preparation guide. preparation guide #2 pair discussion on: monday, january 31st based on: miller: lessons 2 & 3 based on: lesson 2 1. what is behavior? what are the four requirements for explaining behavior? does behavior refer only to events or activities that are observable? explain your answer, and give some examples of behavior that are unobservable. which of the following would be examples of behavior: (a) running, (b) thinking, (c) perception, (d) memory, (e) eating, (f) emotion, (g) glancing, (h) knowledge, (i) dating, and (j) personality? what seems to distinguish behavior from “non-behavior”? 2. do behavior analysts deny that thinking exist? why or why not? is thinking a public behavior, a private behavior, or both? what is one way that “thinking” could be observed and measured, in other words considered a behavior? 3. what is a behavioral definition? what is the importance of included and excluded behavior when constructing a behavioral definition? what are the two benefits of clear behavioral definitions? what are some possible explanations for inaccurate behavioral definitions? 4. what are self-reports? what are common forms of self-reports? what are some potential problems with self-reports? why do behavior analysts avoid self-reports? what type of behavioral data do behavior analysts prefer? 5. what is the principle of direct observation? what are some key differences between selfreports and direct observations? is memory and recall of behavior necessary with direct observation? why or why not? which method of behavioral data collection is the most accurate, self-reports or direct observations? based on: lesson 3 6. what are considered “uniform” behaviors? what are “nonuniform” behaviors? what are some examples of both uniform and nonuniform behaviors? 7. what is outcome recording? when is outcome recording useful? what are some examples of behavior that leave a “result” behind? could outcome recording to be used to measure snow shoveling behavior? why or why not? 8. what is event recording? when is event recording used? can event recording be used for simple behavior? complex behaviors? why or why not? 9. what is an interval type method of observation? what are some novel examples of nonuniform behaviors in which interval recording could be used? what are continuous intervals when using interval recording? 10. what are the differences between interval and outcome recording? what are the main aspects of interval recording, in particular? 11. what is time sample recording? what type of interval is a time sample recording method most often used and why? what is an example behavior in which time sample recording would be appropriate? rosales, r., soldner, j.l., & crimando, w. journal of the scholarship of teaching and learning, vol. 14, no. 5, december 2014. josotl.iu.edu 10 references alvero, a. m., bucklin, b. r., & austin, j. (2001). an objective review of the effectiveness and essential characteristics of performance feedback. journal of organizational behavior management, 21(1), 3-29. arntzen, e., & hoium, k. (2011). on the effectiveness of interteaching. the behavior analyst today, 11, 155-160. bowman-perrott, l., davis, h., vannest, k., williams, l., greenwood, c., & parker, r. academic benefits of peer tutoring: a meta-analytic review of single-case research. school psychology review, 42(1), 39-55. boyce, t. e., & hineline, p. n. (2002). interteaching: a strategy for enhancing the user friendliness of behavioral arrangements in the college classroom. the behavior analyst, 25, 215– 226. daniels, a. c., & bailey, j. s. (2014). performance management: changing behavior that drives organizational effectiveness (5th ed.). atlanta, ga: performance management publications. lattal, k. a. (1993). delayed reinforcement of operant behavior. journal of the experimental analysis of behavior, 1, 129-139. miller, l. k. (2006). principles of everyday behavior analysis (4th ed.). toronto: thomson wadsworth. renner, k. e. (1964). delay of reinforcement: a historical review. psychological review, 61, 341-361. roscoe, e. m., fisher, w. m., glover, a. c., & volkert, v. m. (2006). evaluating the relative effects of feedback and contingent money for staff training of stimulus preference assessments. journal of applied behavior analysis, 39, 63-77. doi:10.1901/jaba.2006.7-05 saville, b. k., cox, t., o’brien, s., & vanderveldt, a. (2011). interteaching: the impact of lectures on student performance. journal of applied behavior analysis, 44, 937-941. doi:10.1901/jaba.2011.44-937 saville, b. k., & zinn, t. e. (2009). interteaching: the effects of quality points on exam scores. journal of applied behavior analysis, 42, 369-374. doi:10.1901/jaba.2009.42-369 saville, b. k., zinn, t. e., neef, n. a., van norman, r., & ferreri, s. j. (2006). a comparison of interteaching and lecture in the college classroom. journal of applied behavior analysis, 39, 49-61. doi:10.1901/jaba.2006.42-05 rosales, r., soldner, j.l., & crimando, w. journal of the scholarship of teaching and learning, vol. 14, no. 5, december 2014. josotl.iu.edu 11 scheeler, m. c., mcafee, j. k., ruhl, k. l., lee, d. l. (2006). effects of corrective feedback delivered via wireless technology on preservice teacher performance and student behavior. teacher education and special education, 29(1), 12-25. scoboria, a. & pascual-leone, a. (2009). an ‘interteaching’ informed approach to instructing large undergraduate classes. journal of the scholarship of teaching and learning, 9, 29-37. solomon, b. g., klein, s. a., & politylo, b. c. (2012). the effect of performance feedback on teachers’ treatment integrity: a meta-analysis of the single-case literature. school psychology review, 41(2), 160-175. 503 service temporarily unavailable 503 service temporarily unavailable nginx/1.14.1 503 service temporarily unavailable 503 service temporarily unavailable nginx/1.14.1 503 service temporarily unavailable 503 service temporarily unavailable nginx/1.14.1 503 service temporarily unavailable 503 service temporarily unavailable nginx/1.14.1 503 service temporarily unavailable 503 service temporarily unavailable nginx/1.14.1 503 service temporarily unavailable 503 service temporarily unavailable nginx/1.14.1 503 service temporarily unavailable 503 service temporarily unavailable nginx/1.14.1 journal of the scholarship of teaching and learning, vol. 17, no. 4, october 2017, pp. 31-43. doi: 10.14434/josotl.v17i4.21763 assessing the development of civic mindedness in a cohort of physical therapy students kerstin m. palombaro, pt, phd, caps widener university kpalombaro@widener.edu jill d. black, pt, dpt, edd widener university robin l. dole, pt, dpt, edd, pcs widener university jessica l pierce, pt, dpt powerback rehabilitation marisa r. santiago, pt, dpt easter seals edward j. sabara, pt, dpt wallace and nilan physical therapy and fitness mailto:kpalombaro@widener.edu palombaro, black, dole, pierce, santiago, and sabara journal of the scholarship of teaching and learning, vol. 17, no. 4, october 2017. josotl.indiana.edu 32 abstract introduction and background colleges and universities have an obligation to educate graduate students who demonstrate civic literacy and who can engage in civic inquiry and action. servicelearning and community engagement courses are highly effective in developing civic-mindedness. graduate professional programs may have additional reasons for focusing on development of civic mindedness. the purpose of this paper is 1) to measure the development of civic-mindedness increased in a cohort of physical therapy students exposed to a variety of civic engagement programs and servicelearning courses and 2) to determine if there is a difference in civic-mindedness between students serving on a pro bono clinic student board and their classmates in a graduate physical therapy program. methods the civic-minded professional scale (cmp) is a 23-item, 7-point likert-scale survey designed to measure the domains of self-identity; work, career and profession; and civic attitudes, civic action, and public purpose to measure the construct of civic-mindedness. the cmp was administered to a cohort of graduate physical therapy students at the beginning of the professional curriculum and at the end of each of three years of didactic coursework. results friedman’s repeated measures anova with post hoc testing revealed that civic mindedness increased in our students throughout the course of the graduate physical therapy curriculum. analysis also supports a difference on civic mindedness between those that participated in leadership of a pro bono clinic compared with students who did not. discussion and conclusion students involved in service-learning coursework imbedded throughout a graduate physical therapy program had increases in civic-mindedness. servicelearning coursework may be an effective way to develop civic mindedness in graduate students. key words: civic mindedness, civic minded professional scale, servicelearning palombaro, black, dole, pierce, santiago, and sabara journal of the scholarship of teaching and learning, vol. 17, no. 4, october 2017. josotl.indiana.edu 33 colleges and universities have an obligation to educate graduate students who demonstrate civic literacy and who can engage in civic inquiry and action (national task force on civic learning and democratic engagement, 2012). civic-mindedness refers to a quality of someone who is knowledgable and committed to a community and who values his or her role as a member of that community (pike, bringle, & hatcher, 2014). employers value a workforce that is highly skilled, while at the same time is high in the quality of civic-mindedness (torney-purta, cabrera, roohr, liu, & rios, 2015). a civic-minded professional is someone with professional skills acquired through formal education who acts ethically and can work collaboratively with others to achieve the common good (bringle & steinberg, 2010). thus one goal of post-secondary education should be to foster civic-mindedness in their students so that they can function effectively in the 21st century workforce. service-learning and community engagement courses are highly effective in developing civic-mindedness (crandall, wiegand, & brosky, 2013; smith, yoon, johnson, natarajan, & beck, 2014; steinberg, hatcher & bindle, 2011). the definition of service learning is “a course-based educational experience in which students: a) participate in an organized community service activity that meets identified community needs and b) reflect on the service activity in such a way as to gain further understanding of course content, a broader appreciation of the discipline and an enhanced sense of civic responsibility” (bringle & hatcher, 1995). service-learning courses provide educationally meaningful community service with a focus on reciprocal, democratic partnerships (saltmarsh & hartley, 2011). in this way, both the community partners and the students learn from one another and work together from a place of mutual respect. service-learning thus can both foster skills in a particular field and can also serve to support the development of a sense of civic responsibility (bringle & steinberg, 2010). graduate professional programs may have additional reasons for focusing on development of civic mindedness. physical therapy programs are one such example. the american physical therapy association (apta) has defined seven core values for the physical therapist. the first, “accountability” acknowledges the diverse roles of the physical therapist that impact our patients, the profession and society (apta, 2010). the second, "altruism" calls for physical therapists to put patients' needs above their own (apta, 2010). the third, "compassion/caring" identifies compassion as a precursor to caring and caring as "the concern, empathy, and consideration for the needs and values of others” (apta, 2010). the fourth, "excellence" values among other things the acquisition of new knowledge (apta, 2010). the sixth “professional duty” is meeting professional obligations to serve patients, the profession and society (apta, 2010). the seventh, “social responsibility,” looks towards promoting mutual trust between physical therapists and the patients they serve (apta, 2010). all core values have sample indicators associated with them that serve as further expressions of civic engagement. (table 1) the apta code of ethics defines eight principles that are tied to the code of ethics that call physical therapists to behave professionally and ethically, to become lifelong learners as they refine their skills and work towards meeting the healthcare needs of society (apta, 2006). the accreditation criteria put forth by the commission for the accreditation of physical therapy education (capte) delineates the knowledge and skill acquisition required to practice as an entry-level clinician at licensure and requires that students practice physical therapy in a manner consistent with the apta core values and code of ethics (capte, 2015). an additional criterion states that students should “participate in professional and community organizations that provide opportunities for volunteerism, advocacy, and leadership” (capte, 2015). palombaro, black, dole, pierce, santiago, and sabara journal of the scholarship of teaching and learning, vol. 17, no. 4, october 2017. josotl.indiana.edu 34 table 1 selected apta core values and some associated sample indicators (capte, 2015). core value sample indicators accountability participating in the achievement of health goals of patients/clients and society. altruism providing pro bono services providing physical therapy services to underserved and underrepresented populations. providing patient/client services that go beyond expected standards of practice compassion/caring being an advocate for the patient’s/client’s needs. recognizing and refraining from acting on one’s social, cultural, gender and sexual biases. demonstrating respect for others and considers others as unique and of value. excellence demonstrating an investment in the profession of physical therapy. demonstrating high levels of knowledge and skill in all aspects of the profession. pursuing new evidence to expand knowledge. professional duty facilitating each individual’s achievement of goals of function, health, and wellness social responsibility advocating for the health and wellness needs of society including access to health care and physical therapy services. promoting cultural competence within the profession and the larger public. promoting community volunteerism palombaro, black, dole, pierce, santiago, and sabara journal of the scholarship of teaching and learning, vol. 17, no. 4, october 2017. josotl.indiana.edu 35 participating in political activism. the widener university institute for physical therapy education (ipte) in chester, pa admits one cohort each summer for a three-year doctor of physical therapy graduate education program. all students participate in several community engagement programs designed to address the apta code of ethics and core values statements as well as the capte criteria (apta, 2006; apta, 2010; capte, 2015). students in their first year participate in monthly blood pressure screens at a local low-income senior residence building and in a geriatric mobility screen as part of their lifespan i: adulthood class (palombaro, black, & campbell, 2014). students in their first and second year participate in our mlk day of service blood pressure and mobility screens at sites throughout chester, pa (lattanzi, campbell, dole, & palombaro, 2011). students in their second year of study participate in brain safety fairs for children that provide brain safety education and a properly fitting bike helmet (pierce, palombaro, & black, 2016; pierce, palombaro, & black, 2014). all students participate several times each semester in the chester community physical therapy clinic, a student-led pro bono clinic that provides physical therapy services to uninsured and underinsured residents of chester and the surrounding communities (black, palombaro, & dole, 2013; palombaro, dole, & lattanzi, 2011; palombaro, dole, & lattanzi, 2011b). some students apply to serve as student board members for their three years of education; the student board is responsible for the day-to-day operations of the clinic (black et al., 2013; palombaro et al., 2011; palombaro et al., 2011b). student board members take on additional responsibility as compared to their classmates. those who do not apply or are not selected to become board members are assigned to one of four sites in the chester community as part of the community health practicum, providing weekly physical activity programming to either children or adults (palombaro, lattanzi, & dole, 2010; palombaro et al., 2011). students participating in the community health practicum design, implement and evaluate physical therapy programming in their second year of education, with assistance from students in their first and third years of study (palombaro et al., 2010). the ipte desired to measure the development of civic-mindedness in its students in order to provide evidence that our graduates were increasing in the attributes delineated by the apta core values. the purpose of this paper is 1) to measure the development of civicmindedness in a cohort of physical therapy students exposed to a variety of civic engagement programs and service-learning courses and 2) to determine if there is a difference in civicmindedness between student board members and their classmates. methods participants participants were 39 members of a cohort of graduate physical therapy students at the ipte at widener university. this cohort was part of a three-year curriculum that graduated in the spring of 2016. the civic minded professional scale palombaro, black, dole, pierce, santiago, and sabara journal of the scholarship of teaching and learning, vol. 17, no. 4, october 2017. josotl.indiana.edu 36 the measurement instrument was the civic-minded professional scale (cmp), a tool designed to measure the domains of self-identity; work, career and profession; and civic attitudes, civic action, and public purpose to measure the construct of civic-mindedness (hatcher, 2008; hatcher, manuscript in preparation). it is a 23-item, 7-point likert-scale survey with reported reliability and validity (hatcher, 2008; hatcher, manuscript in preparation) and is a shortened form the original 44-item version. the scale ranges from strongly disagree to agree and total scores range from 23 to 161. the scale measures the five factors of voluntary action with six items related to volunteer activity; identity and calling with five items measuring satisfaction and work identity; citizenship with four items measuring participation in civic events; trustee of knowledge with four items related to the an individual’s valuing of education and professional expertise; and consensus across difference with four items measuring working with individuals from diverse backgrounds (hatcher, manuscript in preparation). steinberg, hatcher and brindle (2011) provided evidence from the literature in support of the connection between civic mindedness and high impact academic practices including service-learning, as they laid the conceptual foundation for the model of the civic-minded graduate. their work identified three inter-related components that are central to this model: identity, educational experiences, and civic experiences. each of these components can also be found in items of the civic-minded professional scale and within the various curricular activities designed to promote civic-mindedness in this study. survey methodology institutional review board at widener university approved this research project. all students in the cohort of the class of 2016 were required to complete the cmp at the beginning of their first professional year of study and at the end of the first, second, and final years of the didactic portion of the physical therapy curriculum. the ipte requires completion of the cmp as part of its program outcomes. however, all students were provided with a letter of informed consent. the letter explained that the students could opt-in to the research study that would allow their cmp scales to be used as part of the project. students were provided this letter at each point in the data collection and reminded that they could opt out at any time. all students elected to opt-in to the research portion of the survey. students who opted in to the research study were provided with a unique identifying code known only to one researcher (jdb). this code was placed on the front of each survey at each data collection point. surveys for each student were placed into an envelope with the student’s name on the envelope. students removed their survey from the envelope and returned their survey only to another researcher (kmp) for input into a database. at the initial point of data collection, students also completed basic demographic information of age, sex, race and year of graduation. data analysis data were analyzed using spss version 20. descriptive statistics were performed on demographic data and on the civic-minded professional scale scores for each data collection point. a friedman’s anova with wilcoxon signed ranks test post-hoc testing were performed on each data collection point for the entire class to determine if there were within-group differences for each time point. mann-whitney u tests were performed on cmp data to determine if betweengroup differences existed for student board and non-student board members. effect sizes were palombaro, black, dole, pierce, santiago, and sabara journal of the scholarship of teaching and learning, vol. 17, no. 4, october 2017. josotl.indiana.edu 37 calculated for all statistically significant results using the formula for cohen’s d (d= x1-x2/sp) (cohen, 1988). results thirty-nine students comprised the completed cohort for the class of 2016. of those 39, 2 students had incomplete data on one or more surveys and were excluded from the analysis. descriptive statistics were run to provide a more complete picture of the cohort. the mean age was 22.4 (sd=1.9) upon entering the program. eleven members were males and 26 were female. the cohort was comprised of 36 caucasians and one person who identified as hispanic/latino. data for the cmp is presented as means and standard deviations because the likert-scale provided a summative score and the data were normally distributed (hatcher, manuscript in preparation). non-parametric statistics were run due to the sample being one of convenience. friedman's repeated-measures anova was significant (p = .001) for between-group differences. wilcoxon signed ranks post hoc tests were performed. significant increases in the cmp were found between pre-test of year 1 and the post-test of year 2 (p = .016, cohen’s effect size value d = .48), the pretest of year 1 and the post-test of year 3 (p ≤ .001, cohen’s d = .70), the post-test of year 1 and the post-test of year 2 (p = .008, cohen’s d = .53) and the post-test of year 1 and the post-test of year 3 (p ≤ .001, cohen’s d = .70). the effect sizes ranged from .48-.70 suggesting moderate to high practical significance (cohen, 1988). (table 2) table 2 descriptive statistics for the civic-minded professional scale for the entire class at each assessment point. test time mean sd pre-test year 1 120.5 13.8 post-test year 1 119.9 12.8 post-test year 2 127.6 11.9 post-test year 3 130.2 12.0 mann-whitney u testing was performed to determine if there were differences between student board and non-student board members at each assessment point. a significant difference was found for the year three post-test (p = .04, cohen’s d = .58). the effect size suggests moderate practical significance (cohen, 1988). (table 3.) palombaro, black, dole, pierce, santiago, and sabara journal of the scholarship of teaching and learning, vol. 17, no. 4, october 2017. josotl.indiana.edu 38 table 3 means and standard deviations for student board and non-student board members at each assessment point test time student board non-student board mean sd mean sd pre-test year 1 119.10 11.45 121.12 14.79 post-test year 1 117.90 9.17 120.76 14.12 post-test year 2 129.20 11.46 126.92 12.28 post-test year 3 134.80 9.10 128.36 12.65 discussion our results indicate that civic mindedness increased in our students throughout the course of the graduate physical therapy curriculum. we propose that this increase may be due to the students’ exposure to service-learning coursework and programming that was directly tied to skills needed to become a physical therapist, although other factors could also influence this change. community-based service learning is a structured experience with defined learning objectives, effective preparation and reflection. community-based service learning places a focus on development of citizenship and reciprocal learning on the part of the students and the community partner (seifer, 1998). all service-learning activities at the ipte require that the students interact with community members in meaningful ways. they serve to interview and educate community members. within the geriatric screens and the mlk day of service, they are making direct application of specific physical therapy screenings. the community health practicum requires that the students educate participants on the importance of physical activity to maintain or improve health. finally, as the students participate in the student-run pro bono clinic, they are participating in client care to the extent that their skills allow. for example, first-year students take vital signs and conduct client interviews. they begin to participate in interventions regarding mobility and exercise. second year students begin to conduct evaluations with newly acquired assessment and documentation skills. third year students apply more complex manual and neuromotor skills as well as supervise second and first year students in client programs. this all occurs under the direct supervision of licensed physical therapists. hence, all of the service-learning and community engagement activities within the ipte engage the students in meaningful physical therapy-related skills and interactions with actual community members. as they do so, they are developing relationship and rapport with the community members and gaining a perspective on the overall impact that culture, lifestyle, and community have on the physical therapy encounter. hence, service-learning may be a powerful experience for fostering civic-mindedness. service-learning coursework may foster professional skills development (brosky, deprey, hopp, & maher, 2006). gupta (2006) found that students who participate in service learning are more likely to continue to perform service after graduation. in a sample of medical students participating in a student-run free clinic, students gained greater knowledge and skills, and improved their attitude towards underserved patient populations and developed greater interest in working with this patient population (smith, yoon, johnson, natarajan, & beck, 2014). qualitative research examining student perceptions after working in a pro bono physical therapy palombaro, black, dole, pierce, santiago, and sabara journal of the scholarship of teaching and learning, vol. 17, no. 4, october 2017. josotl.indiana.edu 39 clinic uncovered themes of core values, development of clinical physical therapy skills, professional growth and professional connections (stickler, grapczynski, & ritch, 2013). crandall, et al. (crandall, wiegand, & brosky, 2013) studied 91 dpt students involved in service-learning courses through interviews, examination of reflection papers and the apta core values self-assessment survey. the survey identified themes of accountability and compassion/caring. student interviews identified the core value of compassion/caring as the strongest with accountability, altruism and professional duty also occurring frequently. the reflection papers yielded themes of compassion/caring, social responsibility, and professional duty. findings from wise and yuen (2013) examined the impact of community-based service-learning on dpt students. students demonstrated a positive trend on all core values with significant increases in altruism, compassion/caring, and integrity. while the present study used the cmp versus the apta core values self-assessment survey, it is our belief that apta core values are elements of civic-mindedness. the present study corroborates the evidence that participation in pro bono service and service-learning classes appears to promote the development of civicmindedness in graduate physical therapy students. overall, students who had taken on additional responsibilities of overseeing the pro bono clinic operations did not demonstrate significantly higher civic-mindedness as compared to other students at the beginning of the curriculum or at the end of their first and second year. this signifies that students who were attracted to student board membership were not significantly different from their non-student board peers at the beginning of the curriculum and that the first and second year experiences did not differ for these students in terms of civic mindedness development. the authors see this as a positive and important finding. the student board is one means to improving civic mindedness, but is not the only means within the ipte curriculum. students take on leadership in the community through the community health practicum or the student board and all students serve in the clinic as student physical therapists. these findings show that while students experience different opportunities to develop civic-mindedness, overall these opportunities created similar outcomes. student board members did exhibit significantly higher civic-mindedness at the end of the third year of the didactic curriculum. the authors believe this may be due to the increased exposure student board members have to making professional presentations in their third year at the apta national student conclave, the pennsylvania physical therapy association conference, apta combined sections meeting and the student run free clinic conference, although first and second-year student board members begin to participate in these experiences as well. bringle et al (2011) define a key element of a civic minded graduate as having listening and communication skills, which includes activities such as spoken communication as well as letter writing and professional presentations. other disciplines such as business (tucker & mccarthy, 2001), pharmacy, (osborne, hammerich, & hensley, 1998) and conservation biology (clark, 2001) describe professional presentations as one area that can assist in the development of civic mindedness. for example, clark (2011) states that aligning educational programs with the society for conservation biology objectives to increase research and professional dissemination of results will foster civic mindedness in graduates. tucker and mccarthy (2001) found increased confidence in presentation ability was significantly related to business students enrolled in a service-learning course as compared to students enrolled in a traditional section of the same course. with these findings in mind, all students should be given opportunities for professional presentations in the coming years. palombaro, black, dole, pierce, santiago, and sabara journal of the scholarship of teaching and learning, vol. 17, no. 4, october 2017. josotl.indiana.edu 40 this study had several limitations. the first is that we cannot control for experiences outside of the service learning experiences that may have increased civic-mindedness such as nonprofessional volunteer work (e.g. being a big brother/big sister, coaching etc). additionally, this is representative of a single cohort of students and for which no control group was available for comparison. lack of a control to understand the natural progression of civic mindedness in dpt students who do not experience service learning and civic engagement limits the interpretation of these findings. research and replication of this work with future classes across the curriculum and the potential addition of professional presentation experiences for all students may also provide improved clarity on the present study’s finding. finally, the cmp is a self-report scale. selfassessment is a skill that is developed as part of a physical therapy curriculum (musolino, 2006), thus self-assessment may become more reliable at each measurement point. conclusion in conclusion, service-learning coursework and community engagement experiences imbedded throughout a graduate physical therapy program may be an important factor in increasing in student civic-mindedness. the students in this study demonstrated consistent increases in civicmindedness as measured by the civic-minded professional scale. students participating in leadership experiences related to a student-run pro bono clinic had changes in civic-mindedness that significantly exceeded the changes exhibited by students who did not engage in those additional opportunities. acknowledgement the authors wish to acknowledge grant support from the widener university faculty development and provost grant programs. palombaro, black, dole, pierce, santiago, and sabara journal of the scholarship of teaching and learning, vol. 17, no. 4, october 2017. josotl.indiana.edu 41 references apta. code of ethics. (2006). (no. hod s06-09-07-12). alexandria, va: american physical therapy association. retrieved from http://www.apta.org/ethics/core/ apta. (2010). professionalism in physical therapy: core values. retrieved from http://www.ptcas.org/professionalism/ black, j. d., palombaro, k. m., & dole, r. l. (2013). student experiences in creating and launching a student-led physical therapy pro bono clinic: a qualitative investigation. physical therapy, 93(5), 637-648. https://doi.org/10.2522/ptj.20110430 bringle, r. g., & hatcher, j. a. (1995). a service-learning curriculum for faculty. michigan journal of community service learning, 2, 112-122. bringle, r. g., & steinberg, k. (2010). educating for informed community involvement. american journal of community psychology, 46 (3-4), 428-441. https://doi.org/10.1007/s10464010-9340-y bringle, r. g., studer, m., wilson, j., clayton, p. h., & steinberg, k. (2011). designing programs with a purpose: to promote civic engagement for life. journal of academic ethics, 9, 149-164. https://doi.org/10.1007/s10805-011-9135-2 brosky, j. a., deprey, s. m., hopp, j. f., & maher, e. j. (2006). physical therapist student and community partner perspectives and attitudes regarding service-learning experiences. journal of physical therapy education, 20(3), 41-48. clark, t. w. (2001). developing policy-oriented curricula for conservation biology: professional and leadership education in the public interest. conservation biology, 15(1), 31-39. https://doi.org/10.1111/j.1523-1739.2001.99007.x cohen j. (1988). statistical power analysis for the behavioral sciences. hillsdale, nj: erlbaum associates. commission on accreditation in physical therapy education (capte). pt standards with evidence. (2015). retrieved from http://www.capteonline.org/accreditationhandbook/ crandall, c. e., wiegand, m. r., & brosky, j. a. (2013). examining the role of service-learning on development of professionalism in doctor of physical therapy students: a case report. journal of allied health, 42(1), e25-e32. hatcher, j. a. developing and evaluating the civic-minded professional scale. manuscript in preparation. hatcher, j. a. (2008). the public role of professionals: developing and evaluating the civic http://www.ptcas.org/professionalism/ https://doi.org/10.2522/ptj.20110430 https://doi.org/10.1007/s10464010-9340-y https://doi.org/10.1007/s10805-011-9135-2 https://doi.org/10.1111/j.1523-1739.2001.99007.x http://www.capteonline.org/accreditationhandbook/ palombaro, black, dole, pierce, santiago, and sabara journal of the scholarship of teaching and learning, vol. 17, no. 4, october 2017. josotl.indiana.edu 42 minded professional scale.. (unpublished phd). indiana university, indianapolis, in. lattanzi, j. b., campbell, s. l., dole, r. l., & palombaro, k. m. (2011). students mentoring students in a service-learning clinical supervision experience: an educational case report. physical therapy, 91(10), 1513-1524. https://doi.org/10.2522/ptj.20100308 musolino, g. m. (2006). fostering reflective practice: self-assessment abilities of physical therapy students and entry-level graduates. journal of allied health, 35(1), 30-42. national task force on civic learning and democratic engagement. (2012). a crucible moment: college learning and democracy’s future. washington, dc: association of american colleges and universities. osborne, r. e., hammerich, s., & hensley, c. (1998). student effects of service-learning: tracking change across a semester. michigan journal of community service learning, 5, 5-13. palombaro, k. m., black, j. d., & campbell, s. l. (2014). geriatric screening as an educational tool: a case report. journal of physical therapy education, 28(2), 54-50. palombaro, k. m., dole, r. l., & lattanzi, j. b. (2011). the development of a community clinic: how a signature project can mobilize commitment to sustainable community. in m. w. ledoux, s. c. wilhite & p. silver (eds.), civic engagement and service learning. (1st ed., pp. 90-108). hauppauge, ny: nova science publishers, inc. palombaro, k. m., dole, r. l., & lattanzi, j. b. (2011b). a case report of a student-led pro bono clinic: a proposed model for meeting student and community needs in a sustainable manner. physical therapy, 91(11), 1627-1635. https://doi.org/10.2522/ptj.20100437 palombaro, k. m., lattanzi, j. b., & dole, r. l. (2010). creating sustainable community engagement initiatives in a graduate physical therapy program. metropolitan universities, may, 61-75. pierce, s. r., palombaro, k. m., & black, j. b. (in press). brain safety fair. alexandria, va: apta. pierce, s. r., palombaro, k. m., & black, j. d. (2014). barriers to bicycle helmet use in young children in an urban elementary school. health promotion practice, 15(3), 406-412. https://doi.org/10.1177/1524839913512329 pike, g. r., bringle, r. g., & hatcher, j. a. (2014). assessing civic engagement at indiana university–purdue university indianapolis. new directions for institutional research, 162, 87 97. https://doi.org/10.1002/ir saltmarsh, j., & hartley, j. m. (2011). to serve a larger purpose: engagement for democracy and the transformation of higher education. philadelphia, pa: temple university press. https://doi.org/10.2522/ptj.20100308 https://doi.org/10.2522/ptj.20100437 https://doi.org/10.1002/ir palombaro, black, dole, pierce, santiago, and sabara journal of the scholarship of teaching and learning, vol. 17, no. 4, october 2017. josotl.indiana.edu 43 seifer, s. d. (1998). service-learning: community-campus partnerships for health professions education. academic medicine: journal of the association of american medical colleges, 73(3), 273-277. https://doi.org/10.1097/00001888-199803000-00015 smith, s. d., yoon, r., johnson, m. l., natarajan, l., & beck, e. (2014). the effect of involvement in a student-run free clinic project on attitudes toward the underserved and interest in primary care. journal of health care for the poor and underserved, 25(2), 877-889. https://doi.org/10.1353/hpu.2014.0083 steinberg, k. s., hatcher, j. a., & bindle, r. g. (2011) civic-minded graduate: a north star. michigan journal of community service learning, 18, 19-33. stickler, l., grapczynski, c., & ritch, j. (2013). student perceptions of outcomes from participation in physical therapy pro bono clinics. journal of allied health, 42(1), 46-55. . torney-purta, j., cabrera, j. c., roohr, k. c., liu, o. l., & rios, j. a. (2015). assessing civic competency and engagement in higher education; research background, frameworks, and directions for next-generation assessment. (no. rr-15-34). https://doi.org/10.1002/ets2.12081 tucker, m. l., & mccarthy, a. m. (2001). presentation self-efficacy: increasing communication skills through service-learning. journal of managerial issues, 13(2), 227-245. wise, h. h., & yuen, h. k. (2013). effect of community-based service learning on professionalism in student physical therapists. journal of physical therapy education, 27(2), 58 64. https://doi.org/10.1097/00001888-199803000-00015 https://doi.org/10.1353/hpu.2014.0083 https://doi.org/10.1002/ets2.12081 503 service temporarily unavailable 503 service temporarily unavailable nginx/1.14.1 503 service temporarily unavailable 503 service temporarily unavailable nginx/1.14.1 503 service temporarily unavailable 503 service temporarily unavailable nginx/1.14.1 503 service temporarily unavailable 503 service temporarily unavailable nginx/1.14.1 503 service temporarily unavailable 503 service temporarily unavailable nginx/1.14.1 503 service temporarily unavailable 503 service temporarily unavailable nginx/1.14.1 503 service temporarily unavailable 503 service temporarily unavailable nginx/1.14.1 503 service temporarily unavailable 503 service temporarily unavailable nginx/1.14.1 1 running head: “doing” phenomenological research “doing” phenomenological research: connecting nursing education, research, and professional practice rebecca s. sloan, rncs, ph.d. melinda swenson, rncs, ph.d. indiana university school of nursing department of family health nursing 1111 middle drive, nu 312 indianapolis, indiana 46206 corresponding author: rebecca s. sloan, rncs, ph.d. indiana university school of nursing nu 312 1111 middle drive, indianapolis, in 46202 phone (317) 278-1413 e-mail: rsloan@iupui.edu key words:graduate research education, phenomenology, nursing education, nursing research, nursing practice abstract 2 the interactive practices associated with faculty teaching and students learning to conduct phenomenological research provided an important opportunity for advanced practice nursing students to reevaluate their world view of patient care and the lived experience of health and illness. further, these students demonstrated transformation in their personal perspectives on nursing education, research, and practice. the investigators examined a variety of classroom action outcomes to explore the effectiveness of current teaching strategies in our masters research study course. to describe graduate students’ new understandings of health care, nurse educators analyzed self-evaluation documents and reflective narratives from masters nursing students who were learning to conduct phenomenological research. the following themes were identified (a) finding the exceptional in the routine, (b) resonating with stories of health and illness, and (c) putting this research into daily practice. we conclude that “doing” phenomenology not only results in excellent clinical research directly applicable to nursing practice, but served to transform the professional practice perspectives of graduate student nurses engaged in this research methodology. 3 doing phenomenological research: connecting nursing education, research and practice since florence nightingale's time, efforts have been made to unite the conversations between nursing education, research, and practice on a grand scale (1859/1969). unfortunately, the interrelationships between these three aspects of the nursing profession remain elusive and these topics are often addressed as if each were unrelated to the others. the investigators found the classroom actions for completing phenomenological research resulted in student research that was philosophically grounded, methodologically sound, and useful to nurses caring for individuals and families with chronic illness. further, the students described how the activities associated with actually doing phenomenological research resulted in transformation of their personal perspectives on nursing education, research and practice. further, they developed new understandings of patient care and the lived experience of health and illness. the investigators used interpretive phenomenology to explore the many facets of the students’ self-described transformation through doing phenomenological research. the following themes were identified: (a) finding the exceptional in the routine, (b) resonating with stories of health and illness, and (c) putting this research into daily practice. classroom actions for students learning to conduct phenomenological research 4 adult and family nurse practitioner students completed eight months of independent research in clinical areas of their choosing to meet the final requirements for the masters in nursing science degree. these projects were approved by the indiana university human studies committee. to date, thirty-five students have participated in the phenomenological research option. phenomenological research is useful in describing and understanding a human experience (phenomenon) through analysis of narrative text. heidegger’s (1962) phenomenological view of the person centered on “lived experience” as an interpretation of the person in relation to his world, as a being for whom things and events have significance and value, and as the experience of things and events as they are self-interpreted through the person’s perception and interpretation of physical body experiences, time, and those things they care about (leonard, 1994). students used interpretive phenomenology methods to understand the phenomenon of chronic illness. the investigators used the same methods to explore the phenomena of students learning to use phenomenology methods of research. the objective of the course was to teach students to conduct nursing research using a phenomenological approach. from a classroom action research perspective (mettetal, 2001), our question was to discover what other form(s) of learning were outcomes of these traditional research training activities. students met as a group on a weekly basis for round table discussions regarding their research projects. these were facilitated by the investigators in our roles as co 5 teachers. classroom discussions addressed phenomenological research philosophies, methods, and analysis procedures. early in the learning process, both students and teachers offered stories of their own nursing education, practice experiences and personal experiences with health and illness. these served to bring light to shared meanings and common practices found within our own lived experiences with health and illness. each student selected individuals with chronic illnesses as participants in his or her research study. family members were invited to share their experiences as well. each student engaged his or her participants in in-depth tape recorded interviews lasting from 1-2 hours each. the interviews were transcribed verbatim and shared with the other students and teachers. the students and teachers read and made independent analyses of each transcript (diekelmann, allen & tanner, 1989). over the next several months, the students and faculty shared their analyses, made suggestions, and offered readings and literature reviews useful to other students. the expected outcome of this process was accomplished as each student developed his or her own analysis, produced a manuscript of the research findings from his or her own data, and connected the conceptualizations of the individual narratives across all the interviews. methods for classroom action research study of lived experience of doing phenomenological research data for the present study were gathered from (a) group conversations and 6 interactions throughout the students' research experience, (b) through statements of topical presence and reflexive evaluation included in individual completed research papers, (c) from course and teacher evaluations, and (d) from oral comments made to the teachers at the end of coursework. using the same interpretive analysis techniques (diekelmann, allen & tanner, 1989) employed when analysing the transcripts from individuals with chronic illness and family members, the investigators analyzed the students’ verbal and written narratives and identified themes and patterns involved in this transformative process. in this process narrative data was read numerous times. themes within each individual narrative were identified. themes were clustered into patterns, common and recurring phenomena, which appeared in multiple narratives. patterns are the highest level of findings coming from phenomenological research. findings initially, students described how having conversations with patients about various illnesses was a hallmark of traditional nursing care and not a new skill for these much experienced graduate nursing students. they recalled how their nursing conversations were focused to discover the "facts" rather than the "meaning" of a patient's particular situation. by the completion of individual research projects, students found new insights in seemingly routine conversations with patients and families. these insights resulted in an enrichment of the students’ own practice base and the possibilities of sharing new ideas with other nurses. students who engaged in the phenomenological research of chronic illness 7 demonstrated significant insight into the illness experience of their participants. not only did students provide narratives from the study participants, they also provided their own personal narratives from doing this research. these personal narratives repeatedly included phrases such as "i will never look at patients the same way again". in learning about health and illness, these students experienced transformation in how they practiced as nurses. we identified three themes related to that transformation including (a) resonating with the stories, (b) finding the exceptional in the routine, and (c) putting this research into practice. resonating with the stories. heidegger described how learning to resonate with the essence of a lived experience allowed new ways of thinking to come forward (1971, pg. 5): as soon as we have the thing before our eyes and in our hearts an ear for the word, thinking prospers. one of the first assignments was for students to write about a personal experience with a health or illness situation describing the experience and its personal meaning. as students shared these with each other, they began to understand "lived experience" as a significant and unique journey. each found ways in which another's experience reflected his or her own experience or brought new understandings of the illness experience. they then used these skills to find connections and understandings in the participants’ stories of health and illness. no longer did the students interview participants to gather medical histories. 8 rather, they participated in "inter-views” (kvale, 1996) that allowed new insights into others' stories which resonated with and became part of their own stories as nurses. one described this phenomenon as: listening to and absorbing the life stories, they become a part of my experience and me. seeing the patient and family at their most vulnerable and weakest times, and sometimes at their best, draws me into their world with an intensity that often only nurses have been privileged to feel. "resonating" went beyond seeing or hearing differently. for some, a new bond occurred with patients and nurses as the graduate student nurses came to understand health and illness beyond pathophysiological parameters: listening to their life experiences links me to them. the power of their words describes their roller coaster experiences and the overwhelming constraints in their lives. i will take their words with me as i move in and out of other patients' lives. i will listen more carefully to hear exactly what their illness means to them and their families. perhaps i can smooth the jagged edges of adjustment from health to chronic illness for another family ... finding the exceptional in the routine. 9 most of our students had long professional experience in caring for patients with various illnesses and health concerns. in fact they were so experienced they frequently described patient-nurse interactions in language denoting "ordinary" or "routine". early in the course work, they frequently used proscribed nursing jargon ("inability to care for self secondary to profound mental retardation”) or pathophysiological labels (polymicrogyria, polycystic ovary syndrome, or cardiomyopathy) to describe their patients situations. students also described the “meaning” of these conditions from purely psycho-social or behavioral perspectives (i.e., denial, enmeshment, and noncompliance). these labels reflected an outside-in view of what individuals experience (what one thinks another experiences), rather than the phenomenological inside-out perspective of understanding the unique meaning of the illness experience for those who have chronic illness conditions.. heidegger described phenomenology as the ability to find the "splendor in the simple" (1971, p. 7). once students were able to resonate with the stories, they were able to go beyond psycho-social labels. while patients' health problems might be ordinary, the students’ found the lived experiences of participants were powerful and unique. in a study of severe rheumatoid arthritis, one student found : [the woman] kept talking about wanting to be normal. i believe that she is anything but normal -she is exceptional. 10 putting this research into my practice. as nurse educators, we expected that students would successfully complete their research projects, develop their research papers, and find the research course to be exciting and challenging. we did not anticipate that students’ professional practices would substantively change as a result of doing phenomenological research. transformation of graduate student nurses’ view of health and illness and how they practiced nursing was an unexpected outcome. an example from a student demonstrated this transformation: the lived experience of chronic illness (was) very powerful, forcing me to consider my past care for individuals and their families. although i have been concerned for them and their "living", i have rarely attempted to truly understand their experience. . . in the future i will be better able to truly listen. . . i believe i will be a more caring person and nurse. another graduate nursing student wrote: i learned that nursing concerns are not always client concerns . . . i will always look at each person as different and unique and will treat them as such. i learned new ways to interact with my clients that i had not thought of before. this will help me be less of a black-and-white kind of person and will enable me to see the gray areas as well. 11 while each student completed satisfactory research papers, not all students experienced a transformation in their world view of patients and families managing chronic illness. reflecting on the students who did not demonstrate transformation in their nursing practice, we found the following. these students commonly described their comfort with rigid empirical design methods, statistical analysis, and validity and reliability assurances. some students struggled initially in adopting qualitative research philosophies, finally appreciation of the method late in the research process. others never did find comfort in these methods. these students were not able to incorporate reflective interview skills and their research interviews were not very different from the familiar nursing history taking which had always been part of their nursing practices. these students were in the minority as most of the students actually came to embrace phenomenological research methods, engaged in reflective analysis of their narrative data, and described a transformation in the way they practiced nursing care for individuals and families facing chronic illnesses. the students described how doing phenomenological research provided them with a means of "seeing with new eyes" or "hearing with new ears". further, they were able to find concrete ways to change not only their own nursing practices, but the practice of others as well. as one noted: i can now play an instrumental role in erasing this stereotyped image (of aging) and replace it with one of an individual who is vibrant and alive. 12 conclusions graduate nursing students described how active participation in phenomenological research served as a catalyst for them to share their research, education, and practice experiences through conversations with the research team. while the outcome of each student’s independent research will advance nursing practice in and of itself, an additional outcome was found in this classroom action research study. students described how collectively engaging in the interactive process of teaching and learning phenomenological research transformed their individual thinking and changed them as practicing nurses. we find these classroom activities provided graduate student nurses a unique opportunity to connect for themselves the conversations which bridge and enrich nursing education, research, and practice in ways that traditional academic efforts have not been able to provide. references diekelmann, n., allen, d., & tanner, c. (1989). the nln criteria for appraisal o baccalaureate programs: a critical hermeneutic analysis. (pub. no. 15-2253. new york: national league for nursing press. 13 heidegger, m. (1962). being and time. j. macquarrie & e. robinson, trans.) san francisco: harper & row. (original work published 1926). heidegger, m. (1971). the thinker as poet. in (a. hofstadter, trans). poetry language and thought. ny: harper & row, publishers, inc. kvale, s. (1 996). interviews: an introduction to qualitative research interviewing. thousand oaks, ca: sage publications, inc. leonard, v. w. (1994). a heideggerian phenomenological perpective on the concept of person. in p. benner (ed.), interpretive phenomenology: embodiment, caring and ethics in health and illness (pp. 43-63). thousand oaks, ca: sage. mettetal, g. (2001). the what, why and how of classroom action research. the journal of scholarship of teaching and learning 2, 6-13. retrieved august 10, 2001 from www.iusb.edu/~jostl.htm. nightingale, f. (185911969). notes on nursing: what it is, and what it is not. new york: dover publications. the what, why and how of classroom action research the editors of josotl have received many inquiries about classroom action research (car). what is it? why should you consider doing it? how do you do it? how does it differ from traditional research on teaching and learning? this essay is an attempt to answer those questions. i will also discuss why car is an excellent expression of the scholarship of teaching and learning, accessible to teachers in all disciplines. what is classroom action research? classroom action research is a method of finding out what works best in your own classroom so that you can improve student learning. we know a great deal about good teaching in general (e.g. mckeachie, 1999; chickering and gamson, 1987; weimer, 1996), but every teaching situation is unique in terms of content, level, student skills and learning styles, teacher skills and teaching styles, and many other factors. to maximize student learning, a teacher must find out what works best in a particular situation. there are many ways to improve knowledge about teaching. many teachers practice personal reflection on teaching; that is, they look back at what has worked and has not worked in the classroom and think about how they can change their teaching strategies to enhance learning. (hole and mcentee (1999) provide useful steps for enhancing such reflection. a few teachers (most notably education professors) conduct formal empirical studies on teaching and learning, adding to our knowledge base. car fits in the center of a continuum ranging from personal reflection at one end to formal educational research at the other. car is more systematic and data-based than personal reflection, but it is more informal and personal than formal educational research. in car, a teacher focuses attention on a problem or question about his or her own classroom. for example, does role-playing help students understand course concepts more completely than lecture methods? which concepts are most confusing to students? (see comparison chart at http://www.iusb.edu/~gmetteta/research_about_teaching_and.htm ) action research methods were proposed by kurt lewin in 1946, as a research technique in social psychology. more recently, donald schön (1983) described the reflective practitioner as one who thinks systematically about practice. classroom action research is systematic, yet less formal, research conducted by practitioners to inform their action. the goal of car is to improve your own teaching in your own classroom (or your department or school). while there is no requirement that the car findings be generalized to other situations, as in traditional research, the results of classroom action research can add to the knowledge base. classroom action research goes beyond personal reflection to use informal research practices such as a brief literature review, group comparisons, and data collection and analysis. validity is achieved through the triangulation of data. the focus is on the practical significance of findings, rather than statistical or theoretical significance. findings are usually disseminated through brief reports or presentations to local colleagues or administrators. most teachers, from pre-school through university level, can be taught the methods of action research in a single course, a series of workshops, or through extensive mentoring (mettetal, 2000). for more information on traditional educational research, see texts such as educational research (gay and airasian, 2000). the boundaries between these categories are not distinct. some car projects may become comprehensive enough to be considered traditional research, with generalizable findings. other car projects may be so informal that they are closer to personal reflection. in this essay, i will describe the prototypical car project. why do classroom action research? first and foremost, classroom action research is a very effective way of improving your teaching. assessing student understanding at mid-term helps you plan the most effective strategies for the rest of the semester. comparing the student learning outcomes of different teaching strategies helps you discover which teaching techniques work best in a particular situation. because you are researching the impact of your own teaching, you automatically take into account your own teaching strengths and weaknesses, the typical skill level of your students, etc. your findings have immediate practical significance in terms of teaching decisions. second, car provides a means of documenting your teaching effectiveness. the brief reports and presentations resulting from car can be included in teaching portfolios, tenure dossiers, and other reports at the teacher or school level. this information can also help meet the increasing requirements of the assessment movement that we document student learning. third, car can provide a renewed sense of excitement about teaching. after many years, teaching can become routine and even boring. learning car methodology provides a new challenge, and the results of car projects often prompt teachers to change their current strategies. car projects done as teams have the added benefit of increasing peer discussion of teaching issues. how do you conduct classroom action research? classroom action research follows the same steps as the general scientific model, although in a more informal manner. car methods also recognize that the researcher is, first and foremost, the classroom teacher and that the research cannot be allowed to take precedence over student learning. the car process can be conceptualized as a seven-step process. (for more detailed information about conducting car research, see authors such as bell, 1993; sagor, 2000; and hubbard and power, 1993) step one: identify a question or problem. this question should be something related to student learning in your classroom. for example, would a different type of assignment enhance student understanding? would a strict attendance policy result in better test scores? would more time spent in cooperative learning groups help students understand concepts at a higher level? the general model might be "what is the effect of x on student learning?" since the goal of car is to inform decision-making, the question or problem should look at something under teacher control, such as teaching strategies, student assignments, and classroom activities. the problem should also be an area in which you are willing to change. there is no point in conducting a car project if you have no intention of acting on your findings. larger institutional questions might be tackled, if the institution is committed to change. finally, the question or problem should be feasible in terms of time, effort and resources. in general, this means to think small--to look at one aspect of teaching in a single course. angelo and cross (1993) suggest that you not start with your "problem class" but rather start with a class that is progressing fairly well. as you become more comfortable with car methods, you may attempt more complicated projects. step two: review literature you need to gather two types of information, background literature and data. the literature review may be much less extensive than traditional research, and the use of secondary sources is sufficient. sources such as cross and steadman (1996) or woolfolk (2000) will often provide background information on learning, motivation, and classroom management topics. another source is the educational resources information center (eric) database, which contains references to a huge number of published and unpublished manuscripts. you can search the eric database at http://ericir.syr.edu/ your campus' teaching and learning center should also have many useful resources. step three: plan a research strategy the research design of a car study may take many forms, ranging from a pretest-posttest design to a comparison of similar classes to a descriptive case study of a single class or student. both quantitative and qualitative methods are appropriate. the tightly controlled experimental designs of traditional research are rarely possible in a natural classroom setting, so car relies on the triangulation of data to provide validity. to triangulate, collect at least three types of data (such as student test scores, teacher evaluations, and observations of student behavior). if all data point to the same conclusions, you have some assurance of validity. step four: gather data car tends to rely heavily on existing data such as test scores, teacher evaluations, and final course grades. you might also want to collect other data. see angelo and cross (1993) for a wonderful array of classroom assessment techniques. (be sure to check with your institutional review board for policies regarding the use of human subjects. most car with adult students will be exempt from review as long as you do not identify individual students.) step five: make sense of the data analyze your data, looking for findings with practical significance. simple statistical analyses of quantitative data, such as simple t-tests and correlations, are usually sufficient. tables or graphs are often very helpful. qualitative data can be analyzed for recurring themes, citing supporting evidence. practical significance, rather than statistical significance, is the goal. step six: take action use your findings to make decisions about your teaching strategies. sometimes you will find that one strategy is clearly more effective, leading to an obvious choice. other times, strategies may prove to be equally effective. in that situation, you may choose the strategy that you prefer or the one that your students prefer. step seven: share your findings you can share your findings with peers in many ways. you may submit your report to josotl, which has a special section for car reports. these articles will typically be from 4 to 8 pages--shorter than the typical traditional research report. most car reports are appropriate for submission to the eric database (instructions for submission can be found on the eric website at: http://ericfac.piccard.csc.com/submitting). you might also share your work at conferences such as the international conference for teacher-researchers ( http://www.educ.ubc.ca/ictr2001/ ) or at regional conferences for your discipline. most disciplines sponsor a journal on teaching, although car may be too informal to meet publication requirements. judging the quality of car projects although car projects are not as comprehensive as traditional educational research, their quality can still be assessed using the guidelines of glassick, et al (1997) in scholarship assessed. i recently worked with colleagues to develop an evaluation plan for the car projects of k-12 teachers in a local school district (mettetal, bennett and smith, 2000). the resulting rubric has been adapted for josotl and is used by our reviewers for car, traditional research, and essay (http://www.iusb.edu/~josotl/rubric/rubric.htm). classroom action research rubric criteria for quality proposal and projects needs improvement on target exemplary goals goals are not clearly identified. goals are identified and relate to teaching and learning. goals are clearly stated, relate to teaching and learning and will inform action. background information no reference to previous research or theory. two to three references to relevant research or theory. integrates and synthesizes four or more sources of relevant research or theory. methods less than three sources of data. three sources of data from current classroom. many sources of data from current classroom (case study) or data that are compared with data from another relevant source (i.e., last year’s class, another class in the school, state data). results results are not communicated in an appropriate manner. communicate results through themes, graphs, tables, etc. results identify key findings. communicate results clearly and accurately through themes, graphs, tables, etc. reflection little or no relevant discussion of teaching and learning related to one's own classroom. discusses how results affect one's own teaching and learning in classroom. discusses how results affect own teaching and learning in classroom and implications for teaching setting (i.e., other classroom, schools, district, etc.). also, identifies future research questions. presentation • paper not clearly written • results are not shared with other audiences. • paper clearly written • results shared with a local colleagues • paper is clear, insightful, and comprehensive • results are shared with a wider audience. this rubric shows that it is possible to meet the standards of glassick et al (1997) within the context of a classroom action research project. one of the most difficult criteria to meet is that of presentation, since there have been few forums for the publication of car projects. josotl hopes to correct that problem. conclusion classroom action research fits comfortably under the umbrella of scholarship of teaching and learning. along with traditional educational research and course portfolios, car is a way of systematically examining teaching to gain new insights. one can certainly be an excellent teacher without engaging in car (or other types of sotl), but participation in some version of sotl enhances one's knowledge of the profession of teaching. car is very attractive to faculty at all types of institutions. those at primarily research institutions may welcome the opportunity to look at teaching with the same scholarly eye that they use for disciplinary research. those at primarily teaching institutions (including vocational tech and community colleges) usually lack support for disciplinary research. they may find that their institutions provide a rich source of car data and that administrators appreciate these research endeavors. the editors of josotl agree that classroom action research is an appropriate form of the scholarship of teaching and learning. josotl is eager to receive submissions of car articles and will evaluate them using the rubric provided here. references angelo, t.a. and cross, k.p. (1993). classroom assessment techniques: a handbook for college teachers, 2nd edition. san francisco, ca: jossey-bass. bell, j. (1993). doing your research project, 2nd edition. philadelphia: open university press. chickering, a.w. and gamson, z.f. (1987). "seven principles for good practice in undergraduate education." aahe bulletin, 39 (7), 3-7. cross, k.p. and steadman, m.h. (1996). classroom research: implementing the scholarship of teaching. san francisco, ca: jossey-bass. glassick, c.e., huber, m.t., and maeroff, g.i. (1997). scholarship assessed: evaluation of the professoriate. san francisco, ca: jossey-bass. hole, s. and mcentee, g.h. (1999). "reflection is at the heart of practice." educational leadership 56(8), 34-37. hubbard, r.s. and power, b.m. (1993) the art of classroom inquiry. portsmouth, nh: heinemann. mckeachie, w.j. (1999). teaching tips: strategies, research and theory for college and university teachers. boston: houghton mifflin. mettetal, g.and cowan, p. (2000), assessing learning through classroom research: the supporting teachers as researchers project (star). classroom leadership online, 3 (8) at http://www.ascd.org/readingroom/classlead/0005/1may00.html mettetal, g., bennett, j., and smith, j. (unpublished manuscript) educate indiana grant proposal, 2000. palmer, p. (1998) the courage to teach, san francisco, ca: jossey-bass. sagor, r. (2000). guiding school improvement with action research. alexandria, va: association for supervision and curriculum development. schön, d. (1983). the reflective practioner, basic books. weimer, m. (1996). improving your classroom teaching. newbury park, ca: sage. woolfolk, a. (2000). educationalpsychology, 8th edition. needham heights, ma: allyn and bacon. other useful books: bruning, j.l. & kintz, b.l. (1997). computational handbook of statistics. 4th ed. new york: longman. burnaford, g., fischer, j., & hobson, d. (1996). teachers doing research: practical possibilities. mahwah, nj: lawrence erlbaum. cresswell, j.w. (1994). research design: qualitative and quantitative approaches. thousand oaks, ca: sage. glesne, c. (1999). becoming qualitative researchers: an introduction, 2nd ed. new york: longman. hopkins, d. (1985). a teacher's guide to classroom research. philadelphia: open university press. mcniff, j., lomax, p. & whitehead, j. (1996) you and your action research project. new york: routledge. mills, g.e. (2000). action research: a guide for the teacher researcher. columbus, oh: merrill. yin, r.k. (1994). case study research: design and methods, 2nd ed. thousand oaks, ca: sage. 503 service temporarily unavailable 503 service temporarily unavailable nginx/1.14.1 503 service temporarily unavailable 503 service temporarily unavailable nginx/1.14.1 503 service temporarily unavailable 503 service temporarily unavailable nginx/1.14.1 503 service temporarily unavailable 503 service temporarily unavailable nginx/1.14.1 from minsk to pinsk: why a scholarship of teaching and learning? lee s. shulman the carnegie foundation for the advancement of teaching preamble more than 25 years ago, i was serving as an american psychological association visiting scholar to the psychology departments of small liberal arts colleges. i spent two days at a lovely campus in southeastern indiana, hanover college. i particularly enjoyed the energy and intelligence of an undergraduate psychology major named randy isaacson. a short time later, he was admitted to the doctoral program in educational psychology at michigan state university, where i had been teaching since 1963. when randy completed his phd at michigan state, he returned to indiana as a member of the faculty at indiana university, south bend. what a pleasure it has been to reconnect with randy so many years later around our mutual passion for the importance of a scholarship of teaching and learning. i deeply appreciate his role in the creation of this on-line journal. the indiana university system is demonstrating significant national leadership in sponsoring this effort, as well as in its pioneering initiatives to recognize and reward scholarly contributions to teaching and learning among its faculty members. a strange journey as more individual teacher/scholars and their institutions become engaged in the scholarship of teaching and learning, we often find ourselves discussing the history of the phenomenon, the precise definitions of “scholarship,” “teaching,” and “learning,” and some of the methodological and technical standards for conducting such research in an excellent manner. periodically, it is worthwhile to step back and ask: “why are doing this? what are the reasons we are committed to the pursuit of such work?” at such times i am reminded of the old jewish story of the russian itinerant who needed to travel from minsk to pinsk. he caught a ride with a wagon driver whose cart was drawn by a rather ancient horse. as they approached the first significant hill on the minsk-pinsk highway, the driver halted the cart, unhitched the horse, and asked the passenger to assist him in pushing the wagon to the top of the hill. at the top, he hitched up the horse again, and they proceeded on their way until the next small elevation, where they again repeated the previous procedure. after the fifth such ritual, the now-exhausted passenger dropped to his knees at the side of the road and looked quizzically at the driver. “i know why i have to get to pinsk. i suspect you have a reason for going there as well. enlighten me please. why are we 2 bringing the horse?” as we strengthen both our resolve and our capacities for moving faculty in higher education from the minsk of a restricted view of scholarship to the pinsk of a more comprehensive and inclusive perspective, we had better step back and make sure we understand why we need the horse of a scholarship of teaching and learning. three p’s i’d like to suggest that there are three broad rationales for advocating a serious investment in the scholarship of teaching and learning: professionalism, pragmatism, and policy. professionalism refers to the inherent obligations and opportunities associated with becoming a professional scholar/educator, and especially with the responsibilities to one’s discipline symbolized by the phd. pragmatism refers to the activities needed to ensure that one’s work as an educator is constantly improving and meeting its objectives and its responsibilities to students. policy refers to the capacity to respond to the legitimate questions of legislatures, boards and the increasingly robust demands of a developing market for higher education. professionalism. the most important reason for engaging in the scholarship of teaching is professional role and responsibility. each of us in higher education is a member of at least two professions: that of our discipline, interdiscipline or professional field (e.g., history, women’s studies, accounting) as well as our profession as educator. in both of these intersecting domains, we bear the responsibilities of scholars—to discover, to connect, to apply and to teach. as scholars, we take on the obligation to add to the core of understanding, skepticism, method and critique that defines our fields and their everchanging borders. we also assume the responsibility for passing on what we learn to discern and act, through teaching, social action, and through exchanging our insights with fellow professionals. indeed, the core values of professional communities revolve around the expectation that we do not keep secrets, whether of discovery or of grounded doubt. we are expected to share our knowledge by making it public, whether via publication, correspondence, presentations or pedagogy. the new technologies make such exchange even more widely possible than ever before. i have emphasized the professional imperatives for a scholarship of teaching most seriously in other writings. in so doing, i have also emphasized the importance of distinguishing between two equally important and desirable activities—scholarly teaching and a scholarship of teaching. this is a distinction that boyer chose not to make in scholarship reconsidered. scholarly teaching is teaching that is well grounded in the sources and resources appropriate to the field. it reflects a thoughtful selection and integration of ideas and examples, and well-designed strategies of course design, development, transmission, interaction and assessment. scholarly teaching should also model the methods and values of a field, avoiding dogma and the mystification of evidence, argument and warrant. we develop a scholarship of teaching when our work as teachers becomes public, peerreviewed and critiqued, and exchanged with other members of our professional 3 communities so they, in turn, can build on our work. these are the qualities of all scholarship. both scholarly teaching and a scholarship of teaching are deeply valued in the professional community. scholarly teaching is like the clinical work of faculty members in a medical school’s teaching hospital. i would never wish to be associated with a medical school that was not home to outstanding clinical faculty. that clinical work, however valued, does not become scholarship until it is subjected to systematic reflective analysis. such reflection leads to its display or communication in ways that render it community property in the fullest sense—public, reviewed and exchanged. the professional rationale for engaging in the scholarship of teaching is that affords all of us the opportunity to enact the functions of scholarship for which we were all prepared. we can treat our courses and classrooms as laboratories or field sites in the best sense of the term, and can contribute through our scholarship to the improvement and understanding of learning and teaching in our field. thus, the professional imperative for a scholarship of teaching is both individual and communal. we fulfill our own obligations as members of the dual professions with which we identify, and we fulfill our responsibilities to our professional peers to “pass on” what we discover, discern and experience. pragmatism. the professional rationale is critical, but not sufficient. we also have a practical rationale for pursuing the scholarship of teaching and learning. such work helps guide our efforts in the design and adaptation of teaching in the interests of student learning. by engaging in purposive reflection, documentation, assessment and analysis of teaching and learning, and doing so in a more public and accessible manner, we not only support the improvement of our own teaching. we raise the likelihood that our work is transparent to our colleagues who design and instruct many of the same students in the same or related programs. active scholarship of teaching provides the teacher with a very different perspective on what he or she may have been doing for many years. i have recently had such an experience myself. a recent—and personal—example. during the past semester, i have been teamteaching (with my colleague professor linda darling-hammond) a course—principles of learning for teaching--that i have taught at stanford since 1983. for most of those years, i co-taught the class with a variety of colleagues. the team teaching alone fostered serious reflection about the teaching. i have written about my strategies of teaching in the course (e.g., shulman, 1996), but i never actively conducted research on the teaching and learning taking place in the course itself. this year, for the first time, we agreed to conduct more systematic research on the teaching and learning processes. this commitment was in no small measure motivated by my experience in working with carnegie scholars on their own scholarship of teaching projects. the course is offered to all (nearly 60) secondary teaching candidates at stanford. they 4 are preparing to teach mathematics, social studies, english, science or foreign languages in middle and high schools. all students already hold at least a ba or bs in their discipline and will receive an ma at the completion of their teacher preparation. during the academic year, each student is actively teaching in a secondary school for the first half of the day, returning to campus in the afternoon for formal classes, practicums and seminars. at the core of the class is the case-writing assignment. all students are expected to complete a case study of their own practice during the class. they begin with brief “case starts” in which they outline an extended episode that they believe will be “caseworthy.” after feedback from the instructors and from their own colleagues, they prepare a firstdraft case, which is presented in a small working case conference. based on feedback, they then spend nearly a month revising and editing their case (often choosing to write an entirely new case) which is presented at a second case conference and then written in final form. during this period, they continue to read a variety of theoretical and research material on teaching and learning, as well as additional cases written by others. the final version of the case is written up after the second case conference, and is accompanied by two commentaries written by others, and a five-page reflective essay on the whole process. we decided before the class was offered that we would conduct research on the processes of learning through case writing that constituted the central structure of the course. our teaching colleague dr. karen hammerness took on responsibility for documentation and data gathering. karen systematically collected each draft of every case written, including the commentaries and the reflective essays. extensive notes were taken of every class session. selected sessions were videotaped, and selected case conferences were also videotaped. these data will now be organized and analyzed to answer a number of questions about the efficacy of case writing in promoting reflection, deep understanding and motivation among the students in the program. we intend to do several things with these data. we will certainly meet to reflect on our findings and use those insights to redesign the course for the coming year. these meetings have already begun. we will also prepare more formal oral and written presentations on our experiences, methods, and findings. hammerness is also taking leadership in developing a new web-site through which she will communicate our activities in the course, our insights into the learning that did (and did not) occur, and our analyses of the effort. we will also provide abundant examples of evolution of selected cases written during the term. thus the web-site will also include examples of student work and their own reflections. i offer this personal account as an example of how the pragmatics of engaging in a scholarship of teaching on a course i have taught for years has introduced far more intelligent design and analysis of my own work than i have ever done before. moreover, it has brought me into collaboration with close colleagues in new ways. i fully expect that our efforts at rendering this work public, reviewable and available for exchange can serve as a valuable resource for colleagues both within stanford and in the more general 5 community of teacher educators. i also believe that these efforts will lead to significant improvements in the course itself and in the program of which it is a part. policy. we in higher education are also enmeshed in webs of national, state and local policy. those who make policies and approve budgets for our institutions are increasingly asking for evidence that we are making measurable progress toward our educational goals. accrediting agencies are insisting on educational “audits” in which we provide evidence that we are achieving our stated goals and missions. “accountability” and “assessment” have become the themes of the emerging movements toward reform in higher education. these are not bad ideas. they only become problems when the wrong indicators are used to assess the quality of our efforts. they are only problematic if the metrics employed are chosen because of convenience or economy of use, rather than because they serve as authentic proxies for the learning and development we seek to foster. these indicators cannot be “one-size-fits-all” quick-and-dirty off-the-shelf instruments that purport to measure the outcomes of higher education. they should be the result of carefully conceptualized, designed and deployed studies of teaching and learning in each of our fields, conducted by scholars qualified to pursue them. this kind of work cries out for a vigorous scholarship of teaching and learning engaged by discipline and fieldspecific scholars of teaching. the free market is also creating new challenges for higher education. for-profit providers, distance learning, and other new sources for higher education are creating a market wherein institutions must be prepared to document and display evidence that they are fostering learning, deep understanding, passionate commitments and civic virtues in the domains in which they educate. once again, unless we can provide relevant evidence of the processes and products of our pedagogies, we will find ourselves making empty claims and offering degraded arguments. i envisage a scholarship of teaching and learning offering the kinds of evidence that can be powerful in these policy and free market discussions. new forms of institutional research will be developed that are learning-focussed, domain-specific, and oriented toward analyzing the educative experiences and outcomes that institutions support or fail to support. so who needs the horse? i began this essay with a story about making the journey from minsk to pinsk. the protagonists seemed to understand why each of them needed to make the journey; it was unclear why they needed the horse. in this case, i believe it is clear why our professional, pragmatic and policy interests can be supported and enhanced by a scholarship of teaching and learning. it will not be an easy journey. at first, it may seem as if the horse is either useless, or an additional burden itself. ultimately, however, we will need a sturdy horse to carry us on these journeys. we cannot do these things alone. we will need to develop networks of campus-based teaching academies to serve as centers, 6 support systems and sanctuaries for these kind of scholarly efforts. our interest in engaging in such work was summarized by three p’s, our professional interest, our pragmatic responsibilities, and the pressures of policy. scholarship of teaching and learning supports our individual and professional roles, our practical responsibilities to our students and our institutions, and our social and political obligations to those that support and take responsibility for higher education. we should be making all three journeys and we need a really good horse. this journal, its leaders, readers and contributors, are helping to make the journey possible. these remarks were originally prepared for a meeting of the carnegie academy for the scholarship of teaching and learning (castl) campus affiliates hosted by the american association for higher education (aahe) at its 2000 annual meeting held in anaheim, ca, march 29, 2000. lee s. shulman is president of the carnegie foundation and professor of education, stanford university. shulman, lee s. (1996). “just in case: reflections on learning from experience” in j.a. colbert, p. desberg & k. trimble (eds.) the case for education: contemporary approaches to using case methods. boston: allyn and bacon. the web page with the syllabus, instructions for writing cases and other materials for the course can be found at http://www.stanford.edu/class/ed269/. by the way, minsk and pinsk are now cities in belarus. minsk is the national capitol. in the 18th and 19th centuries they were usually part of russia, but at different periods one or the other was in poland or lithuania. i have written about a variety of visions for such academies in my essay “visions of the possible” reproduced at the carnegie foundation website: http://www.carnegiefoundation.org/ourwork/ourwork.htm integrating course 1 1 running head: integrating course and instructional evaluation integrating course and instructional evaluation with a learning history approach alan clardy towson university july 30, 2002 key words: educational evaluation, learning history, on-line learning system integrating course 2 2 abstract student ratings are the traditional approach to course and instructional assessment, yet this procedure has questionable reliability and validity. teacher portfolios, while an improvement, also have some shortcomings. an on-line “learning history” approach to course and instructional assessment was tested in a graduate course in the fall, 2000. the characteristics of this technique as applied to instructional assessment are reviewed and compared to the student ratings and teacher portfolios approaches, and results of this initial pilot test are presented. a generic model for adapting this approach to any course is provided along with recommendations for further testing. integrating course 3 3 integrating course and instructional evaluation with a learning history approach evaluation – of faculty, instruction and/or learning – is a time-honored component of the educational process. presumably, evaluation should serve several functions, including assessing instructor performance for merit and other personnel reviews, identifying effective and ineffective instructional practices, gauging student reactions to courses and faculty, and providing a context for judging the quality of student learning, to name a few (braskamp and ory, 1994). these often competing goals and multiple uses, though, require different kinds of information, making the task of finding and implementing the most constructive and efficient process for evaluation a pernicious issue. this paper will report on an action learning adaptation of the “learning history” technique to evaluation in higher education. the first portion of this paper will review the use of and problems with the traditional student ratings and the more recent teaching portfolio approach. then, the learning history technique will be reviewed, followed by a discussion of how this technique was used in a graduate course. finally, the implication of this approach for practice and future research will be considered. evaluation in education often, in higher education, course evaluation is equated with end-of-semester student ratings of faculty. a 1982 survey of more than 600 liberal arts colleges found that two-thirds (67.5%) always used “systematic student ratings” for evaluating teaching performance, a noticeable increase from the 55% who had reported on such usage five integrating course 4 4 years earlier (seldin, 1984). and in both surveys, student ratings of faculty were the most frequently used method of evaluation. by 1993, seldin found that student evaluations had risen to 86% of the 600 colleges surveyed. evidence on the effectiveness and validity of student ratings of faculty, though, has been mixed (abrami, d’appollonia and cohen, 1990, braskamp and ory, 1994). in spite of their widespread use, the traditional method of student evaluation can be dogged by three problems: bias, coverage, and focus/contamination. the net effect of these problems raises questions about their suitability for evaluation purposes. student rating bias is an obvious possibility when adverse treatment (for example, due to grade disappointments or because of disciplinary actions against the student) is joined to a procedure in which anonymity provides a lack of rating accountability. a resulting student animus to the instructor can easily become translated into ratings that may have little to do with the faculty member’s actual performance in the course. in a field experiment with community college students in which respondent anonymity and course grades were manipulated and communicated immediately prior to student evaluations of their instructor, blunt (1991) found that both factors affected ratings of instructors. under conditions of anonymity, ratings of faculty were consistently lower than when students were asked to sign the evaluations. likewise, students in deflated grade conditions also provided lower instructor ratings. aware of this possibility, instructors can easily gravitate to a position of either complete indifference to student commentary, or to a modification of teaching practices (by limiting demands and assignments, grading easily, and so on) in order to increase the ratings they are given by their students. in a review of grading levels before and after the introduction of student evaluations of integrating course 5 5 teaching (set), stratton, myers and king (1994) found that grade levels increased 11% after the introduction of sets. these effects did decay over time, however. birnbaum (n.d.) surveyed 208 faculty at california state university, fullerton on their judgments about the effects of student evaluations on student learning and their teaching practices. almost three of four (72.1%) faculty respondents believed that the use of student evaluations encouraged faculty to dilute the rigor of their courses in order to curry student favor. in the same study, birnbaum also found that, in a sample of 142 undergrads, virtually all (97.9%) gave higher ratings to easier rather than harder courses. a second problem with traditional student-based evaluations arises from a confusion over whether the coverage of the evaluation process should be formative or summative (adams, 1997). a formative evaluation would collect information about the quality, adequacy and usage of various instructional inputs (like instructor classroom practices, readings, learning activities, instructional media used and so on). further, since the ideal purpose of formative evaluations is to provide data so that improvements can be made as the course is being conducted, evaluative information should be collected and made available for use on a timely basis. a summative evaluation, on the other hand, provides an end-of-course review about course outcomes, specifically the quality and extensiveness of student learning in the course. formative and summative evaluations are not necessarily incompatible. even so, the requirements for doing adequately either formative and/or summative course evaluation are typically much more extensive than what is produced through standard student evaluations. by the time student evaluations are collected at the end of the semester, processed and reported back to the instructor, for example, it is too late to be of much contemporary formative value. in short, the timing integrating course 6 6 of such traditional student ratings of faculty makes them de facto summative, regardless of any formative aspirations. moreover, it is questionable how much of the information collected from students can be meaningfully used in the way of summative results. for example, how much does student opinions of instructor teaching practices (a common item included in surveys) say about the quality of student learning (a key criterion of a summative evaluation)? third, there is the janus-like problem of focus or construct validity and contamination. in this regard, the key issue is what is being assessed? (adams, 1997; scriven, 1995) there are several conceptually distinct aspects of the instructional, educational and learning process that can be assessed, including in-class teacher performance (particularly lecturing style and quality); course planning, design and management (which includes any number of less visible and out-of-the-classroom activities); and student learning. each construct requires its own distinctive evaluation procedures; for example, evaluating a teacher’s performance during a class is a much different task than evaluating how much students learned by the end of a course. without a clear definition of what is to be evaluated, the risk of contamination – gathering information about conditions irrelevant to the construct under study – increases. ideally, an effective evaluation procedure would include the following components. first, the coverage and focus of the evaluation should be well defined in terms of the construct being assessed and its formative/summative intent. a comprehensive and focused evaluation process would include a systematic means for compiling important and representative information about both instructor and instructional practices as well as showing linkages and connections between these integrating course 7 7 practices and student learning achievement and reactions to the course. second, the major instructional inputs and processes used in the course should be identified in order to gather information about how well those inputs and processes worked. third, this information should be collected at several times during the course and made available to the parties involved as quickly as possible. fourth, the data-gathering and compilation process should not be too time consuming and/or labor intensive. finally, a comprehensive evaluation would include multiple voices, including the instructors, students, and any other meaningful stakeholder (like peer evaluators, outside speakers, and so on). clearly, student evaluations do not meet these standards. there are at least three reasons why full and complete course evaluations meeting these criteria are not systematically conducted. first, a complete course evaluation meeting the standards noted involves gathering a lot of information, which can make it seem both labor intensive and time-consuming. second, systematic course evaluation is something that is typically not encouraged or rewarded in faculty evaluation systems (cerbin, 1994). for the typically harried faculty member, a task that requires a lot of time but which has little if any value in promotion and tenure criteria is a task that will probably not be done. third, there is a certain methodological inhibition to evaluation. unlike syllabi, test construction, and other aspects of instructional practice that are more frequently done and have exemplars of practice that can be easily inspected and copied, course evaluation suffers in comparison; there is an aura of procedural uncertainty that shrouds the assessment process. knowing what kind of information to collect, how to collect it and when is not immediately obvious or clear. integrating course 8 8 in addition, the first desiderata of correlating evidence on student learning with instructional practices are seldom met. cerbin (1994), for example, notes that assessments tend to focus exclusively on either student learning or faculty teaching, yet seldom are these two domains of evaluation examined in tandem. particularly difficult to examine are “how classroom practices contribute to learning outcomes” (p. 95). in this gap, cerbin recommends a “learning-centered assessment” that would look at the interaction between instructional practices and student learning. this approach is based on using a course portfolio design1 as a way to carry out a learning-centered assessment. for cerbin, a course portfolio should contain four core components: (1) a statement of the teacher’s assumptions about teaching and learning, goals for the course and the rationale linking the two; (2) an analysis of student learning based on class work and assignments; (3) an analysis of student feedback; and (4) a course summary. a portfolio can include examples of both instructional inputs and student outputs (like test scores, copies of graded student papers with comments, and so on). the result should be a four to seven page overall summary of the course. as a way to focus on how the instructional process impacts student learning, the portfolio approach offers an advance over traditional student evaluations. still, compared to the ideal checklist of elements of a thorough course evaluation, there are several potential limitations. the instructor tends to be the exclusive author and voice, particularly in terms of defining student learning. “the course portfolio has as its center of gravity the data the teacher gathers about students’ learning and development….” (american association for higher education, 2001). while student accomplishments should be included in the portfolio, less certain or systematic is the inclusion of students’ integrating course 9 9 voice. one set of guidelines from samford university (2001) suggests that examples of student voices like interviews, journal entries, student notes, or reports about instructional experiences are optional. further, course evaluations in general should be learning activities for the instructor about his or her learning product; when done well, a course evaluation (here, done as a portfolio) should be designed to help the instructor learn and improve. yet a learning-centered assessment process is summative in nature, putting together the portfolio after the course has been completed. while appropriate on its own terms, this approach does not lend itself to exploring and improving educational processes and their effects as they occur during a course and/or over a semester. as such, this approach would have limited utility as a formative tool to aide in improving the instructional process as it is occurring. the process of evaluation, assessment and critique can provide an opportunity for students to reflect on their own involvement in the learning process. yet, assessments do not necessarily involve students in critiquing or managing their own learning while in the course nor is assessment information shared with students. while certainly true in the case of student evaluations, this is also true of portfolios, and the potential value of portfolios as a device for aiding student learning is also missed. on the other hand, an additional benefit of a process that involves students in the assessment process would be to provide a means to help empower the students in gaining influence over their learning. are there other approaches to evaluation that may overcome some of these limitations and integrate evaluation efforts, that systematically includes the student’s voice on a more real-time basis with a reflective critique of instruction and learning, and that can also provide evidence of the learning that is taking place? can a procedure be integrating course 10 10 created that collects evaluative information from students on a continuous basis and then make that information available to students to assist in their learning and development while also serving as a guide for the instructor in course evaluation? can such a process be done in an economical, efficient manner? one potential solution to these questions is the use of a “learning history” approach to program evaluation. the remainder of this paper describes what this procedure is and the results observed in testing it in a graduate course. from this experience, the underlying characteristics of a learning history approach to evaluation will be identified in order to generalize how this procedure could be applied to course, instruction and student evaluation. a comparison of the three approaches to evaluation reviewed here is included in table 1. table 1 about here the learning history technique in the 1990’s, management consultants roth and kleiner (1998) began developing a new procedure to be used for organizational change and knowledge management in work organizations. calling it a “learning history,” this procedure involves collecting multiple perspectives on some corporate event, like a company reorganization or new product launch (kleiner, roth, thomas, & hamel, 2000). the resulting report captures the rich tapestry of thoughts and reactions experienced by people throughout the organization as the event unfolded. since the primary purpose in creating a learning history is to spur “reflective conversations,” the resulting document is made widely available to members of the organization and is used as the basis for a integrating course 11 11 collaborative review of the process in order to promote organizational learning. according to kleiner and roth (1997, page 176-177), “learning histories seem particularly effective at raising issues that people would like to talk about but have not had the courage to discuss openly.” as a technique, a learning history is defined by both the procedure used and the resulting report format. as originally formulated, the procedure was designed to elicit and record multiple voices of organizational members across the time span of an organizational change project. specifically, the procedure involves several steps, beginning with planning the boundaries and scope of the project. then, a series of “reflective” or depth interviews are conducted by a team of insider participants and outside researchers with a diverse and large sample of people. even though the primary data collected through a learning history are the reflections and observations of people involved in the program, other data, like documents, records, observations, and so on, might also be gathered. then, the data are scoured for common and recurring themes, and a first draft of the report (using the prescribed format) is produced. this report is double-checked for accuracy and significance by showing it to participants. finally, the report becomes part of a workshop where participants go over the history, consider how typical the events were, and look for how they can use the learning history to extract lessons from the experience. the format requirements for writing and presenting the report are specific. first, the program under study should be decomposed into its major segments, stages or episodes. these stages become provocatively titled chapter headings of the report. second, there should be a brief introductory segment to each chapter that describes the integrating course 12 12 nature of what happened, approximate timing, participants, conditions, and so on. third, the data should be presented in a two-column format using the following guidelines: (1) representative or significant quotes taken directly from the interviews should be listed in the right column; participants are identified by job title only; (2) the left column is reserved for interpretations, questions, observed themes or commentary about what is happening. in effect, the right column is for the voices of the participants, while the left column is for analysis. the document may also contain other information than quotes, such as memos, announcements, or other documents. (a copy of the full report generated for the course reviewed here is available at www.towson.edu/~aclardy. a sample of the learning history for one class session is enclosed in the appendix.) adapting the learning history a learning history approach satisfies many of the desiderata for course evaluation procedures. for example, the learning history for any particular course would be comprehensive, covering both instructor and instructional practices as well as student achievement and reactions to the course. second, the history should contain a complete and thorough description of both the inputs and processes used in the instructional process and in turn, there should be information about how well these inputs and processes worked. using procedures described below, all of this information can be collected and reported regularly during the course and should not create excessive demands on time or labor. this approach also gathers multiple voices for the evaluation. i modified and pilot tested the basic approach for creating a learning history in a graduate course i taught in the fall, 2000. there were 17 students in the class; the typical integrating course 13 13 student was a 30 year old, working adult. the course, entitled change in the workplace, is a required three credit hour course in towson university’s masters program in human resources development (hrd). the course met for two and one-half hours once a week for 14 weeks. i had previously taught the course four times. the course has a double focus: to review the historical and current changes impacting the contemporary workplace, and to teach principles of change management. in the field of organizational change, the twin concepts of a “learning organization” and “knowledge management” (davenport & prusak, 1998) are important and were an organizing theme for the course. within the hrd field, the idea of a “learning organization” has become very popular (see, for example, senge 1990). the fundamental concern is how an organization can improve its performance by a self-conscious effort to critique its internal processes and to find and adopt new, more effective practices. in short, the idea is to promote learning. one result of improved learning is that the organization’s storehouse of knowledge will continually be improved; in turn, this also means that knowledge should be better stored, coded and made accessible to all members of the system. to demonstrate these ideas in practice, i treated the course and all class members as an organizational change project with a learning history approach as a centerpiece to the process. the platform which enabled this demonstration to be tested was the learn on line, blackboard 5 (bb5) system, the e-learning based instructional system from blackboard, inc. this system allows an extensive range of course management and instructional options through web-based delivery. in 1999, towson university installed the blackboard 5 system; it is available to all students at no cost and can be accessed from either onor offcampus locations. bb5 allows the instructor to create a website integrating course 14 14 for the course. the homepage for the course website contains a set of menu options that allows the following functions: posting of course documents (like overheads, text files, case studies, and so on); providing assignments and assessments (such as course and individual surveys, complete with tabulated results); email communications to members of the class and individual emails directly with the instructor; a section for links on the worldwide web to other, related websites; and a “discussion board” format that collects and maintains a historical file of all individual comments and opinions about a “forum” topic. in addition to posting individual comments, the discussion board function accepts, stores and makes available for general inspection attachments such as student papers. in the initial pilot of this project, the primary element used to support the creation of the learning history was through the discussion board.2 over the period of the semester, students were required to make and post comments to specific questions on the discussion board. the first major modification of this approach to the normal learning history method was that no interviews were conducted; instead, student comments were taken from their posted comments and remarks. for example, during the first week of classes, students were required to post information about their learning goals for the course and about their personal and/or occupational background with organizational change. then, at three, equally spaced times during the semester, students were required to make entries about what they were learning to a forum called the knowledge management set (kms). these were preset discussion fora with the same basic question: what have you learned about organizational change to date? in addition, students had to post two class assignments to the discussion board. one was a group integrating course 15 15 project designing a diversity management program, and the other was a copy of a major paper on some organizational change technique they prepared for the class. in order to demonstrate the knowledge management intention for this course, students posted several of their individual learning products, making what they learned available to all. students were encouraged to download copies of other papers that they might want to use latter in their careers. students were expected to complete an assessment of the course and instructional practices. this assessment survey was posted and answered on-line (instructions in class and with the survey emphasized that all responses were anonymous). a week later, the class was given the composite data from the assessment as part of a instructional module on survey feedback.3 the class was held on thursday nights. in adapting the learning history approach, each class was treated as a separate event. to write the learning history, i would write a brief summary of what happened the night prior, along with any of my comments or analyses. in effect, in a second modification to the learning history technique, i wore two hats: course participant qua instructor and program observer. where possible, i would take student comments from various sources, including the discussion board or emails to me directly. even though student comments to the discussion board were posted for all to see, if their comments were included in the history, students were only identified as a “student” and not by their name. then, the updated learning history of the class would be posted as a course document every few weeks. students could see the evolving history of the course as it was being developed and written. three points deserve note here. first, it took about half an hour on average each week to write each entry to the learning history. other than typing my own comments integrating course 16 16 while writing the history, the inclusion of student comments was a cut and paste operation. students typed their own comments in response to the questions and assignments they were prompted to provide. as a result, a large of amount of material taken directly from students was included fairly easily. second, because the history was updated on a weekly basis, the result was an on-going and emergent document that provided an almost real-time account of what was happening in the course. by writing my comments within 24 hours after the class was completed, the record of what happened was reasonably contemporaneous. wearing the hat of instructor, my personal comments ranged from critiques of my own instructional practices, to observations about student involvement in the course, and to ideas for improvements to instruction the next time the course was to be taught. third, at the end of the semester, i used the complete document as the basis for evaluating what worked well and what did not in the course; these results will be discussed more fully below. results based on the pilot testing of a learning history approach to course evaluation, the following effects were observed. first, in the regular learning history protocol, the authors are different than the participants. in this class, though, i had to wear both hats as author and participant. one risk in participant observation studies is bias in what is seen and recorded, which may be colored by personal feelings and attitudes about the topic. while my affective responses were included (see the next section), i believe i completed the authoring duties without too much distortion. the main job of the author is to create a short summary of events, obtain and include comments from a variety of people integrating course 17 17 representing a range of viewpoints, and then offer interpretive comments and questions. the descriptive summary that introduced each section (i.e., each class) was a paragraph that chronicled the sequence of events as they played out in each session. since this summary was almost a narrative reporting of the agenda followed in the class, there was little opportunity for bias to be introduced. in addition, i made a conscious effort to find both typical as well as atypical comments and include both in the history. second, as would be expected from any truly formative evaluation, the act of preparing a learning history for the class had an impact on my performance in the class. the process of regularly writing about my intentions for, interpretations of and reactions to each class shortly after each session as well as seeing and including student reactions to those same events did affect my classroom instructional practices. for example, in one message, a student complained that the class did not always start precisely on time; this was due in large part to the vagaries of waiting on a working, commuting graduate population to park and arrive in class. nonetheless, i made a conscious effort to start on time thereafter, regardless of the numbers of students present. likewise, when i wore the participant hat and could present my own personal feelings about my experiences, the result was cathartic. i was able to vent any frustrations and/or doubts i was having with the class or my instructional design. for example, after class 8, i made the following entry after grading the first set of papers required for the course: i was disappointed by the general level of scope in the first papers, particularly given what i thought approached a cut and dried assignment…. more often than not, the reports were incomplete in covering [some of the key points covered in a history of workplace change]….in the syllabus, i should explicitly mention including a historical summary as part of the account of workplace change. after class 10, in an entry labeled “how much is me? how much is them?”, i noted integrating course 18 18 there is a recurring pattern of students not completing what i think are simple, clear and straightforward instructions. i’m getting both pissed and concerned about this. why is it so hard to get the basic instructions through? normally, i would not think of sharing this type of information with students, but, for me, this means of presenting my version of what was transpiring in the course allowed me to raise issues i was observing in the unfolding of the course. i could step outside the role of distant and isolated instructor to become more of a participant in the learning organization that was the class. third, the use of the bb5 system did not require a lot of additional time. on-line collection of student comments made it easy to cut and paste their voice into the history. by channeling student comments into specific sections of the website (the discussion board, in particular), i was able to quickly find examples of student voices. the convenience of this means of collecting information comes at the expense of an important feature of the traditional learning history method (personal communication, roth. 2000): there was no intensive interviewing of participants. further, a situation in which students are asked to provide critical information without anonymity does raise flags about compromised and self-serving reporting due to fear of reprisal or a desire to curry the instructor’s favor. i did not have any reason to believe, though, that punches were being pulled. for example, after class 10, one student emailed me with this comment: i have a concern that i have to “get off my chest” regarding the last project [a review of a student-selected book on change]. i felt insulted in the presentation on [the book] because i had the understanding that we (the grad students) had to pursue more proven or “researched” methods of change management – not a selfhelp book….i felt that [that student’s] selection was both degrading and diminishing to the level of studies that we have pursued in this course so far. to make matters worse, [this student] was bragging that he was reading the book for another class. integrating course 19 19 by the end of the semester, the complete learning history contained 37 basically single-spaced pages. after the history was finalized, i reviewed all the remarks and observations it contained. through this analysis, i was able to identify 16 specific and substantial items to change when i taught the course for the following semester. one would expect that while there might a large number of corrections the first few times a course is taught, one would also expect that the number of corrections should also taper off each additional time the course is taught. as noted, this was the fifth time i taught the course, a point by which one might expect the number of substantial modifications to be few. this suggests that the learning history approach may be a robust method for reflecting on practice in order to generate a continuously constant stream of ideas for course improvement and innovation. i do not believe i would have generated as many corrections in so systematic a manner through any other venue, nor, given my recordkeeping, would i have retained or stored these ideas in a single source for convenient future reference. the learning history was not without its drawbacks and limitations. even though the emerging learning history was posted for student inspection at the website every few weeks, i did not require them to look at it, and most students did not read the history as it was occurring. in addition, i did not include any of the following potentially helpful items in this history. first, the bb5 system provides various aggregate data about when students used the site and what features of the site they used. in general, neither did i include student comments picked up from class discussion. both kinds of data would have added to the depth of the account. second, i did not include actual copies of assignments (such as the diversity training plan the students had to design using virtual integrating course 20 20 groups) or the results of a survey which the students anonymously completed about me and the course. at least in this first attempt 4, examples of student learning were not as systematic as might be found in a portfolio. finally, the length of the resulting document (almost 40 pages) is a disincentive for other faculty to review. toward a general learning history procedure for evaluation based on this pilot test, a more general model for how to apply and use a learning history approach to course evaluation can be proposed. while it is conceivable that a learning history approach can be implemented manually, clearly the advantage goes to using some type of web-based electronic communication process. in principle, while a basic email system could serve the same basic purpose, a web-based instructional support system is the preferred. given a web-based e-learning system, the underlying design of a modified learning history approach to course and instructional evaluation can be built using the following six-point blueprint. first, the course itself would be divided into segments; depending on the type of course and scheduling cycle used, segments could be based on each class, each week, or each instructional topic (that might span class sessions). soon after completing each segment, the instructor would write a brief description of what happened in each segment. the description would report primarily on the instructional events used but could include other significant events, such as spontaneous discussions of current events or assignment clarifications. the two-column format would be used to record both participant observations and commentary on those observations. second, the voices of all classes of participants must be solicited and included. certainly, instructor integrating course 21 21 thoughts, assessments and feelings about the events within each segment would be expected. as a co-participant in the learning enterprise that is a course, the instructor should make comments and observations. at times, those comments may focus on student involvement in the course (what students were doing well or badly); at times, those observations could be self-reflective commentary on what the instructor was doing well or badly. third, student input would be expected. this can be done by requiring responses to specific questions at several times during the course. for example, students could be asked to identify their learning goals for the course or by asking about their expectations or preferences for classroom management or instructional practices. other prompts might ask students to identify the key lessons they have learned about a topic, to report on (and even include copies of) activities they are completing (such as lab work or artistic creations), to record their critical opinions about a subject, or to make suggestions and recommendations. some modest participation grade can be used to encourage thoughtful involvement. fourth, while more optional, it is possible to include data from survey assessments, like anonymous course evaluations. (in the bb5 system, survey data is anonymous; descriptive data is instantly available.) fifth, various kinds of student learning outputs, like copies of papers (perhaps already posted and available on-line), should be included. finally, the learning history of the class would be posted continuously during the semester. one requirement could be to have students read and respond to the history. a graphic depiction of this application model is shown in figure 1. figure 1 about here integrating course 22 22 further study the learning history approach to evaluation needs further study. for example, since a learning history is posted during the course, making instructor thinking and reactions more transparent and accessible, it would be interesting to see what effects, if any, a learning history might have on motivation, student learning or instructional quality. for example, would there be any differences in motivation or learning if students were given responsibilities for preparing or maintaining portions of the learning history for the course. a student in a subsequent section of this course made this comment about the learning history i had prepared for her class: “it really gave me a different perspective on the class. the comments and observations were surprisingly honest and forthcoming. i can see how this could be a useful tool …, it seems that it could make individuals more aware of and accountable for their actions/behavior.” does the use of an e-based medium make any difference in the quality, depth and “honesty” of student comments? would student comments made directly to an email prompt be any different compared to data collected in interviews by outside third parties? the learning history was used in a course because it was actually part of the instructional content (a technique for knowledge management and organizational change). can a learning history approach be used in courses where it is not part of the instructional content? that is, can it be used equally well in science, literature or physical education courses? this approach was used with a somewhat older and more mature population of graduate students. can it be used with a more traditional undergraduate population? finally, how do faculty and/or administrators compare a learning history report in value to the traditional student surveys and/or the portfolio model? since this size of a learning history might be four to integrating course 23 23 five times that of a course portfolio, it is important to look at the relative value of this approach in terms of quality, specificity and value of the information provided. improving the quality of learning and instruction is a lofty and important goal. unfortunately, the most common methods of student evaluations are also the weakest in terms of supporting the achievement of that goal. the portfolio approach, clearly an advance, is still somewhat limited in terms of timing and perspective. the learning history approach, using an on-line capability, offers a new way to move toward this goal by integrating various forms of evaluation into one common procedure. even though this approach is not without its limitations, there are also potential benefits. based on the first results of the pilot test reported here, a learning history can be a new tool in the quest for educational excellence and deserves further consideration. integrating course 24 24 endnotes 1. this procedure has been amplified and promoted by the american association for higher education’s teaching initiatives project (www.aahe.org/teaching/teaching_initiative_home.htm). braskamp and ory (1994) distinguish between course portfolios (for one course at a time) and teaching portfolios (a composite of several courses). 2. unavailable and hidden to student view in bb5 is a set of usage reports that indicate when students use the system, how often they access various elements, and who completed what assignments. while this information could be included, it was not in this pilot test. 3. survey feedback is a technique of organizational change. members of a social system complete some kind of attitude or opinion survey, say about the leadership practices of their boss, or their job satisfaction. those results are compiled and then a summary of the results is presented back to the members, typically meeting as a group. the members then study the data to recommend changes and improvements to the system. 4. i continue to use a learning history for the same reasons in the same course. now, i include a variety of student products and outcomes, like examples of papers, assignments and survey results. integrating course 25 25 references abrami, p.c., d’apollonia, s., & cohen, s. (1990). validity of student ratings of instruction: what we know and what we do not. journal of educational psychology, 82, 2, 219-231. adams, j.v. (1997). student evaluations: the rating game. inquiry, 1, 2, 10-16. american association for higher education. (2001). teaching initiatives. retrieved on june 12, 2001 from http://www.aahe.org/teaching/teaching_portfolio.htm. birnbaum, m.h. (no date). a survey of faculty opinions concerning student evaluations of teaching. retrieved on september 24, 2001 from http://psych.fullerton.edu/mbirnbaum/faculty/3.htm. blunt, a. (1991). the effects of anonymity and manipulated grades on student ratings of instructors. community college review, 18, 4, 48-53. braskamp, l.a., & ory, j.c. (1994). assessing faculty work, enhancing individual and institutional performance. san francisco: jossey bass. cerbin, w. (1994). the course portfolio as a tool for continuous improvement of teaching and learning. journal of excellence in college teaching, 5, 1, 95-105. davenport, t.h., & and prusak, l. (1998). working knowledge, how organizations manage what they know. boston: harvard business school press. kleiner, a., & roth, g. (1997). how to make experience your company’s best teacher. harvard business review, 97, 5, 172-177. kleiner, a., roth, g., thomas, t., & hamel, e. (2000). oil change: perspectives on corporate transformation. new york: oxford university press. integrating course 26 26 roth, g.l. (2000). constructing conversations: lessons for learning from experience. organization development journal, 18, 4, 69-78. roth, g.l., & kleiner, a. (1998). developing organizational memory through learning histories. organizational dynamics, 26, 43-60. samford university. (1998). the samford pbl [problem-based learning] initiative, course portfolio contents. retrieved on june 12, 2001 from http://www.samford.edu/pbl/aboutsu3.html. scriven, m. (1995). student ratings offer useful input to teach evaluations. eric/ae digest. retrieved on september 24, 2001 from http://www.ed.gov/databases/eric_digests/ed398240.html seldin, p. (1984). changing practices in faculty evaluation. san francisco: jossey bass. seldin, p. (1993, july 21). the use and abuse of student ratings of instruction. the chronicle of higher education, a-40. senge, p.m. (1990). the fifth discipline, the art and practice of the learning organization. new york: doubleday. stratton, r.w., myers, c.m. & king, r.h. (1994). faculty behavior, grades and student evaluation. journal of economic education, 25, 1, 5-15. sustaining change by learning from collective experience. retrieved on july 9, 1999 from http://www.learning.mit.edu/res/projects/change.html. integrating course 27 27 appendix. sample record from the learning history document _____________________________________________________________________ class 7. let the games begin! after collecting the assignments due for tonight, we spent about 10 minutes going over assignments again. dr. clardy then made the transition into the shift in the focus of this course from looking at the forces impacting the workplace to the question of how to manage change. tonight was the inauguration of my change management model. we covered the material up to resistance, then did the assignment about force field analysis for poor levels of supervisory performance. this took about 45 minutes. we ended by the assignment 3 project teams meeting again. frustration with students understanding the assignments instructor: again last night, we killed about 15 minutes at the start of class going over what the assignments will be. this included me spending 10 minutes at the break going over assignment 2. all that plus the first few papers i’ve graded for assignment 1 were not complete. i’m getting very frustrated. how can we be spending some much time going over the same assignment expectations and still not getting it? i don’t know how i can be any more specific than what i’ve already included in the syllabus, plus the supplemental resources (professional practice paper and examples of prior papers) in bb5. this is a real problem and i need to investigate what’s causing it. we’ve probably killed close to an hour of class time (excluding the first class) beating these dead horses. i’m running out of patience on this matter. instructor: the force field analysis exercise probably should be presented as text, rather than as a hypothetical example circling around examples of poor supervisory performance. or, perhaps better still, keep the brainstorming listing of poor supervisory practices, but provide a simple version of the conditions in place in the organization: for example, here’s the supervisor evaluation form, here’s a story from the culture, etc. this spoon feeds the answer a bit, though. integrating course 28 28 dr. clardy sent email announcements to all students with a notice about an upcoming shrm meeting as well as a notice about having posted learning organization overheads. also, the instructor survey ratings were obtained. there were 11 responses of 16 students. how to be able to use survey data most effectively? this suggests the importance of training and discipline in: * rational emotional therapy and dealing with irrational beliefs * humility things done well:  humor  encouraging participation  feedback  use of exercises and activities instructor: opening up personal survey data is always a challenge. curiously, i was more nervous and concerned about seeing this data than i am when i get course survey data (from the university). possible reasons: it’s more timely and it involves a set of factors which were developed from and with the students. interpreting results is always a challenge. anything less than perfection is riveting and produces feelings of despair or defensiveness or both. how to train oneself to accept the information humbly and to use it constructively without getting defensive or depressed? after 25 teaching and training, that skill still eludes me. student 1. encourage student participation. offer ideas in group discussion. ie: have you thought about it this way... student 2. he should continue to show videos and have small group discussions because the hour is late and i'll fall asleep if he doesn't. student 3. he motivates the class. often uses humor to motivate the class. always answer class questions. provides feedback on all assignments. i enjoy the mini projects that assigned during class. student 4. very good use of class time; exercises are engaging and appropriate (dr. clardy is very good at showing connection/relevance); sometimes feel like a guinea pig, although stretching comfort zone, it is challenging and thought provoking. student 5. willingness to adjust class schedule and due dates to class needs. ability to make material interesting incorporation of exercises and media, etc. student 6. i enjoy dr. clardy's playfulness and creativity in class, and it is a change for me, most instructors being more focused on lecture. he seems very contemporary, very concerned with the well-being and fairness of each of the students. he keeps it light, which is a nice change of pace. integrating course 29 29 things to change or improve: several comments about the amount of web-based activity and the fact that they are ungraded is student saying that more time should be devoted to group work and discussion? it is difficult to identify action suggested if it is not listed. student 1. start on time & end on time. student 2. he should stop giving so many course info assignments. the projects and the readings are plenty of work in themselves. student 3. way too much work on the web for students. even if dr. clardy sees this as the way the world is going in terms of communication, it might be too soon to introduce it. i suggest he reduces the amount of web assignments for the students. perhaps he is before his time? student 4. there are assignments on course info that are not graded, but they seemed like a mini project. course info may be over-used. student 5. i think that this class has a lot to offer and i would like to learn from some of my fellow students (including & beyond the syllabus). they've shown some great ideas and creativity in group work and class discussions. this class has a lot of people who have diverse experiences and backgrounds. no responses or not sure: 4 instructor: i’m wondering if there were any concerns from students about the anonymity of their responses and whether they hedged their answers in any way. students were to complete the first knowledge management set (kms) entries by this time. all but 3-4 students had done so. a summary of their comments is shown below. the larger context in which change is to implemented must be taken into account. this includes the global setting as well as the full range of organizational practices that either support or inhibit change. same point by another student. student 1. in a more micro view of change, individualistic resistance to change must be approached by a strong organizational culture and action. how often organizations talk about change management, yet do not set up the environment or reinforcers for such behavior. change has to be managed at all levels of the organization. i feel that two important characteristics of organizational change is proactive responsiveness and adaptation. by “proactive responsiveness” i mean that an organization may project what customers want and test the product in all forms before distribution, … and fix them in a timely manner. this can apply not only to products issues, but also management policy. adaptation is simply this: moving out the “comfort zone” of business practices. in the technological revolution going on, it is difficult to develop an original idea and keep it as one. as soon as it gets out, it is changed and adapted into hundreds of other forms. integrating course 30 30 change applies to the traditional role of hrm as a rule maker. must keep some ethics as rules. creating alliances to help make change. imperatives for change call for new or different roles for hr. resistance to change is seen in the ford motor example. resistance is a recurring theme in these comments. yet companies need to change in order to survive. another student agrees about the extent and amount of change taking place. recognizes a contingent approach to change management. people involvement is important. techniques: swot and smart student 2. human resources has seemed to exist in it's own world for years. acting as the creators and deliverers of "the rules" have been the norm…. business partnering within an organization can help to facilitate this move and link hr with the rest of an organization. this partnering can promote organizational change. but, sometimes it seems that we are so eager to change an organization and move forward, that we may forget the "rules" we must keep in order to remain ethical. student 9. i agree with jeannine in terms of ethics & the role hr. i think that for years we've had a mechanistic approach to change (linear cause and effect and "cog in the wheel"), but we're moving into an organic approach (everything is connected and affected by each other and the environment). this certainly opens hr into a role as team member, diplomat, consultant & facilitator. this shift means that the hr professional not only needs to know employee/employer rights, but also have developed skills to work effectively in teams, skills in diplomacy, and interpersonal communications student 4. so far, i have learned a number of important ideas about change: 1. people are extremely resistant to change (as in) the movie that we viewed about the ford motor company, we saw the extent to which the employers resisted the change. 2. while companies like to bury their heads in the sand when it comes to change, analysis and change is the key to a company's survival. if they don't grab the opportunity to change, others will pass them by and they will soon be non-existent. 3. nowadays, it seems that organizations are in a constant state of change so that it is almost dizzying to think about. 4. it is vital to weed out the important changes an organization must make from the trends and styles of society. to follow every whim of society is a sure key to failure. student 5. i feel the biggest thing i have learned so far is that organizational change is not easy to understand or manage. i have also realized that the changes in my office have not been handled in the best ways. every situation requires a different approach and a different solution. i think the most important thing is to involve people in the change process. no one likes to be forced around when it comes to change. student 6. the hardest part about change is preparing for it. i think that in order for companies to be successful in their change, they need to take into consideration two models (swot and smart). first the companies need to figure out their strengths, weaknesses, opportunities, and threats. second i think that the companies need to understand whether or not their change strategies are specific, measurable, achievable, realistic and timely. integrating course 31 31 three generators of change: employers, employees, government. must look at broad complex of factors. problem: how to generate change? how do you create a motivation for change, particularly among longterm staff? is it a “discussion” board or a “posting” board? student 7. i believe there are three different types of organization change. the first type of change is being lead by the organization. this incorporates organization hierarchy, policy, procedures, missions, anything that is catered to the organization. another type of change is lead by the employees (ford). employee influenced organization change would include unions and civil rights groups. finally, government mandates is the third method of change. examples of mandated change would be eeo, aa, fmla, ada, etc. student 10. so far, i have learned that diagnosing, planning, conducting and/or evaluating organizational change has to encompass many factors outside of the change itself. what is/will be the impact of the environment? technology? people involved? etc. i've never considered the components that forced the transition from colonial to industrial to contemporary workplaces. and i've been a part of the change due to technology, so i've not known what it would be like to work without a computer and networks. and i believe honesty is one of the most important concepts that should be involved in any change process. secretive or dishonest methods of instituting change just make people resist the ultimate change. student 13. the question i have about organizational change appears to be easy to answer. what can a manager do when an organization implements change in procedures and policies -but nothing changes! for example, a new program has been implemented in an organization along with new staff to train the old. the problem is the veteran employees are resistant to the change-what does a manager do to change their mindset? management does not want to terminate the employee/employees because they are valuable to the corporation. instructor: i’m noticing that the students as a rule are simply posting their comments and not engaging in any discussion on the items posted by others. on the other hand, i’ve resisted responding to each posting, other than the diversity reports. i probably need to prime the pump but responding more frequently. integrating course 32 32 table 1. a comparison of three evaluation procedures components of an effective evaluation process student evaluations portfolio learning history well-defined focus questionable yes, summative only yes, both formative and summative major instructional inputs and processes used in the course identified no yes yes, along with ongoing critique of those processes regular collection of information as the course unfolds no no yes easy to use process yes no yes includes multiple voices no may yes integrating course 33 33 figure 1. a blueprint of a learning history course evaluation structure regular weekly inputs instructor inputs:  descriptions of what happened in each segment  assessments and commentary course segments of the course (here, weekly class sessions) planned periodic inputs during the course student voice inputs student learning goals and expectations open discussion boards email communications periodic reports on what they have learned posted student assignments with instructor comments survey data summative information and output products system administration information usage data from the system: timing, participation examples of student achievements copies of papers and reports test scores q&a forum 503 service temporarily unavailable 503 service temporarily unavailable nginx/1.14.1 503 service temporarily unavailable 503 service temporarily unavailable nginx/1.14.1 critical thinking and student success 1 a model for student success: critical thinking and “at risk” students randall e. osborne indiana university east critical thinking and student success 2 abstract there appears to be a significant gap between faculty expectations for incoming college students and these same students perceptions of their abilities. incoming college students are not very confident of their critical thinking abilities, yet faculty expect students to enter college already being able to critically evaluate information and to reach conclusions based on a critical analysis of the data. the current study challenges the preconception that critical thinking cannot be taught and delineates a model for critical thinking that can be employed regardless of one’s discipline. outcome data strongly suggests critical thinking can lead to both proximal and distal increases in student success. critical thinking and student success 3 introduction in a recent poll of campus faculty, osborne (1998) noted a large discrepancy between faculty expectations for incoming students and incoming students’ perceptions of their own abilities. in particular, faculty expect college students to: (1) critically think, (2) manage their time, (3) monitor their own stress levels, (4) solve problems, (5) clearly articulate what they do and do not know, and (6) prioritize tasks so more important tasks are afforded more time. in this same poll, however, first year students cited the following areas of weakness in their own preparation for college: (1) poor time management skills, (2) ineffective methods for coping with stress, (3) frustration with communication abilities, and (4) poorly developed critical thinking skills. many students, in fact, suggest critical thinking is not only under-appreciated in high school but also actually punished. one student said it best when he recalled a comment a high school teacher made on an essay exam of his. in response to this student’s effort to speculate on the causes of the issue being critical thinking and student success 4 addressed, the teacher wrote in the margin, “do not tell me what you think, tell me what i told you to know.” although we do want students to know what we believe it is important for them to know, should we actively discourage their efforts to place their knowledge within a context? research suggests, in fact, that first year students are significantly less likely to be successful in making the transition to college level work when they take introductory courses that do not require written work (e.g., boice, 1990; rickabaugh, 1993). additional research shows that emphasizing study skills, as many “first year experience” or “transition to college” programs do, actually does little to promote student success unless metacognitive skills are taught as well (e.g., flavell, 1979; gardner & boix-mansilla, 1994). yet, when discussing critical thinking as a method for assisting a first year student’s transition into college, it is not uncommon to hear faculty suggest critical thinking cannot be taught. indeed, even students appear to enter the college critical thinking and student success 5 environment believing you either have what it takes or you do not (e.g., sydow & sandel, 1996). in an effort to increase the retention of first year students, a pilot program was developed pairing an introductory psychology course with a course on critical thinking (browne & osborne, 1998). over a five-year period, students completing this pairing of courses were tracked and both proximal and distal student success measures were gathered. the focus in this article is two-fold. first, information will be provided to challenge the prevailing attitude that students cannot be taught to critically think or the perception that, at the very best, if students can be taught to do so it cannot be accomplished within one semester. second, longitudinal data from this pairing of courses will be provided to delineate the long-term benefits of promoting critical thinking skills in “at risk” students. in a longitudinal study of “at risk” students, browne and osborne (1998) established the link between the development of critical thinking abilities and longterm measures of student success. students enrolled in randomly selected sections critical thinking and student success 6 of an introductory psychology course also participated in a two credit-hour critical thinking laboratory. activities in the lab were linked to the weekly content of the introductory psychology course. students were placed in these special paired courses based on college entrance assessment scores, low high school ranking, and/or having already been placed on academic probation. the challenge became one of delineating the process of critical thinking. surprisingly, the literature is very sparse in terms of information about what critical thinking is or how it can be taught. although there are a few well known exceptions (e.g., chaffee, 1994, halonen, 1995 & smith, 1995), critical thinking is ill-defined and even more difficult to incorporate into the general education of first year students. rather than adopt one of the discipline specific approaches to critical thinking (e.g., halonen, 1995) browne and osborne (1998) chose to use applications from the the education literature. benjamin bloom (1956) articulated six cognitive levels for student learning. these levels, then, became the building critical thinking and student success 7 blocks upon which browne and osborne built their “critical thinking as process” approach. bloom’s cognitive levels include: (1) knowledge – facts, (2) comprehension – an understanding of those facts, (3) application – an ability to utilize an understanding of that information for addressing a problem, (4) analysis – an assessment of what aspects of that knowledge are meaningful, (5) synthesis – a reintegration of those pieces into a more meaningful whole, and (6) evaluation – an assessment of the learning that has taken place and a comparison of what is now known versus what was initially known. exams and other classroom assessment methods that require recitation of facts only require students to demonstrate the first level of understanding in bloom’s taxonomy. browne and osborne, therefore, developed in-class activities, demonstrations, and assessment rubrics requiring students to demonstrate an ever-increasing ability to go beyond what is simply known. an example critical thinking and student success 8 before proceeding with a discussion of the outcome data, it would be useful to provide an example of an activity that encourages students to move up the cognitive levels of bloom’s taxonomy. a technique employed during the first week of the course was called, the costs and benefits of critical thinking (see osborne, laws & weadick, 1999 for a detailed description of this activity). this activity was designed to illustrate the relationship between the effort associated with critical thinking and the benefits gained. students were placed in five-person working groups. each group was given $240 of play money, some sketchy information about a sniper killing incident, and the opportunity to purchase additional information to assist them with the task of answering the question, “why did rick kill those people?” three categories of information were available for purchase, (1) $25.00, (2) $40.00, and (3) $75.00. the more expensive information was said to be more critically important for answering the question. critical thinking and student success 9 student groups then spent time deciding how to spend their money (the analogy was made to this paralleling and “investment” in their critical thinking), purchasing information and preparing an in-class presentation as to why rick killed people. students were given 30 minutes to purchase their information and develop their response to the question. groups could earn points in two ways. first, they would earn points for having money left over. if there were six groups, for example, the group with the most money left over would earn six points, the next highest amount of money left over would earn five points, and so on. additionally, groups could earn points based on the total critical thinking score assigned to their presentations by the judges. judges were given bloom’s taxonomy and instructed to utilize it in assessing the critical thinking demonstrated within each presentation. the group receiving the highest critical thinking score would earn six points, the next highest group score would earn five points, and so on. data from the groups was then plotted to show the relationship between the costs of critical thinking (the spending of the money) critical thinking and student success 10 and the benefits of critical thinking (the critical thinking score). each time this technique was employed, the data showed almost a perfect inverse relationship. the more money the groups spent, the higher their critical thinking scores tended to be. it is important to note that the information students employed in reaching their conclusions about why rick killed people, is less important than how that information is employed. as an example, one group purchased a $25.00 piece of information revealing rick drove a “beat up black ford pinto.” a critical thinker would be expected to set this information aside and reach the reasonable conclusion that, at least in isolation, this information is not informative for answering the question as to why rick did what he did. the group who purchased the information, however, proceeded to suggest that this was important because “the fact he chose to drive a black car indicates that he was depressed” (represented by the black color of the car), and “in this state of depression he decided to kill himself and take others with him.” critical thinking and student success 11 not surprisingly, this analysis of why rick did what he did received a very low critical thinking score by the judges. other groups showed a similar unwillingness to set aside information they had purchased and this became a focal point in the discussion that followed the activity. in a general sense, people may be unwilling to “give up” something they have invested a lot of time, effort and/or money on. this certainly could explain why some groups utilized apparently useless information at the expense of their own critical thinking scores. in contrast, some groups received very high critical thinking scores despite the fact that most of their information did not “converge” on a clear answer. one group, for example, speculated that he may have had some form of brain damage (they had information stating that on the morning of the murders he suffered three blackouts). they went on to speculate that it could be a brain tumor of some kind that was affecting his perceptions of reality. although they had very little information to build this case upon, their honesty in drawing their conclusions struck the judges as particularly indicative of critical thinkers. this group ended their critical thinking and student success 12 presentation by stating that their analysis was entirely speculation and could only be corroborated with an autopsy. it is worth noting that a piece of information that might have been purchased stated that an autopsy after the killing spree revealed major damage to his amygdala (a brain structure responsible for a person’s levels of aggression and fear). this activity was then followed by an in-class discussion about why students should invest in critical thinking. the costs, of course, include: (1) critical thinking is time consuming, (2) it requires a great deal of intellectual effort, (3) it requires thinkers to be open and honest about what they do or do not know, and (4) it demands that one not go “beyond the data.” the benefits discussed with students include: (1) an ability to understand, apply, and analyze what is known, (2) a clearer understanding of what is not known, and, ultimately, higher grades, (3) more confidence in one’s knowledge, and (4) an enhanced ability to discern what information still needs to be discovered in order to be confident in drawing conclusions. critical thinking and student success 13 the following class period students were required to bring in syllabi for their courses for the semester. a content analysis of these syllabi showed a very clear expectation on the part of faculty that students would be able to demonstrate critical thinking. such statements as, “students should be able to apply course principles to real world issues,” and “students are expected to develop informed opinions and to separate opinion from fact” reinforce the point that faculty expect students to already be able to do the kind of thinking outlined by bloom. other course activities included assessing news programs, persuasive speeches, and newspaper/magazine articles on bloom’s levels. students were required to read a magazine article, for example, and assess the degree to which the author had or had not, demonstrated critical thinking according to the six level model being used. as a final project in the course, groups were assigned chapters from the introductory psychology course textbook. it was their responsibility to learn the material, and prepare an in-class presentation of that material. these group presentations had specific criteria such as inclusion of at least three different visual critical thinking and student success 14 aids, and a minimum of two handouts. students in the course, then, utilized a scoring rubric based on bloom’s taxonomy to assess the critical thinking level demonstrated by the groups in their presentations. this assessment was a major component of the students’ final grades in the critical thinking laboratory. the data proximal measures were gathered to assess the impact of this critical thinking course on students. these measures included: (1) grade point average in the paired introductory psychology course, (2) overall gpa for the semester, and (3) number of students in the paired courses receiving d, f, or withdrawal grades. these data were compared to a matched sample of students completing the introductory psychology course during the same semesters but not completing the critical thinking lab. the samples were matched for gender, high school rank, number of credits hours completed, declared major, and admission placement scores. critical thinking and student success 15 proximal data showed clear effects of the critical thinking lab. students completing the paired courses received higher grades in the introductory psychology course (3.0 versus 2.19 on a 4-point scale), received fewer d, f, or withdrawal grades (24% versus 46%) in the introductory psychology course, and had higher overall gpa’s for the semester in which the introductory course was taken (2.7 versus 2.22 on a 4-point scale). more important than these proximal indicators of success, however, were the distal or long-term changes that were tracked. students completing the paired courses were significantly more likely to re-enroll the next semester (80% versus 56%), were significantly more likely to have graduated within the next five years (45% versus 33%), and were significantly more likely to declare majors within the behavioral and social sciences (14% versus 8%). discussion the data from this study appear to support the contention that critical thinking is a process that can be taught. additionally, the data strongly suggest critical thinking and student success 16 that the process of critical thinking can be taught and modeled well enough in one semester to initiate some long-term change. one primary challenge to employing the process of critical thinking to promote student success, however, is convincing faculty and students alike that critical thinking can be taught. even with a model for critical thinking in hand, however, student success is not guaranteed. students need ongoing and frequent practice with applying the critical thinking model, and they need practice with applying the model in diverse ways. discipline-specific applications on the process of critical thinking may be less effective in the long run than requiring students to implement critical thinking in ways that are relevant to their daily lives. one student, for example, suggested that the critical thinking course destroyed her ability to get pleasure out of watching television because she felt most programs were insulting her intelligence. this student went on to lament television commercials and asked, “just how many dentists did they survey when they decided four out of five dentists surveyed recommended a particular gum?” critical thinking and student success 17 although the class found the comment to be comical, it also illustrates an ability on the part of this student to critically evaluate what others are asking her to believe. several faculty commented on the poll mentioned earlier that they want students to become informed consumers of information. it appears the aforementioned student, at least, has made progress on that goal. critical thinking is a process that must be nurtured, encouraged, and rewarded. the costs are high. it is difficult for students not because they cannot do it but because they have not been encouraged to practice it. one student proclaimed after the first class period, “this kind of thinking makes my head hurt.” from that day forward, a bottle of ibuprofen sat on the front desk to remind the class that the outcome is worth the pain. in the end, this student graduated and recently sent a note to the author stating, “i just had to tell you that i got a big promotion at work. the boss said my ability to critically think was the primary reason he recommended me for the promotion. it really was worth the headaches.” references critical thinking and student success 18 osborne, r.e. (1998). first year experience programs: what faculty and students believe students know. unpublished data. indiana university east. bloom, b.s., englehart, m.b., furst, e.j., hill, w.h., & kratwohl, o.r. (1956). taxonomy of educational objectives: the classification of educational goals. handbook 1: the cognitive domain. new york: longman. boice, r. (1990). faculty resistance to writing-intensive courses. teaching of psychology, 17, 13-17. browne, w.f., & osborne, r.e. (1998). critical thinking: developing a model for lifelong learning. paper presented at the international conference on the application of psychology to the quality of teaching and learning, hong kong, china. chafee, j. (1994). thinking critically. boston, ma: houghton mifflin company. flavell, j.h. (1979). metacognition and cognitive monitoring: a new area of cognitive-developmental inquiry. american psychologist, 34, 906-911. critical thinking and student success 19 gardner, h., & boix-mansilla, v. (1994). teaching for understanding – within and across disciplines. educational leadership, 51, 14-18 halonen, j. (1995). the critical thinking companion for introductory psychology. new york: worth publishers. osborne, r.e., laws, j., & weadick, k. (1999). the costs and benefits of critical thinking. in benjamin, l.t., nodine, b.f., ernst, r.m., & broeker, c.b. (eds.), activities handbook for the teaching of psychology, 4th edition, 11-17. washington, dc: american psychological association. rickabaugh, c.a. (1993). the psychology portfolio: promoting writing and critical thinking in psychology. teaching of psychology, 20, 170-172. smith, r. (1995). challenging your preconceptions: thinking critically about psychology. pacific grove, ca: brooks/cole publishing. sydow, d.l., & sandel, r.h. (1996). making student retention an institutional priority. paper presented at the national institute for staff critical thinking and student success 20 organizational developmental conference on teaching and leadership excellence, austin, tx. reviewers questions 1. what curriculum changes could this research suggest, given that you have demonstrated that critical thinking can, indeed, be taught? i think just as much in non-education fields as in the more obvious teaching professions. 2. we could extend this discussion to high school teachers as well. students come to the university already having been taught not to think, just to recall. this research suggests that we would serve students well to examine how high school teachers are teaching. 3. what steps could be taken to convince faculty to utilize these approaches? 4. what do you do when a group scores high in critical thinking skills but is unsuccessful in solving problems? 5. many faculty are trouble by the number of at-risk students admitted to our universities who drop-out or flunk-out during the first or second semester. some fail because of a lack of maturation, others because of a lack of general academic ability, and some because of difficulty with critical thinking. is it the responsibility of university professors to address deficits in critical thinking, self-regulation of study skills, and other tasks associated with priorities and goal setting? how should university professors facilitate the development of these skills and attitudes in their classes, particularly classes intended primarily for freshman and sophomores? the journal of scholarship of teaching and learning (josotl) volume 3, number 1 (2002) copyright 2002, the trustees of indiana university. except as may be expressly provided elsewhere in this publication, permission is hereby granted to reproduce and distribute copies of individual works from this publication for nonprofit educational purposes, provided that copies are distributed at or below cost, and that the author, source, and copyright notice are included on each copy. this permission is in addition to rights of reproduction granted under sections 107, 108 and other provisions of the u.s. copyright act. sponsored by uce acet, and indiana universit uth bend the journal of scholarship of teaching and learning t, f y so the scholarship of teaching and learning: facilitating adult learning barbara j. daley adult and continuing education department of administrative leadership university of wisconsin – milwaukee p.o. box 413 milwaukee, wi 53201 bdaley@uwm.edu abstract in the scholarship of teaching and learning (sotl) project i developed, i chose to investigate how constructivist teaching strategies influence the learning processes of adult students in higher education. i chose to teach two groups of students to use a constructivist strategy called concept mapping. they used this strategy during the courses i taught in the first semester of this study. then, i followed these students during semester two to see if they continued to use concept maps and to find out how the use of maps impacted their learning. to accomplish this, i checked the students’ first map and final map from semester one, and their maps, if any, from semester two. in addition, we interviewed the students at the end of semester one, and again at the end of semester two, to find out how the use of mapping affected student thinking and learning. results indicate that 65% of students continued to use maps in the second semester and all students reported changes in their thinking. mailto:bdaley@uwm.edu framing the question the question of how adults learn has always held a deep fascination for me. i believe this is because i started my professional career as a nurse and my first exposure to formal teaching was as a staff development instructor in an acute care setting. in that setting, i was always intrigued by the fact that staff development programs for adults produced such varied and unpredictable results. some adults used the information presented and some did not. the raised two questions for me, “how do these adults learn within the context of their practice?” and, “what can i do to facilitate that type of learning?” as i moved on in my career, i decided that i wanted to understand this learning question in a much deeper sense and, thus, chose to pursue doctoral work in adult and continuing education. ultimately, i accepted a faculty position in that discipline. however, before that time my teaching experience included working with adult students in community college and university settings. often, i saw these adults enter higher education relying solely on learning strategies that had worked for them in the past. most often these strategies were rote learning, including memorization, recall of information and passive learning. i began to think about how i could not only teach the content in my courses, but how i could also help adults to understand their own learning processes. from my work as an adult educator, i knew that adults had experiences that were rich resources for learning and, yet, i often saw adults avoid using that experience in a higher education setting. barbara j. daley the scholarship of teaching and learning: facilitating adult learning page 15 journal of scholarship of teaching and learning (josotl), volume 3, number 1 (2002), pp. 14-23 © 2002 board of trustees of indiana university at about that same time, i came across the work of stephen brookfield. brookfield (1995) advocates the position that if we are to become critically reflective teachers, we need to examine how we as teachers learn from our students, from our autobiography, from theory, and from our colleagues. i decided to take brookfield’s work seriously and began to think about my own experience as a learner, or my own autobiography. when i looked back at my own learning, i recognized that it was in my doctoral program at cornell university that i began to understand my own learning. in that program, i was fortunate to be able to work with dr. joseph novak (1984, 1998), and to learn more about constructivist learning and the use of concept mapping. as i began doing concept maps, i recognized that i did not understand how to link concepts. additionally, i had not learned how to search out interconnections across bodies of knowledge, nor had i learned how to develop a shared meaning with the instructor. using concept maps helped me to understand how i learned. as i developed this understanding, i started to use concept mapping in the courses i was teaching and each time i used it, i saw significant changes in how students learned. i also saw resistance from students and from other faculty in the use of this particular strategy. i would get questions from both students and faculty such as: “why are you doing this?” “does doing concept maps really make a difference?” since i was teaching in a college of nursing at that time, a few colleagues and i, who were interested in mapping decided to study the learning outcomes for nursing students using maps (daley, 1996; daley, et al, 1999). in this work, we did see changes in learning and were able to document some information on how mapping influenced the learning process. the issue for me then took on another aspect. did students, once they left the courses that i taught using mapping, continue to use that strategy? i was curious about long-term student changes where mapping was not required. to me this question is very important because if the purpose of the mapping is to help students understand their own learning and to foster a “learning how to learn” (novak, 1984) approach, then it seemed really important to know if they continued using this strategy. also it seemed important to know if mapping was incorporated into their learning and thinking activities irrespective of its use in specific courses. at about the time i was pondering this question, the university of wisconsin – milwaukee, under the direction of tony ciccone with the center for instructional and professional development (cipd), began participating in the scholarship of teaching and learning program with the carnegie foundation for the advancement of teaching. cipd sponsored a center scholars program and provided funding for me to investigate how adult students learn with concept maps. the funding allowed me to follow students for a year and see what impact the maps had on their learning. context of the work: adult and continuing education graduate program i currently teach in an adult and continuing education graduate program. the students in our program are all masters or doctoral students who come to us from a variety of disciplines. many of our students are trainers in business and industry, staff developers in health care, faculty in vocational technical institutions or teachers of adults in community-based agencies. our students, all adult learners themselves, are on average 35-40 years old. i chose to use two different courses in this project. the first is a masters degree course which is the initial course our students take when they enter our program. since these students are very new to graduate education, they are often concerned that they do not have the ability to succeed in graduate school, and are often unsure of the requirements. many have been out of school for a number of years and feel their academic skills are a bit rusty. the second course in this project is an elective in our program attracting predominately doctoral students. the second course was also taught completely on-line, with only one faceto-face orientation meeting. in each course, i taught students to use concept maps by first having them read literature on concept mapping. then we discussed mapping, either face-to-face or on-line, and they practiced developing maps by mapping out an article from their reading. in the first course, students did concept maps on their reading as a way to frame a paper on their development as adults, and in classroom exercises as a way to link conceptual material from the course to their own experiences. in the second, the on-line course, students did concept maps of case studies and used the maps to link the case study to their reading. additionally, they mapped out and compared and contrasted two books. finally, students in the second course created concept maps of their readings. barbara j. daley the scholarship of teaching and learning: facilitating adult learning page 16 journal of scholarship of teaching and learning (josotl), volume 3, number 1 (2002), pp. 14-23 © 2002 board of trustees of indiana university the face-to-face course met once a week during the evening. in this course, students developed maps any way they wanted. some chose to hand write the maps, some did them on a computer program called inspiration (http://www.inspiration.com), and some used other programs such as microsoft word or powerpoint. the on-line course was structured in seven modules, including readings, learning activities, individual or group project work and on-line discussion time. in this course, students created their concept maps in cmap, a server-based program created at the university of western florida (http://cmap.coginst.uwf.edu/) and installed on the uwm school of education server. students learned to access the cmap program and then to develop maps on the server. in this way, students in this course could view their colleagues’ work. the major question that i wanted to investigate in this study was how concept mapping impacts adult student learning over time. to do this, i chose first to request that students participate with me in this venture. all the students enrolled in both courses agreed to have their course work analyzed and used in this study. while some students requested that certain pieces of their work not be used, as they saw them as very personal accounts of their growth and development as adults, most students were very agreeable about being interviewed and having their work analyzed. in the fall 2000, i collected the first and final concept maps created by 21 randomly selected students from these two courses. then in december 2000, my assistant interviewed these 21 students. at first, i wanted to talk with students about how the mapping influenced them. however, i came to realize that if i did the interviews the students might tell me what they thought i wanted to hear. to avoid this potential bias a doctoral student in adult education, completed the interviews. we structured the interview guide so that she asked the following questions: 1. what was it like to use concept maps as a learning strategy? 2. what did you learn while doing concept maps? 3. where else have you used the maps since the completion of your course (if at all)? 4. how was doing the maps the same or different than other learning strategies you have used previously? 5. what did you like most/or like least about using concept maps? 6. what changes, if any, did you see in your thinking ability since using concept maps? 7. what was the most significant learning you remember from this course? 8. if you were going to describe concept mapping to another graduate student, what would you say? 9. how do you see using/or not using this learning strategy in the future? at the end of the first semester, we scored the first and final maps, then the purpose of this study was not necessarily to look at the impact of technology on the mapping process. however, because one course was face-to-face and one was on-line, the project did end up acquiring a technology facet. still my major interest is how mapping shapes learning, whether face-to-face or on-line. finally, i used quantitative and qualitative research methods since both research methods are valued within the field of adult and continuing education. moreover, since the purpose of this study was two-fold, first to see changes in concept maps, and second, to understand the student experience of learning with maps, it seemed to me that a mixed-method approach was needed. gathering evidence barbara j. daley the scholarship of teaching and learning: facilitating adult learning page 17 journal of scholarship of teaching and learning (josotl), volume 3, number 1 (2002), pp. 14-23 © 2002 board of trustees of indiana university http://www.inspiration.com/ http://cmap.coginst.uwf.edu/ barbara j. daley the scholarship of teaching and learning: facilitating adult learning page 18 journal of scholarship of teaching and learning (josotl), volume 3, number 1 (2002), pp. 14-23 © 2002 board of trustees of indiana university analyzed the qualitative data collected by developing categories and coding the data. in may 2001, we again contacted each of the 21 students and completed a second interview. if these students had constructed a concept map in the second semester, we then asked them to send us a copy of that map. once again, we scored the maps (see novak and gowin, 1984 for scoring formula) and analyzed the interviews through a system of categories and codes. this is the dimension of the study which is most unique since, to my knowledge, there has been little work done following students over a period of time to see if and how their learning strategies change after learning concept mapping. emergent findings and broader significance so what has this work demonstrated about student learning? during the first semester there was a statistically significant change in student concept map scores from the first to the final map. the mean score on the first set of maps was 44.81 and the mean score on the final maps at the end of the semester was 121.43, for a difference of 76.62. this change in mean scores indicates that students learned to subsume lower order concepts under higher order concepts, to progressive differentiate concepts and to synthesize concepts on their maps. these findings indicate students learned to link, develop interconnections, analyze and synthesize course information with their experiences. what was really exciting to me, however, was that 65% of the students continued to use maps into their second semesters. for those students who did use the maps, the mean score on the maps from the second semester was 120.22. this seems to indicate a significant change from the end of the first semester in the quality and development of maps in those students who continued concept mapping. two things are of particular interest; first, many students continued to do maps, and, second, that the mean score was virtually the same. i anticipated that the means would decrease to some degree compared to the end of the first semester. when analyzing the interview findings, we were able to categorize student responses in three basic categories that indicated how they learned and used cognitive mapping. these categories were: developing cognitive maps, learning with cognitive maps and follow-up. an example of a cognitive map is presented in figure 1. developing cognitive maps what students indicated was that their learning was facilitated when they understood how to develop maps. this involved understanding their own initial reactions, which were often negative, and being able to articulate how that reaction changed over time. additionally, students indicated that they needed to be able to describe mapping to others and discuss what they liked and disliked about it and where they were having difficulty in creating maps. finally, students expressed the view that their comfort and familiarity with computer software often impacted how they felt about mapping and how much they were able to learn from mapping. barbara j. daley the scholarship of teaching and learning: facilitating adult learning page 19 journal of scholarship of teaching and learning (josotl), volume 3, number 1 (2002), pp. 14-23 © 2002 board of trustees of indiana university relies on fosters impacted on through through resulting in resulting in relies on changes through through through through results in includes leads to more comfort with i.e. i.e. i.e. i.e. leads to leads to facilitating adult learning in higher education developing maps initial reaction developed reaction mechanics of mapping descriptions liked difficulty software learning with maps understanding own learning learning strategies linking interrelationships creating meaning schemes constructing changes in thinking follow-up use non-use figure 1: facilitating adult learning in higher education learning with cognitive maps students also indicated that mapping helped them to understand their own learning. it was through understanding their own learning that students began to use maps to develop the learning strategies of linking concepts, developing interrelationships, creating meaning schemes, and constructing knowledge. we could see from student responses in the interviews that linking was the first step in developing these new learning strategies. students learned to link concepts in ways that made sense to them and connected with their previous experiences. they also learned to search out relationships among concepts. they told us that they learned how they were creating meaning schemes and constructing their own knowledge base through this process. it was very exciting to hear student say things such as: you read it first and then you pull out the basic concept, the major concepts that are within that framework and you draw connections between those concepts, and you are going to see connections and you are going to see distinctions that were not apparent to you before you sat down and actually did that. that is how you construct your new knowledge or, concept mapping is a way to take the idea, apply it, and get a deeper meaning out of it at the very end. it is not just a matter of learning a concept, learning about theory, defining a word and spitting back a definition. it is actually applying it to what you know so that it makes more sense in the actual world. i would say things like the purpose of concept maps is to help us explore the meanings, the inner-relationships, that we are making as we developing our understanding of the concepts. so that it is a meaning/making process. that is what really grabs me, anyway. and, finally, it made you look at whatever it was you were doing in its entirety. it made you look at it as a whole. and then started breaking it down by concepts and then you would rebuild it by linking stuff and i guess that is how i constructed new knowledge or how i found myself looking at things differently. you feel the knowledge building. you just feel yourself seeing things differently than before you started doing that. there were some students in the study who had difficulty creating maps and using mapping as a strategy. the difficulties seemed to be related to the time required. often students would indicate that this type of learning activity required more time than they were prepared to, or wanted to, give to the particular assignment. other students admitted that the difficulty they had with maps had more to do with changing how they learned. students stated that the maps required them to think differently and some students just did not like that. as this student indicates: but, i guess what i hated the most was that i had to change my thinking mode. it is before, like, well, i am just reading this information, and i am picking out what i see is in the writing or what the writer is trying to present. i guess i just didn’t like the idea of changing old habits and doing things differently. or, barbara j. daley the scholarship of teaching and learning: facilitating adult learning page 20 journal of scholarship of teaching and learning (josotl), volume 3, number 1 (2002), pp. 14-23 © 2002 board of trustees of indiana university follow-up in interviewing the students on follow-up we found that 65% did continue to use mapping as a learning strategy, but even those who did not use cognitive maps in future learning reported that their thinking had changed. students were able to describe how, when they approached learning a new topic, they started to think conceptually, searching out interrelationships and looking for ways to connect the information with their experiences. what i found very interesting was that students were able to describe changes in their thinking even if they no longer sat down and formally developed maps. in this project, students who continued to use mapping reported that they did so for a number of reasons. they seemed to use maps to understand course material in subsequent graduate courses. they also relied on maps as a way to understand particularly difficulty material. many participants reported that when they felt “in trouble” in a course or that they “did not get it,” they would try mapping out the material as a way to develop their understanding. additionally, learners tended to use maps to frame projects for subsequent courses or work-related projects. one participant described how he had a big project to do at work and as a way to help his team understand the scope of the project, he mapped it out and shared the map with them. another student described how she used a concept map in a subsequent class to demonstrate decision-making. the students who did not use mapping indicated that they chose not to because it was not required, took too much time or they did not have access to concept mapping software. this last statement was a surprise to me. in teaching the classes, i thought i had been clear that the maps could be constructed in most any way the students chose. however, since the software does facilitate the actual mechanics of mapping it seemed that for those who did not create the maps, the lack of access to software most likely compounded the time problem. another interesting aspect of this study was that it included both an on-line and a face-to-face component. it did not appear that there was any difference in the quality of maps created by the students in the on-line and face-to-face courses. however, in the on-line course, there was a higher percentage of students who continued to do maps at the one-year follow-up. my sense is that this finding emerged because there were mostly doctoral students in the online class and those students saw the maps as tools to assist them in the conceptualization of their dissertations, as a way to synthesize literature reviews and as a way to conceptually link research and theory courses. i did not get the sense that the on-line component of the course facilitated map development, but rather that the students’ need to use the maps in subsequent scholarly work seemed to be the motivator to continue. that said, i think the connections among mapping, software and technology are still very important issues for further investigation. conditions of doing scholarship of teaching and learning part of what made it possible to do this project was timing. the university of wisconsin – milwaukee has begun to look at the sotl approach and has initiated a number of activities designed to inform faculty of its possibilities. barbara j. daley the scholarship of teaching and learning: facilitating adult learning page 21 journal of scholarship of teaching and learning (josotl), volume 3, number 1 (2002), pp. 14-23 © 2002 board of trustees of indiana university lessons learned being involved in some of the initial committee and planning meetings, i found myself getting very excited about the possibilities for research and scholarship around teaching and learning issues. so, when cipd offered small grant opportunities to design sotl projects, i applied. the grant funding was important because it did allow me to buy myself out of one course and to fund the typing of transcripts. however, more important to me was the acceptance of this kind of work by colleagues within my own department and across the university. funding is great, but without a commitment by the institution and one’s colleagues, it would be difficult to persist in work studying our teaching and learning practices in higher education. as i think back on the lessons learned in this project, one of the things that struck me was how difficult it is to get learners to change their learning strategies. i knew there would be resistance to doing concept maps, but i did not expect students to articulate that one of the things they did not like about mapping was having to change their ways of thinking. it again points out to me the depth to which learning strategies are engrained. cerbin (2000) seems to agree when stating, “i now believe much more firmly that changing students’ minds, moving them to ‘deep understanding,’ is quite a bit harder than is usually recognized” (pg. 16). i also learned that it is important to continue investigating the connections between learning and technology. in this project, students expressed how the use of software was important in learning to develop concept maps. moreover, the on-line course results were in some ways different than the face-to-face results. this indicates to me that much more work is needed in this area. benefits of the work i see the benefits of this work as threefold. first, the students benefit. by studying, analyzing, reflecting on and changing our teaching practices, we offer new insight to students and facilitate their learning in ways that we may not have thought about previously. second, i , personally benefit. i found myself very excited about this project because it allowed me to look at my teaching and ask questions that i felt could only help me become a better teacher. it also motivated me to continue using mapping once i saw the results. additionally, i think that i learned just how invaluable our peers can be and how much we can learn from them. because this project was funded by cipd, there were other scholars working on sotl projects on campus. a group of five center scholars met monthly. these group meetings provided a safe place to talk about the work, to discuss the set-backs and to offer peer-based critique and feedback. finally, the institution benefits. as we develop an institutional reputation for focusing on teaching and learning, that reputation can only enhance our credibility and authenticity within the communities in which we live and provide our services to students. finally, i have come to believe in the strength of the sotl approach in higher education. sotl offers faculty a way to understand their teaching and student learning, as well as, to initiate deep and barbara j. daley the scholarship of teaching and learning: facilitating adult learning page 22 journal of scholarship of teaching and learning (josotl), volume 3, number 1 (2002), pp. 14-23 © 2002 board of trustees of indiana university barbara j. daley the scholarship of teaching and learning: facilitating adult learning page 23 journal of scholarship of teaching and learning (josotl), volume 3, number 1 (2002), pp. 14-23 © 2002 board of trustees of indiana university long lasting change in both. pat hutchings (2000) explains that sotl is characterized by three factors. she writes, “ . . . the scholarship of teaching and learning is deeply embedded in the discipline; its questions arise from the character of the field and what it means to know it deeply” (pg. 6), “ . . . the scholarship of teaching and learning is an aspect of practice” (pg 8) and finally, “the scholarship of teaching and learning is characterized by a transformational agenda” (pg. 8). the discipline, practice and transformation are all aspects of sotl that will continue to impact higher education as we move ahead in the future. references brookfield, s. (1995). becoming a critically reflective teacher. san francisco, ca: jossey-bass, inc cerbin, w. (2000). investigating student learning in a problembased psychology course. in hutching, p. (ed.) opening lines: approaches to the scholarship of teaching and learning. the carnegie foundations for the advancement of teaching. menlo park, ca. daley, b., shaw, c., balistrieri, t., glasenapp, k., & piacentine, l. (1999). concept maps: a strategy to teach and evaluate critical thinking. journal of nursing education, 38, 1, 1-6. daley, b. (1996). concept maps: linking nursing theory to clinical nursing practice. journal of continuing education in nursing 27, (1), 17-27. hutching, p. (2000). opening lines: approaches to the scholarship of teaching and learning. the carnegie foundations for the advancement of teaching. menlo park, ca. novak, j., & gowin, b. (1984). learning how to learn. cambridge university press. novak, j. (1998). learning, creating and using knowledge: concept maps as tools to understand and facilitate the process in schools and corporations. lawrence erlbaum associates. 503 service temporarily unavailable 503 service temporarily unavailable nginx/1.14.1 v3n1total.pdf the journal of scholarship of teaching and learning (josotl) volume 3, number 1 (2002) copyright 2002, the trustees of indiana university. except as may be expressly provided elsewhere in this publication, permission is hereby granted to reproduce and distribute copies of individual works from this publication for nonprofit educational purposes, provided that copies are distributed at or below cost, and that the author, source, and copyright notice are included on each copy. this permission is in addition to rights of reproduction granted under sections 107, 108 and other provisions of the u.s. copyright act. sponsored by uce acet, and indiana universit uth bend the journal of scholarship of teaching and learning t, f y so development of a simple mathematical predictor of student performance in general chemistry nathan w. bower chemistry department colorado college colorado springs, co 80903 abstract colorado college uses a sequential course structure exclusively in its calendar that is similar to those used in summer programs at other institutions. in this approach, a student takes, and a faculty member teaches, only one four-hour course each month. this format enhances longitudinal studies of the factors that affect student grade performance and retention. in this study, standard predictors of success, such as act and sat scores, are compared with a simple mathematical background knowledge probe. other factors that may impact student performance such as economic background, gender, learning style, and time between courses are also discussed. introduction in addition to being used for admissions and financial aid, these tests are also sometimes used to replace college-wide requirements and for placement in courses.5 some years ago, pickering did an interesting long-term study using the sat math score to identify students a priori who were expected to do poorly in general chemistry. in a controlled experiment, he offered an intensive supplementary course in problem solving to a subset of the students with sat math scores below 610. the modest improvement in grade noted (0.41 on a 4.0 scale for n = 43) versus the control group and the effect on their subsequent general and organic chemistry grades (0.17 and –0.08, respectively) for the same students raises questions about the long-term efficacy of such efforts.6,7 one of the major problems facing faculty who teach science courses that have a significant reliance on a mathematical foundation is determining whether individual students entering the class have the appropriate preparation in this ancillary area. compounding the difficulty of determining and then enforcing pre-requisites are other factors that can enter the picture, such as math anxiety as a separable issue from math or science competence.1 a simple background probe that can be administered in a few minutes and which is relatively free of confounding bias is needed to provide important feedback at the outset of the course. such an instrument would allow the teacher to do some last minute finetuning of the course level as well as offering the opportunity for scheduling of individualized remedial help. despite the disheartening results in pickering’s study, science teachers continue to try to identify which students are likely to need help as early as possible. this seems especially important with the intensive course structure known as the “block plan” in which students take (and instructors teach) one course at a time for about a month. this structure is used at many background the past decade has seen a fierce national debate over the validity of using standardized exams such as the sat in college admission decisions, especially as affirmative action has come under attack.2 newsweeklies expound upon these tests and their role in our society, and it seems likely the supreme court will soon have to sit on their constitutionality.3,4 4 lehman, n., behind the sat, newsweek, 134(10) 52-57 (sep. 6, 1999). 5 coley, n., prediction of success in general chemistry in a community college, journal of chemical education, 50(9) 613615 (1973). 1 hembree, r., the nature, effects, and relief of mathematics anxiety, journal of research in mathematics education, 21, 3346 (1990). 6 pickering, m., helping the high risk freshman chemist, journal of chemical education, 52(8) 512-514 (1975). 2 mealer, b., moves against affirmative action fuel opposition to standardized admissions tests, chronicle of higher eduction, 48(8) a40-a41 (oct. 17, 1997). 7 pickering, m., the high risk freshman chemist revisited, journal of chemical education, 54(7) 433-434 (1977). 3 cloud, j., what does sat stand for? time, 150(20) 54-55 (nov. 10, 1997). nathan w. bower development of a simple mathematical predictor of student performance in general chemistry page 5 journal of scholarship of teaching and learning (josotl), volume 3, number 1 (2002), pp. 4-13 © 2002 board of trustees of indiana university institutions for summer session courses, but it is used for the entire calendar of courses at colorado college. one of the advantages of the block plan is that it fairly readily allows students to switch courses on the first day of class without increasing their years of matriculation if they can quickly determine what course is best for them. even if a placement quiz or “knowledge probe” at the start of a course is not used to determine who may need extra help, it can provide students with additional information about whether they are in the appropriate course for their background and interest level. another use of such knowledge probes is to help in selecting groups for cooperative learning strategies. because students exhibit a variety of learning styles,8 it is useful to identify those that are stronger or weaker in the traditional algorithmic approaches. depending on the tasks set by the instructor, it may then be desirable to form groups that are diverse in their abilities, or, if it makes sense is to spend more time with those groups needing additional help, it may be desirable to form groups with similar backgrounds. at colorado college, courses are limited to 25 students and there are up to eight different faculty teaching general chemistry in any given year. approximately 200 students take the introductory chemistry course each year, or about a third of each graduating class. no differentiation is made between students majoring in chemistry or any other field. general chemistry i and ii are offered almost every block, or nine and eight times per year, respectively. the courses are equivalent to the first and second semester of general chemistry taught elsewhere, and each block is three and a half weeks long (followed by a half week break for grading and setting up the next course). the unique nature of the block plan provides a “laboratory” for testing new ideas in education. with different teachers and so many students from varied backgrounds involved in the introductory courses, it is possible to obtain data on a host of variables with minimal confounding. because sat and act scores are not routinely available to chemistry faculty at colorado college for reasons of privacy, different “quizzes” have been devised that can be taken in a few minutes on the first day of class. the quizzes are taken without calculators, and they are designed to ask a few questions that are a little outside of the routine “algorithmic” approaches students learn, especially those that rely on a calculator. the results of these quizzes are shared with the students immediately so that they will be able to determine for themselves whether they need to arrange for additional tutoring or whether they should postpone the course until they are better prepared. often the quizzes are simply exchanged with a neighbor and the scoring is covered in a couple minutes as a method of nearly instant feedback. in those instances, the quizzes may not even be collected, so the instructor may get no direct feedback on a given student’s needs, and the student is given full responsibility for their own decisions regarding what to do with the results. the purpose of this paper is to report on the efficacy of two of these quizzes, comparing them with other predictors and factors affecting long term “success”. success in this case is 8 felder, r. m., matters of style, asee prism, 6(4) 18-23 (1996). nathan w. bower development of a simple mathematical predictor of student performance in general chemistry page 6 journal of scholarship of teaching and learning (josotl), volume 3, number 1 (2002), pp. 4-13 © 2002 board of trustees of indiana university measured by grades in subsequent chemistry courses, but other measures, such as retention and the number of subsequent courses taken are also considered. personal reflections on what sorts of knowledge probes need to be developed in order to continue to improve through this classroom action research process are included in the ensuing discussion. in one of the courses (a typical course of 25 students in spring of 2000), data was collected in class on the learning preferences of the students using the index of learning styles developed by felder and soloman.9 the students self-scored this instrument, and made use of the suggested published strategies as they saw fit. throughout the course, the instructor made the point of sharing with the students the various assessment tools being used and what was learned from them. because the students saw themselves as involved in an experiment in this class, they seemed more involved in how the course was taught and in thinking about how to optimize their learning right from the start of the course. in the following year (spring of 2001), the same instructor in a matched class of students in terms of class size, content, text, diversity, and timing, repeated the experiment but without informing the students about the educational research aspects and assessments tools being used until the end of the course. such classroom action research with individual courses is common, and these experiments add to the enthusiasm that both the instructors and the students feel in these courses. experimental five general chemistry ii courses (n = 132 students) spread over five years (1995 to 2000) were randomly selected from those courses in which a twoquestion math quiz had been administered. five additional courses (n = 117 students) were selected from those in which a seven-question quiz (appendix 1) had been given. three of these were general chemistry i courses and two were general chemistry ii courses. the first two questions of the seven-question quiz are the same as those used on the two-question quiz. these courses involved two different teachers with varying degrees of cooperative learning strategies incorporated in their courses. additional retrospective information was obtained from student transcripts, such as the sat and act scores for math and verbal reasoning, total financial aid, work-study grants, and grades in prior and subsequent chemistry courses. information was also collected on how long students waited between courses, who taught each course, the format used in the course, length of time between courses, self-reported ethnic background, and the gender of the student. this information was correlated using multiple linear regression and anova between the predictors and factors. minitab version 12 was used throughout the analysis. results at colorado college, students wait an average of 4.7 months between general chemistry i and ii, although they are advised to take them within the same semester. the average waiting period lengthens to 7.4 months between 9 soloman, b. a., and felder, r. m., index of learning styles, north carolina state university, www2.ncsu.edu/unity/lockers/users/f/felder/p ublic/ilspage.html (2000). nathan w. bower development of a simple mathematical predictor of student performance in general chemistry page 7 journal of scholarship of teaching and learning (josotl), volume 3, number 1 (2002), pp. 4-13 © 2002 board of trustees of indiana university the profile of the students entering general chemistry ii is typical of the student body as a whole, with 60% on financial aid and two-thirds of these on work-study. about 55% are female, 16% are a self-identified ethnic minority (about half are hispanic), and the median sat math and english scores are 630 and 620 respectively. the average grade obtained in the first course is a 3.0 on a 4.0 scale. this drops slightly to a 2.9 in general chemistry ii and a little further to a 2.7 in organic chemistry i. the average returns to a 2.9 in organic chemistry ii as primarily majors in chemistry, biochemistry, neuroscience, and premedical students take it. general chemistry ii and organic chemistry i. between the organic chemistry i and ii, this period shortens to 2.6 months. chemistry majors tend to take their courses somewhat closer together than this, and the organic i and ii progression reflects this as only biochemistry and chemistry majors (and those planning to attend medical school) are required to take the organic chemistry ii. on the other hand, geology majors (who are required only to take general chemistry i) often wait three years between the first and second course. only those geology majors planning to continue into graduate school in the field return to take general chemistry ii. based on soloman and felder’s fourdimensional index of learning styles,10 a given class will be moderately (but significantly) more visual than verbal and somewhat (but significantly) more active than reflective in their learning style preferences. the class will also be slightly (but not significantly) more sequential than global, and nearly equally balanced on the sensing versus intuitive dimension. although the averages fall near the middle of the scale, at least a third of the students will have a strong preference for at least one learning style. out of 25 students, each of the different dimensions were represented by at least two students with a strong preference for that style except the reflective and verbal dimensions, and even these had more than one student with a moderate preference in that direction. this profile matches the expected general science student population at colorado college, which attracts outdoor-oriented, athletic students who like a balance of creative despite these widely varying times between courses, there is little evidence that students’ grades were impacted by either putting off their chemistry courses or by taking them back-to-back. this may possibly be because most students who go on continue to mature in a parallel science and they bring that mental maturity with them. alternatively, this may indicate that the learning strategies and motivation that the majority of students bring to their courses is more important to their overall success than the content we manage to impart in our chemistry courses, despite how sequential we think they are. (students who skip a course or take them out of sequence do suffer, however, so there is something they are mastering even if we can’t find an adequate test for it.) whatever the reason, the gaps between courses are not drastically different from that experienced by students in a semester system, where the average time between material in successive semesters is 4.5 months (mid-fall to mid-spring) and 7 months (mid spring to mid fall). 10 soloman, b. a., and felder, r. m., ibid, (2000). nathan w. bower development of a simple mathematical predictor of student performance in general chemistry page 8 journal of scholarship of teaching and learning (josotl), volume 3, number 1 (2002), pp. 4-13 © 2002 board of trustees of indiana university nathan w. bower development of a simple mathematical predictor of student performance in general chemistry page 9 journal of scholarship of teaching and learning (josotl), volume 3, number 1 (2002), pp. 4-13 © 2002 board of trustees of indiana university outlets to go with the intensive study of the block plan. these profiles indicate that using a variety of approaches on every major topic should be expected to be necessary in order to reach all of the students. at the end of the course in which the students were actively informed of the results of the various assessment tools, student evaluations indicated that they appreciated the efforts made to respond to their different styles of learning, and most students were more proactive than similar classes in trying to make the best use of the resources geared to their preferences. the following year’s class (22 students), which had a slightly (but insignificantly) higher sat-m score of 643.5 40.3, 64% female, 14% minority (compared to 631.1 53.2, 68% female, 16% minority), had a more typical (lower) level of engagement and interest in the class. the final standardized exams from the american chemical society, which are designed to have normal distributions around 50%, were also nearly equivalent (64.1% versus 62.0% for the second class compared to the first class, matching the ratio of the sat-m scores). although these results indicate the classroom involvement in the learning style research project had little or no impact on the student learning or grades achieved, the anecdotal evidence of retention and interest beyond the class is very much different: from the first class with the slightly lower sat-m scores, five students (three women and two men) immediately selected the author as their academic advisor and two of these indicated an interest in majoring in chemistry, while only one from the second class did so in the in the two months following each course. a multiple linear regression analysis of the various predictors (table 1) shows that the two-question math quiz, graded 0 to 3 in half-units, is a better predictor of the grade a student will obtain in general chemistry ii than either the sat or act math sub-tests. (deviation from a normally distributed variable for the three predictors and the dependent “grade” is not significant despite their discrete functions.) general chemistry ii is one in which math ability plays a major role, as this is the course that deals with thermodynamics, acid-base equilibria, and kinetics. the act’s better performance compared to the sat may be due to a variety of factors. for example, the cram courses now available for improving a student’s sat test scores may be clouding the test’s predictive power. also, the act (in this study) seems to be taken by a larger percentage of students with a more diverse background of abilities and economic advantages compared to the sat. table 1: national test and math quiz predictors of general chemistry ii grades regression: predictor coef stdev t p anova (f) constant 0.052 1.131 0.05 0.963 math-quiz 0.2549 0.1193 2.14 0.038 10.25 sat-m 0.00027 0.0022 0.13 0.899 1.89 act-m 0.07982 0.0472 1.69 0.098 2.86 r = 0.504 r (adj. for d.f.) = 0.451 n = 48 (due to few students taking both tests) overall regression: p = 0.005 (manova f has 1 d.f./44 d.f.) a regression of each predictor alone gives adjusted correlation coefficients of 0.381 (n=131), 0.257 (n=111), and 0.539 (n=65) for the math pre-quiz, sat-m, and the act-m tests respectively. as was noted above, the act test’s apparently better performance is a result of the unique subset of students sampled, and table 1 represents a better indicator of the relative merits of each test applied to the same subset of students despite the fact they are not truly independent variables. table 2: math quiz and categorical predictors of general chemistry ii grades regression: predictor coef stdev t p anova (f) constant 2.2372 0.3125 7.16 0.000 math-quiz 0.30384 0.07654 3.97 0.000 22.94 fin. aid -0.2869 0.1881 -1.53 0.130 2.92 class 0.07920 0.09878 0.80 0.424 0.03 instructor 0.0430 0.1448 0.30 0.767 0.22 gender 0.04722 0.07062 0.67 0.505 0.68 major 0.3691 0.1166 3.17 0.002 8.59 ethnicity -0.21308 0.09525 -2.24 0.027 5.00 r = 0.452 r (adj. for d.f.) = 0.397 (n = 129) overall regression: p = 0.000 (manova f has 1 d.f./121 d.f.; durbin-watson stat. = 2.01; lack of fit p > 0.1) in table 2, the two-question math quiz is coupled with a number of categorical variables and one continuous variable (financial aid, expressed as a fraction of the full cost of attending). the class variable has four levels (1 – 4), and the other variables have all been reduced to two levels (-1, 1). the underlying assumptions of the regression model are violated by departures from normality for these predictors, but the results are still useful for making some qualitative observations. the results of this regression and manova suggest that the class (first, second, third, or fourth year student), instructor, and gender of the student have little or no impact on the grade achieved in these courses. the major of the student (one of the chemistry options versus non-chemistry majors), nathan w. bower development of a simple mathematical predictor of student performance in general chemistry page 10 journal of scholarship of teaching and learning (josotl), volume 3, number 1 (2002), pp. 4-13 © 2002 board of trustees of indiana university the ethnic background of the student (caucasian versus all others), and the financial need of the student are more important. as might be expected, chemistry majors tend to achieve higher grades, although at this stage of their careers only a small fraction have declared their major. thus, cause and effect are still undifferentiated. there is also a correlation of financial need and ethnic background, as students from more diverse backgrounds tend to have higher financial need at colorado college. the financial need is usually (but far from always) coupled with more time spent on work-study, and this is time that may interfere with time spent on the course. based on anecdotal student information, another contributor to the financial need effect that is largely independent of the student’s ethnic background is that students who have a large loan often are under pressure from their families to transfer to a less expensive institution. this effect is most pronounced in the blocks taught at the end of the year as students begin to mentally disengage from the course and the institution. while a complete analysis of the results of the seven-question pre-quiz will not be presented here because of the similarity of its results to the twoquestion pre-quiz, it should be noted that the seven-question pre-quiz doubles the range of possible scores (0 to 7 instead of 0 to 3). this improves its value for individual person diagnostics. it also adds a component that tests for recollection of chemical content from previous courses. as a result, it does a slightly better job of predicting the grade a student is likely to achieve. although the results presented in this paper have focused on general chemistry ii, both the two-question and the sevenquestion quizzes have been administered in general chemistry i with very similar results. even though memory of prior course content was not a variable emphasized in this study, comparison of the twoquestion prediction to the full sevenquestion prediction indicates that chemistry content memory is not as important as facility with math in predicting a subsequent course grade. the memory portion was also more subject to loss as the time interval between courses increased. there is little evidence that general chemistry is strictly sequential, as various textbooks order the material differently. instead it seems there are a variety of valid starting points and the more grasp the student has of the global picture, the more easily new material can be placed in a meaningful context. thus, the grade obtained in subsequent courses seems to be more closely connected to some longer lasting skills or a more global knowledge than it is to any specific content recollection. however, at colorado college, content tests such as the american chemical society general chemistry tests do correlate strongly with the course grade (p = 0.000) when they are administered at the end of the course, as does the gre-subject test in chemistry (p = 0.011), which is taken by many of the majors at the end of their undergraduate career. the average grade on this latter test is comparable to the national scores for students from other schools, indicating that the block plan does allow an accumulation of content that can be measured to some extent. reflections and future directions the efficacy of this simple knowledge probe for detecting those students who have math difficulties relevant to the course has been born out over the nathan w. bower development of a simple mathematical predictor of student performance in general chemistry page 11 journal of scholarship of teaching and learning (josotl), volume 3, number 1 (2002), pp. 4-13 © 2002 board of trustees of indiana university years. the positive impact of taking the time to involve the students in understanding their own learning processes is equally apparent, although more work is needed to find ways to make such involvement more time efficient in order to avoid additional loads on the faculty. much less clear is what intervention measures to take with those students who have math difficulties in order to affect a long-term gain in chemistry. at a minimum, a good math review at the outset, additional tutoring outside of class, or a remedial math course is required. for some students this will prove to be adequate, but for a large percentage, something more is needed. in a recent paper, ashcraft and kirk have provided some valuable insights into how math (or other anxieties) affect other performance.11 by proving math anxiety is separable from math incompetence (and that they are independently treatable), they point the way to other testing that can indicate where students may obtain the help needed to overcome these two common hurdles. not surprisingly, students with math anxiety often develop lower math aptitudes as they progress through their education, and as a society we must recognize (and treat) this problem in the same way we are beginning to recognize handicaps such as dyslexia. the demonstration of the impact of math anxiety on the speed of mental processing for students who are competent in math despite having such anxiety indicates that giving these students longer to respond will allow them to reveal their actual level of competency on the subject matter of interest. ashcraft and kirk argue that problems that involve some form of math beyond the level of multiplication or addition tables, and which call upon other forms of memory at the same time, compete for “space” in the smaller “working memory” available to these students compared to others. an analogy might be comparing two computers, one with a smaller or “busier” cpu (due to interference from the anxiety) than the other. both can solve the same complex problems, but because more shuffling of the data is needed in the smaller/busier cpu machine, longer time must be spent to achieve the same end result. except for the time factor, both will achieve the same final goal. if one machine lacks the proper programs (math competence), it will not be able to solve the problem until such programming is provided, at which time it may be faster or slower depending upon multiple factors, including the “working memory” it has. this anxiety preoccupation in the “cpu of the mind” also is relevant when the material is presented in the class if a mathematical presentation is involved. “taking it in” will take longer for these students just as purely written (textual) presentations will take a dyslexic student longer to process correctly. the inherent abilities except for this time factor are in no way diminished. this suggests again that multiple modes of presentation are needed in order to reach the diverse population of students that we will encounter in our classes, especially if we are to help all of them achieve their full potential for contributing to society. 11 ashcraft, m. h., and kirk, e. p., the relationships among working memory, math anxiety, and performance, journal of experimental psychology: general, 130(2), 224-237 (2001). what does this suggest should be done in the way of modifying the knowledge nathan w. bower development of a simple mathematical predictor of student performance in general chemistry page 12 journal of scholarship of teaching and learning (josotl), volume 3, number 1 (2002), pp. 4-13 © 2002 board of trustees of indiana university nathan w. bower development of a simple mathematical predictor of student performance in general chemistry page 13 journal of scholarship of teaching and learning (josotl), volume 3, number 1 (2002), pp. 4-13 © 2002 board of trustees of indiana university probe described here? first, at least some measure of the level of math anxiety should be obtained so that the student may be steered to the appropriate source of help. math anxiety is highly correlated with chemistry anxiety in general,12 underscoring the need to determine the influence of this factor on student performance at an early stage. two simple probes that may be of use for this include asking the students to report the number of high school math courses they have taken and to rank their own anxiety on a scale of 1 to 5. both of these showed significant correlation (p = 0.05) with a much longer test of math anxiety that ashcraft and kirk employed, and this author intends to include at least these two in the next version of the background knowledge probe. as claude fuess once said, “i was still learning when i taught my last class.”13 a slightly longer math test taken without calculators that includes a more active, non-mathematical, non-verbal visual test component (such as rotating or constructing actual stick and ball models of stereo isomers)14 might also be useful for predicting academic success in introductory chemistry courses. this would offer a better range of responses so individuals can be more accurately diagnosed as well as keeping its 12 eddy, r. m., chemophobia in the college classroom: extent, sources, and student characteristics, journal of chemical education, 77(4), 514-517 (2000). 13 claude m. fuess, after 40 years at phillips academy, independent schoolmaster, atlantic monthly press 52, http://www.bartleby.com/63/30/2530.html. 14 habraken, c. l., perceptions of chemistry: why is the common perception of chemistry, the most visual of sciences, so distorted?, journal of science education and technology, 5(3), 193-201 (1996). predictive power for the group. it might also better test for the multiple intelligences that are correlated with the necessary skills to do well in chemistry. the question of how sequential general chemistry really is, especially in the context of developing critical thinking skills, is still open for debate. clearly, it would be very useful to proceed to background knowledge probes that can be administered on the first day that will determine what level of development students have achieved in the area of critical thinking skills, and then to select questions that help lead the students to move to the next level. some work is beginning to be done in this area,15 but a great deal of foundation still needs to be developed for a large percentage of students, including simply moving them beyond a state of anxiety. whether the instructor includes additional questions that probe other learning styles and intelligences, a short math quiz that does not allow the use of a calculator “crutch” taken at the start of the course seems to be a better predictor than the national act-m or sat-m test. the short quiz described here with its immediate availability of results for either the students or the instructors (or both) provides a viable alternative to the much-maligned national tests. 15 kogut, l. s., critical thinking in general chemistry, journal of chemical education, 73(3), 218-221 (1996). 503 service temporarily unavailable 503 service temporarily unavailable nginx/1.14.1 microsoft word jungst_with_columns_v3_n3.doc providing support for faculty who wish to shift to a learningcentered paradigm in their higher education classrooms steven e. jungst dept of natural resource ecology & management 253 bessey hall iowa state university ames, iowa 50011 515-294-1587 sejungst@iastate.edu barbara l. licklider, and janice a. wiersema dept of educational leadership and policy studies n247b lagomarcio hall iowa state university ames, iowa 50011 abstract research shows that students who experience a learning-centered paradigm outperform those who experience a more traditional teaching-centered paradigm. faculty are generally not well prepared to adopt a learning-centered paradigm, and trying to do so without ongoing support is difficult, at best. we describe a model used at iowa state university that provides the program and process necessary to help faculty make such a paradigm shift. introductory and advanced workshops are coupled with ongoing biweekly meetings to help faculty develop the strategies and understanding of the learning process that are necessary to develop a learning-centered classroom. the approach used here could serve as a model for other colleges and universities wishing to help faculty who wish to make such a change. mailto:sejungst@iastate.edu “we won’t meet the needs for more and better higher education until professors become designers of learning experiences and not teachers” (spense, 2001). introduction there is no question that we need better higher education. classroom professors lament the fact that students don’t perform well on exams, that they seem indifferent to materials being taught, and that they don’t seem able to transfer what they have “learned” even from one class to the next, let alone from the university setting to the rest of their lives. employers demand more than just technical knowledge of university graduates. they also want graduates who can think on their feet, work effectively in teams, communicate effectively, and create new knowledge that will give their employers the advantage in today’s fast-paced world (gardiner, 1994; nsf, 1996; brown and lassoie, 1998). students complain that they don’t see the relevance of classroom material to what they want to do for the rest of their lives. and through it all, frustration grows. meanwhile, a revolution is underway in some college classrooms. once regimented to lectures, tests and student apathy, these classrooms are evolving into active, interesting and engaging places to be. the key is engagement, a notion that emphasizes a move from a passive to an active learning environment. teachers and students alike are discovering what cognitive research of the past decade has shown; a world of difference exists between rote recall of facts and a deeper understanding of the principles underlying facts and processes. it is at this deeper level of understanding that true learning occurs, learning that can be transferred to the world beyond the classroom. “effective learning strategies almost always require the learner to participate” (sanders, 1998). that statement is at the heart of attempts by faculty on many campuses to create a more learningcentered atmosphere in college classrooms. “triggered by the 1983 report, a nation at risk, that warned ‘the educational foundations of our society are presently being eroded by a rising tide of mediocrity’, learningcentered efforts were energized by a second wave of reform reports that began appearing in the early 1990’s. these reports focused the reform efforts on a common theme: to place learning first. a 1993 report, an american imperative, called for ‘putting student learning first’ and ‘creating a nation of learners.’ in 1994 the education commission of the states urged a reinvented higher education system that would reflect a new paradigm shift centered on learning. in 1995 the association of american colleges and universities issued a paper titled, the direction of educational change: putting learning at the center.” (o’banion 1998). underpinning the learning revolution is a growing recognition that facing the challenges of the 21st century will require more than minor adaptations to current practice (mullin 2001). current instructor-centered methods simply cannot sufficiently effect the complex outcomes (higher order thinking skills, problem solving, the ability to see from diverse perspectives, ethical reasoning, and life-long learning) that a prepared citizenry needs. effecting these outcomes will require many changes in jungst, licklider, & wiersema providing support for faculty who wish to shift to a learning-centered paradigm in their higher education classrooms page 70 journal of scholarship of teaching and learning (josotl), volume 3, number 3 (2003), pp. 69-81 © 2003 board of trustees of indiana university the way faculty members approach the job of helping students learn. leading those changes will be faculty who are skilled in designing educational experiences that make students active participants in their own learning. the question isn’t whether students who are actively involved in their own learning fare better than those exposed to a more passive style. that active learning techniques work is well documented in the cognitive research literature (salvin 1990, nastasi and clements 1991, gough 1987, marzano et al 2001, national research council 2001). not only do students involved in active learning situations out-perform those who learn through more passive classroom approaches, but also most of them actually prefer learning-centered classrooms once they understand the new set of expectations (qualters 2001). the question is, what will help educators on college campuses move toward a learning-centered paradigm—both in belief and in action? the shift to a learning-centered classroom atmosphere presents several challenges for college campuses and for faculty. such a change is not easy, even for those who wish to make it. while some faculty have been successful in implementing a learningcentered approach in their classrooms, many dedicated faculty with genuine interest in improving classroom instruction grope for better ways of doing their jobs but are hampered by their own past experiences and lack of training in cognitive science. they are hindered further by university, college, and departmental administrators who cling to the idea that “teaching is telling, learning is absorbing, and knowledge is subject-matter content” (spense, 2001). learning-centered strategies are based on knowledge of how students learn. yet most faculty know little about how students learn. moreover, because college professors have seldom learned much about teaching in the first place, they know even less about new views. what are needed, in order to surmount these obstacles, are opportunities for faculty to come together as learners to learn about learning. to make lasting change, faculty need a chance to experience learning in a learning-centered atmosphere. they need opportunities to practice in their own classrooms with continuing support from those knowledgeable about practical applications of current cognitive research. they also need the support of their peers who are experiencing the same trials, tribulations, rewards, and joys of designing learning experiences for their students. providing these kinds of learning opportunities is the mission of project lea/rntm (learning enhancement action/resource network). the model project lea/rntm, at iowa state university, is one model for providing support for faculty as they move from a teaching-centered to a learningcentered paradigm. initiated by an education leadership professor in 1993 in response to requests from the college of engineering, it was further expanded through a usda higher education challenge grant to introduce additional faculty at iowa state university and alabama a&m university to the theory and practice of a learningcentered paradigm. jungst, licklider, & wiersema providing support for faculty who wish to shift to a learning-centered paradigm in their higher education classrooms page 71 journal of scholarship of teaching and learning (josotl), volume 3, number 3 (2003), pp. 69-81 © 2003 board of trustees of indiana university project lea/rntm is not simply a program. more importantly, it is a process. grounded in research (licklider, et. al. 1998), it draws on several core elements of adult learning theory: critical reflection, purposed discussion with colleagues, accountability, and action (cranton, 1994, knowles, 1994, mezirow, 1991). these critical aspects of learning are operationalized in project lea/rntm through its core structures that apply best practices from staff development research (joyce and showers, 1996, sparks and richardson, 1997, darlinghammond, 1998, sparks, 1993). in the program, participants (faculty members, administrators, teaching assistants, and staff) meet bi-weekly for two hours in large group sessions (15-20 members) led by a facilitator with a strong background in learning and pedagogy. large group sessions are conducted using a learning-centered approach so that educators experience learning in the same ways their students will. participant learning is extended in several ways including individual practice in the classroom, utilization of learning partners, and collaborative inquiry into educational literature. these structures support the most critical aspect of learning: reflection (brookfield, 1995, schon, 1983 & 1987). opportunities to reflect, practice, and see themselves through the eyes of others enhance learning and enable lea/rntm participants to achieve student learning. these structures also acknowledge that learning takes time. for both personal and institutional changes to take root, assumptions must be confronted and challenged. all aspects of the project are oriented to this need for individuals to confront assumptions coupled with the opportunity and support necessary to assist in making desired changes. each aspect of the program is grounded in learning theory and has a particular curricular focus. for instance, the focus for first-time participants exploring learning theories and the application thereof includes: interactive strategies designed both to involve students in their own learning and to develop their interpersonal skills with others; effective questioning strategies; articulating purpose of instruction; identifying student learning outcomes; and planning lessons. the curricular focus for ongoing participants depends on the needs of the group and what they want to accomplish, but most typically the groups next identify the need to learn about and apply theories and strategies for assessing student learning. higher education challenge grant this manuscript focuses on work supported by a higher education challenge grant and the reaction of participants to the project lea/rntm model of support. the challenge grant supported four introductory workshops, one advanced workshop, and bi-weekly meetings during the three-year span of the project. seventy-four faculty, teaching assistants, or teaching staff participated in one of the introductory workshops including six faculty from alabama a&m university. eleven faculty, including three from alabama a&m, participated in the advanced workshop. twenty-seven of the participants were involved in bi-weekly meetings for at least one year, with some being involved for all three years of the project. jungst, licklider, & wiersema providing support for faculty who wish to shift to a learning-centered paradigm in their higher education classrooms page 72 journal of scholarship of teaching and learning (josotl), volume 3, number 3 (2003), pp. 69-81 © 2003 board of trustees of indiana university in it’s simplest form, project lea/rntm involves an introductory four-day workshop, an advanced four-day workshop, and bi-weekly two-hour meetings during the regular school year to provide additional learning opportunities and a support group for faculty who are working to apply what they have learned to their particular classroom situation. both the workshops and the bi-weekly group meetings immerse faculty in a learningcentered environment so the educators can experience many of the things their students will experience as the new paradigm is implemented in their classrooms. workshops and bi-weekly meetings are facilitated or co-facilitated by faculty and staff from the college of education who not only are well grounded in current research in learning-centered instruction, but also can assist novices with transferring research into practice. the introductory workshop focuses on comparing a competitive, teachercentered approach to a cooperative, learning-centered approach. participants frequently work in small groups during the workshop as they experience specific learning-centered strategies. the workshop also introduces participants to the necessary elements for effective teams (johnson et. al. 1998), allows them to learn specific team skills that contribute to more effective team functioning (johnson et. al. 1991), and gives them a safe place to practice teaching team skills. central to the workshop is the notion of faculty taking time to confront their beliefs about learning. in addition to experiencing several specific learning-centered strategies for enhancing student achievement, faculty have opportunities to begin to adapt the strategies for application in their own classroom settings as they work together to plan specific lessons for their particular courses. typically, these workshops involve from twelve to thirty participants from numerous disciplines. the cross-disciplinary interactions help faculty internalize the concepts of learning theories as they confront their beliefs and assumptions about teaching, learning and students. observations during the introductory workshop, participants often struggle with their own beliefs about how college-level courses should be taught. much of the struggle revolves around their own past experiences, successes, and failures with a teacher-centered paradigm (lecture format), and an ingrained sense that they must “cover” the material. curiously, there seems to be an allpervasive assumption on the part of participants that if they cover the material in lectures, students learn it. if that were entirely true, complaints about students not performing well on exams or about students not being able to transfer knowledge from one situation to another or from one course to another should be much less pervasive than they are. the struggle with beliefs usually begins to manifest itself during the second day of the workshop and is most often heard first from those whose disciplines are in the “hard sciences.” paraphrased, it goes something like this: “i can see how this might work in a history class, but i can’t see how i can use it in chemistry (or math or engineering, jungst, licklider, & wiersema providing support for faculty who wish to shift to a learning-centered paradigm in their higher education classrooms page 73 journal of scholarship of teaching and learning (josotl), volume 3, number 3 (2003), pp. 69-81 © 2003 board of trustees of indiana university etc.). i have to cover the whole book on introductory chemistry (or math or engineering, etc.), and i don’t have time to do these things.” as the workshop progresses, there usually is a softening of this stance as participants begin to experience learning in a different way and interact with colleagues from many disciplines. those who keep an open mind and genuinely want to make changes usually begin to see applications in their own classrooms for at least some of the pedagogical strategies they are experiencing. the following comments are from participants in the introductory workshops. “i was having trouble understanding how i could implement some of these concepts in the classroom and now, after this workshop and interfacing with others in the group, i think i have a better understanding of the concepts and how they can be utilized effectively in the classroom.” “i can see where some group activities can be used in a technical course. previously, i saw them as difficult to use here. i have seen the usefulness and how to facilitate people talking in a course. i can recognize the difference in questions and how to rethink and restructure questions even more than before.” “my course is field botany, so it’s intended to be holistic, relaxed, and interactive, yet i had no idea how to make it work successfully. modeling from our facilitators showed me how to bring about active student interactions yet how to reign it in and set limits if things got too off topic.” the advanced workshop introduces faculty to more complex strategies for creating learning-centered classrooms and extends their abilities to apply more highly structured activities. in addition to expanding the number of strategies faculty may use, the advanced workshop delves more deeply into research findings related to learning and learning-centered paradigms. participants continue to develop ways to adapt their own new learning for application in their classrooms. participants are asked to continue to confront their own beliefs about learning and teaching. they continue to shift their focus from a teaching-centered paradigm to a learning-centered paradigm throughout the workshop, but the resistance to that change is less pronounced. this is probably due to the fact that the advanced workshop is self-selecting for participants who already know something about learning and learning-centered classrooms because of participation in the introductory workshop. those people who elect to participate in the jungst, licklider, & wiersema providing support for faculty who wish to shift to a learning-centered paradigm in their higher education classrooms page 74 journal of scholarship of teaching and learning (josotl), volume 3, number 3 (2003), pp. 69-81 © 2003 board of trustees of indiana university advanced workshop tend to be those who genuinely want to make the shift in paradigms. they tend to be open-minded enough to realize that there are probably applications for the new ways of designing learning experiences in their classes even though they may not see them immediately. in addition, participants in the advanced workshop seem more willing to risk doing things differently in their classrooms and appear much more comfortable discussing both successes and failures with the colleagues they have come to know and value. when asked why they came to the advanced workshop and what they expected to gain, typical responses were: “i felt the need for reinforcement of the principles learned in the first workshop. also, [i] expected to get additional tools to improve my teaching. i like the people i’ve met with similar interests in improving teaching so this is a support group.” jungst, licklider, & wiersema providing support for faculty who wish to shift to a learning-centered paradigm in their higher education classrooms page 75 “my project lea/rntm group and the first workshop have given me ideas to increase the cooperative learning for the students. it has given me the language of learning and a supportive atmosphere to pursue improving labs and what the students will get out of it.” “this workshop was important for me as a way of reviewing information about applying cooperative learning in my classroom, to help consolidate and practice information and ideas i have already tried, and to deepen my understanding and appreciation of this approach. the workshop framework is an excellent way to get an overview of a variety of issues and to reaffirm my commitment to cooperative learning.” “my confidence level has increased due to clear instruction, modeling and opportunities to practice and receive feedback. because of increased confidence, i will use the interactive strategies more often and take a few risks in stretching my expectations for what students can contribute and take with them from their own learning experiences.” experience with project lea/rntm, both as a part of the challenge grant and as a part of the program since its inception, suggests that workshops are sufficient to start the faculty shift from teachingcentered to learning-centered classrooms, but the workshops by themselves aren’t sufficient to maintain the effort. to ensure that faculty have a chance to continue to build on their successes, to find out how to create successes from their failures, and to continue their own learning, ongoing support and on-site technical assistance journal of scholarship of teaching and learning (josotl), volume 3, number 3 (2003), pp. 69-81 © 2003 board of trustees of indiana university is necessary. that support is provided in the form of bi-weekly meetings during the academic year, led by the same facilitators who led the workshops. the facilitators plan each meeting as if it were a lesson for students in the classroom, as, indeed it is, complete with expected educator learning outcomes. the facilitators are committed to providing experiences for the participants that are based on learning--they expect faculty to do what faculty ask students to do including assignments to implement their learning in their own classrooms between meetings. during the two-hour meetings, participants have a chance to discuss what has worked for them as well as what needs improvement. the interaction in the group is typically rich with ideas for new approaches to creating learning-centered atmospheres in classrooms. additional readings are often assigned between meetings, and there is an expectation that participants will try a specific strategy in their classroom during the intervening time between meetings. the meetings help hold participants accountable for continuing the conversion to learningcentered classrooms as well. participants who continue to be involved with project lea/rntm through the biweekly meetings typically continue to enhance the learning-centered environments in their classrooms. those who don’t attend the meetings often make marginal or no progress in implementing changes in their classrooms after the newness of the workshops wears off. jungst, licklider, & wiersema providing support for faculty who wish to shift to a learning-centered paradigm in their higher education classrooms page 76 fifty of the seventy-four participants in the project responded to a post-project survey. those fifty led learning for students in sixty-one different courses in 18 departments from 5 different colleges. of those fifty respondents, thirty-nine had an opportunity to attend bi-weekly lea/rntm meetings. responses for those individuals were categorized by the frequency with which the respondent had attended bi-weekly meetings. categories were “always”, “frequently”, “infrequently”, and “never”. participants were asked to rate the impact of changes they made in their classrooms on a number of student behaviors. response options were “much worse”, “worse”, “no difference”, “better”, and “much better”. table 1 shows the percentages of participants who responded with “better” or “much better” for each of the student behaviors by frequency of the respondents’ participation in bi-weekly group meetings. when divided into the four categories, the number in each group is relatively small, so additional investigation is needed to substantiate statistically the differences, but the percentages of responses in each group are suggestive none-the-less. the most striking difference is between those who elected not to attend bi-weekly meetings and those who did attend, regardless of the frequency of their participation. involvement in the bi-weekly group meetings gives participants an opportunity to fine-tune their learning-centered strategies for their particular classroom settings. the meetings not only help hold participants accountable for continuing efforts to enhance student learning, but also provide the assistance and support faculty need to explore ways to turn failures into successes. journal of scholarship of teaching and learning (josotl), volume 3, number 3 (2003), pp. 69-81 © 2003 board of trustees of indiana university table 1 percent of participants reporting “better” or “much better” student behaviors based on participant frequency of attendance at bi-weekly lea/rntm support meetings student behaviors frequency of attendance interaction with other students interaction with instructor willingness to discuss material willingness to ask questions learning in general preparation for class always (n=6) 100 100 100 83 67 50 frequently (n=11) 91 82 73 73 91 45 infrequently (n=10) 90 70 90 80 90 40 never (n=12) 67 58 58 58 33 17 the pattern of participant responses for "learning in general" and "preparation for class" appears to be different from the other four student behavior categories. in all categories except "learning in general," those who always attended bi-weekly group meetings had the highest percentage of participants reporting better or much better student behavior. a lower percentage of participants who always attended reported better or much better "learning in general" observed among their students as compared with those who frequently or infrequently attended. given the findings of other researchers about improved performance of students in learning-centered classrooms (cited earlier in this paper), a 67 percent response level by those who always attended bi-weekly meetings is somewhat surprising. certainly, small sample size may be entirely responsible for the discrepancy. however, the possibility that faculty begin to hold higher expectations for student learning as a result of participation in project lea/rn warrants further study. as faculty begin to understand more about student learning they may begin to hold higher expectations for student achievement. those who always attend bi-weekly meetings for more than one year often begin to explore the use of classroom assessment techniques (angelo and cross, 1993) and other methods to assess student achievement of intended learning outcomes more frequently. these techniques provide faculty with numerous ways of quickly determining whether students are learning the important points for a class period or a unit covered. faculty who begin to use these techniques are often surprised by the amount of confusion or misunderstanding that remains with the students at the end of a class period. in more traditional classrooms, that confusion and misunderstanding may not become evident until exam time. once this discovery is made, faculty then have to decide how to address the problems. it is possible that the lower response percentage in this category is a result of faculty reacting to initial learning rather than the final learning achieved by students at the end of the course. further inquiry is needed in this area. however, the trend of lower response percentages from participants who never attended bi-weekly groups than for those who did attend holds, even for this category. jungst, licklider, & wiersema providing support for faculty who wish to shift to a learning-centered paradigm in their higher education classrooms page 77 journal of scholarship of teaching and learning (josotl), volume 3, number 3 (2003), pp. 69-81 © 2003 board of trustees of indiana university the percentages for all participants reporting better or much better "preparation for class" on the part of students was lower than the other five categories. this isn’t surprising because the initial emphasis of project lea/rntm is on approaches to learning led by faculty in the classroom rather than strategies for increasing preparation for class. however, biweekly groups do occasionally discuss the problem of how to get students to do a better job of preparing for class. it is possible that a shift in emphasis in project lea/rntm could lead to participants implementing more strategies that would improve student preparation for class. despite no direct emphasis on approaches to promote student preparation for class, higher percentages of faculty who received ongoing support through bi-weekly meetings reported that students were better prepared for class as compared with faculty who never attended biweekly meetings. this indicates it may be worth investigating, among students, their thoughts about preparation for class in more learning-centered classrooms. it is encouraging that strong percentages of all faculty who participated in on-going learning opportunities reported better student interactions with each other and with the instructor, better willingness to discuss material, and more willingness to ask questions. learning theories tell us these are all behaviors that enhance the likelihood for learning to happen. reflections several things have become apparent during the course of the project that can be of assistance to others who may implement a similar program. participation in such a project should be voluntary. those who enroll in the workshop because they have a genuine desire to change their teaching styles are much more likely to invest the time and effort necessary to be successful than are those who are “asked” to attend by their department heads. as the joke goes, “how many psychiatrists does it take to change a light bulb? one, but the light bulb has to want to change.” a facilitator with expertise in education and leading effective professional development should plan and lead the workshops and on-going meetings. not only must facilitators have a firm grounding in literature related to teaching, learning, and the learningcentered paradigm, but they also must be adept at helping faculty transfer that research to their own practices. that background is essential for helping participants maintain focus, grounding the workshops in sound research findings and for helping faculty make connections across disciplines. in addition, facilitators have the responsibility to ensure that all meetings and interactions are safe so that faculty will honestly reflect about, and share with colleagues, what is happening in their classrooms. both the workshops and the bi-weekly meetings should be conducted in a learning-centered style. much of the initial “conversion” from teachingcentered to learning-centered thinking takes place because participants are intrigued by the experience of being immersed in a learning-centered workshop. jungst, licklider, & wiersema providing support for faculty who wish to shift to a learning-centered paradigm in their higher education classrooms page 78 journal of scholarship of teaching and learning (josotl), volume 3, number 3 (2003), pp. 69-81 © 2003 board of trustees of indiana university workshops alone will not create the kind of substantial paradigm shift that is needed in higher education. workshops create the initial excitement and interest, but bi-weekly meetings maintain the interest and build on the basics learned in the workshops. they provide the ongoing support and on-site technical assistance research has shown is critical for effective faculty development (joyce & showers, 1996). and finally, university administrators who are eager to see faculty shift to a learning-centered paradigm must be willing to provide base level support for a program such as project lea/rntm. faculty must be able to depend on the availability of introductory and advanced workshops as well as bi-weekly support group meetings. that level of long-term dependability is difficult if not impossible to achieve based entirely on soft money support. final thoughts indeed, we must do a better job in higher education of preparing students to meet the challenges they will encounter after their collegiate years. it appears that faculty working and learning together in a structure like project lea/rn™ may well move us in that direction. one student, involved for several years in the learning revolution in the department of forestry, had this to say about his experience: i have gained from project lea/rn™ in several aspects, both social and academic. socially i have gained a group of friends who will be valued colleagues throughout my career. these colleagues that i learn with on a daily basis are a motivation beyond myself. i learn as fast as i can so i can be a greater asset to the group. academically, we went from being spoon-fed concepts to hunting for knowledge to feed ourselves. i can spend the rest of my life getting fat on ideas. the concepts i worked hard to gain, hold value for me. therefore, i grasp and hold onto them tenaciously." making this kind of difference with students is what we ought to be about. there is a substantial time and energy commitment on the part of educators to change post-secondary student learning experiences, but it certainly seems worth the effort. literature cited angelo, t. a., and k. p. cross. 1993. classroom assessment techniques: a handbook for college teachers, 2nd edition. jossey-bass publishers. brookfield, s. d. 1995. becoming a critically reflective teacher. san francisco: jossey-bass. brown, t. l. and j. p. lassoie. 1998. entry-level competency and skill requirements of foresters: what do employers want?. journal of forestry vol. 96, no. 2. pp. 8-14. cranton, p. 1994. self-directed and transformative instructional jungst, licklider, & wiersema providing support for faculty who wish to shift to a learning-centered paradigm in their higher education classrooms page 79 journal of scholarship of teaching and learning (josotl), volume 3, number 3 (2003), pp. 69-81 © 2003 board of trustees of indiana university development. journal of higher education vol. 65 pp. 726-744. darling-hamond, l. 1998. teacher learning that supports student learning. educational leadership vol. 55. pp. 6-11. gardiner, l. 1994. redesigning higher education: producing dramatic gains in student learning. vol. 23, no. 7. washington, dc: the george washington university, graduate school of education and human development. (ashe-eric higher education report). gough, p. b. 1987. the key to improving schools: an interview with william glasser. phi delta kappan, 68, 656-662. johnson, d. w., r. t. johnson, and holubec, e. 1991. cooperation in the classroom, edina, mn: interaction book company johnson, d. w., r. t. johnson, and k. a. smith. 1998. active learning: cooperation in the college classroom. edina, mn: interaction book company. joyce, b. and b. showers. 1996. staff development as comprehensive service organization. journal of staff development vol. 17, no. 1, pp. 2-5. knowles, m. 1984. andragogy in action: applying modern principles of adult learning. san francisco: jossey-bass. licklider, b. l., c. fulton, and d. l. schnelker. 1997. revisioning faculty development for changing times: improving teaching and learning. journal of organization, staff, and faculty development vol. 15, no. 1, pp. 17-28. marzaon, r. ju. d. j. pickering, j. e. pollock. 2001. classroom instruction that works. association for supervision and curriculum development. mezirow, j. 1991. transformative dimensions of adult learning. san francisco: jossey-bass. mullin, r. 2001. the undergraduate revolution. change, the magazine of higher learning (sept./oct. 2001) pp. 54-58. nastasi, b. k., and d. h. clements. 1991. research on cooperative learning: implications for practice. school psychology review vol 20, no. 1:110-131. national research council. 2001. how people learn: brain, mind, experience, and school. national academy press. washington d.c. 374pp. national science foundation. 1996. shaping the future: new expectations for undergraduate education in science, mathematics, engineering, and technology. report on the review of undergraduate education. washington, dc: national science foundation directorate for education and human resources. o’banion, t. 1998. the learning revolution in higher education. satellite down link produced by jungst, licklider, & wiersema providing support for faculty who wish to shift to a learning-centered paradigm in their higher education classrooms page 80 journal of scholarship of teaching and learning (josotl), volume 3, number 3 (2003), pp. 69-81 © 2003 board of trustees of indiana university r. jan lecroy center for educational telecommunications, dallas county community college district. presented by pbs adult learning satellite service, january 29, 1998. qualters, d. m. 2001. do students want to be active? the journal of scholarship of teaching and learning vol. 2, no. 1. pp. 5260. salvin, r. e. 1990. cooperative learning: theory, research, and practice. englewood, nj: prentice hall, inc. sanders, b. r. 1998. the learning revolution in higher education. satellite down link produced by r. jan lecroy center for educational telecommunications, dallas county community college district. presented by pbs adult learning satellite service, january 29, 1998. schon, d. a. 1983. the reflective practitioner: how professionals think in action. new york: basic books. schon, d. a. 1987. educating the reflective practitioner. san francisco: jossey-bass sparks, d. and j. richardson. 1997. a primer on professional development. journal of staff development vol. 18, no. 4, pp. 1-8. sparks, g. m. 1983. synthesis of research on staff development for effective teaching. educational leadership vol. 43, no. 3, pp. 65-72. spense, l. d. 2001. the case against teaching. change, the magazine of higher learning (nov./dec. 2001) pp. 11-19. jungst, licklider, & wiersema providing support for faculty who wish to shift to a learning-centered paradigm in their higher education classrooms page 81 journal of scholarship of teaching and learning (josotl), volume 3, number 3 (2003), pp. 69-81 © 2003 board of trustees of indiana university providing support for faculty who wish to shift to a learnin barbara l. licklider, and janice a. wiersema abstract introduction the model higher education challenge grant observations reflections final thoughts literature cited 503 service temporarily unavailable 503 service temporarily unavailable nginx/1.14.1 503 service temporarily unavailable 503 service temporarily unavailable nginx/1.14.1 503 service temporarily unavailable 503 service temporarily unavailable nginx/1.14.1 503 service temporarily unavailable 503 service temporarily unavailable nginx/1.14.1 503 service temporarily unavailable 503 service temporarily unavailable nginx/1.14.1 503 service temporarily unavailable 503 service temporarily unavailable nginx/1.14.1 503 service temporarily unavailable 503 service temporarily unavailable nginx/1.14.1 503 service temporarily unavailable 503 service temporarily unavailable nginx/1.14.1 503 service temporarily unavailable 503 service temporarily unavailable nginx/1.14.1 using role play simulation and hands-on models to enhance students’ learning fundamental accounting concepts beth b. kern division of business and economics indiana university--south bend p.o. box 7111 south bend, in 46634 (219) 237-4352 (phone) (219) 237-4866 (fax) e-mail: bkern@iusb.edu may 1999 using role play simulation and hands-on models to enhance students’ learning fundamental accounting concepts abstract this paper both documents and assesses the use of a role play simulation exercise that can easily be incorporated into the first week of the first course in accounting. the exercise actively involves students by having them assume roles and simulate transactions to start a business. in addition, the students also perform a record-keeping role. both the medium of exchange and record-keeping are accomplished by using white, red and blue poker chips. a primary focus of the exercise is to learn how transactions affect the fundamental accounting equation (assets=liabilities+owners’ equity). in addition, several other fundamental accounting concepts are introduced. to assess if the exercise enhances student learning, other sections of the same accounting course learned identical subject matter with a lecture format. an assessment instrument was administered to both student groups. the results indicate that role play simulation enhances students’ transfer problem solving, but there is no evidence that it enhances conceptual recall beyond that which can be attained via a traditional lecture. in addition, there is evidence that the role play simulation exercise helps student better retain problem solving learning over a four week period. 1 using role play simulation and hands-on models to enhance students’ learning fundamental accounting concepts in its call for accounting curricular reform, the accounting education change commission (aecc 1992, p. 250) states that “teachers of the first course in accounting should put a priority on their interaction with students and on interaction among students.” this paper documents and evaluates an in-class exercise using role play simulation along with hands-on learning materials to enhance student interaction while learning fundamental accounting concepts. assessment of this in-class exercise indicates that it does enhance students’ transfer problem solving1 capabilities, but there is no evidence that it enhances conceptual recall beyond that which can be attained via a traditional lecture oriented learning environment. 1 transfer problem solving is the ability to transfer what one has learned to solve new problems. 2 role playing typically involves unstructured situations in which students improvise behavior according to their assigned roles (mckeachie 1994, p. 167). simulations are usually defined more precisely with guiding principles, rules and structured relationships (bonwell and eison 1991, p. 47). role play simulation entails assigning students to unique roles within a group as the group addresses a series of issues (deneve and heppner 1997, p. 234). role play simulations can accomplish multiple objectives. these are: “(a) to arouse student interest for a particular field of study, (b) to help students apply material learned in class, (c) to help students develop insight into the group dynamics of problem solving situations, (d) to give students a chance to develop leadership skills, and (e) to provide students with a working grasp of the scientific method” (deneve and heppner 1997, p. 234). while role play simulation can accomplish a number of objectives, the focus of this study is assessment of enhanced student learning. since the accounting education change commission’s (aecc 1990, p. 309) call for active learning strategies in the accounting curriculum, a number of studies have documented their use. these include the use of cooperative learning (ravenscroft, et. al. 1995; peek et. al. 1995; ciccotello et. al. 1996; lindquist and abraham 1996; hite 1996), simulation (knechel and rand 1994; albrecht 1995), and writing (scofield and combes 1993). while there has been a notable increase documenting the use of active learning strategies, much work remains in terms of assessing the conditions under which they are effective in the accounting curriculum. to fully understand the advantages of active learning strategies, one must assess the conditions under which these strategies are effective. their effectiveness is dependent upon at least four characteristics: (1) the subject matter, (2) the educational setting, (3) the specific learning materials, (4) the desired learning outcomes (deneve and heppner 1997, p. 130). 3 first, with regard to using role play simulations or role plays in an accounting context, several studies have either advocated their use or provided specific examples. tomassini (1974) examines the use of role play simulation in accounting. more recent studies document using role play simulation with monopolytm (albrecht 1995; knechel and rand 1994). with regard to role plays (rather than role play simulations), craig and amernic (1994) examine their use in the context of using accounting information in labor disputes. haskins and crum (1985) document the use of role play in a cost allocations setting. assessment of these methods is relatively sparse, however. one exception is craig and amernic who assess role play by examining student opinions. this study extends upon prior work by not only documenting the use of role play simulation, but by also providing an assessment of student learning. second, students can learn under a variety of learning settings. these range from passive lectures to more active strategies such as role play simulation or cooperative learning. this study differs the learning environment by presenting the same fact scenario with a lecture versus using a role play simulation. third, the learning materials that students use can vary from abstract characterizations to having more realistic hands-on materials (ferguson and hegarty 1995). in this study, students who participate in the lecture learning environment are able to observe a problem being worked on a blackboard. the students who participate in the role play simulation also are able to observe the same material on the blackboard, but are also able to manipulate poker chips as reinforcement. thus, this study jointly assesses the impact of using a more active learning environment along with hands-on learning materials on students’ learning. fourth, it is possible for some learning strategies to affect some learning outcomes and not others. learning outcomes are the knowledge students gain as a result of the learning 4 experience. this study focuses on students’ learning and understanding fundamental relationships in accounting systems. mayer (1989) has performed extensive analysis of ways to enhance students’ learning scientific and computer systems. his research focuses on the usefulness of providing students models as aids to help them understand systems. these models can be explained with words or diagrams. he has found that students who are given model instruction about scientific systems demonstrate enhanced transfer problem solving and conceptual recall. they are less likely, however, to be able to recall verbatim information such as a list of details. in the aecc’s call for improving accounting education, it posits that “the focus should be on developing analytical and conceptual thinking, not on memorizing professional standards” (aecc 1990, p. 308). a focus on helping students form mental models may be an avenue to accomplish the focus on accounting knowledge acquisition for which the aecc calls. in addition, mayer (1989, p. 44) notes that novice students, such as those in the first accounting course, are more likely to benefit from instruction which fosters model building since they are less likely to already possess sophisticated conceptual models which may conflict which those presented during instruction. both instructional settings in the experiment (traditional lecture and role play simulation) attempt to help students form mental models of the fundamental accounting equation (assets=liabilities+owners’ equity). the focus of the study to determine if the more active learning environment along with the hands-on learning material enhances students’ ability to understand the model and thereby perform better in terms of problem solving and conceptual recall. description of role play simulation 5 this in-class exercise assigns students to perform roles within a set of pre-specified transactions forming a simulation of the formation of a new business. it can easily be incorporated in the first week of the first course in accounting and facilitates a high level of interaction among students who often do not know each other prior to attending the course. most of the students in the class are divided into groups of three that form a business. the students in the firm-groups all perform a record keeping and/or transaction engaging role in the role play simulation. one member of each business group is given a cupful of red chips; another is given a cupful of blue chips. the students who are not assigned to roles within a firm are assigned to banker, supplier, landlord, customer and business owner roles. each of these “volunteers” is given a cupful of white poker chips. they are told that these chips represent resources. one student is assigned to the banker, supplier, landlord and customer roles. the number of owner roles is equal to the number of firm-groups. thus, each member of the class has a role. to facilitate formation, usually the firm-group roles are assigned after students volunteer for the banker, supplier, customer, landlord and owner roles. each member of the class is then given a handout which lists a series of twelve transactions in which the business will engage. these twelve transactions are provided in appendix 1. each business is beginning its operations. the students are asked what resources the business has at this time. it is obvious that each firm has none. they are then asked to look about the classroom. all the potential resources that a firm needs to begin a business are external to the firm; the firm must engage in transactions with these parties in order to procure the resources necessary to start a business.2 2for schools who have adopted a contracting perspective for their accounting curriculum, 6 this exercise provides an excellent illustration of these concepts. 7 the record-keeping function is then introduced. the student in each business group who does not have any poker chips is assigned the task of keeping track of the resources as they are procured and disbursed. the concept that there must be a source for every resource is discussed. the importance of keeping track of the sources of resources, liabilities and owners’ equity, is addressed. finally, the fundamental relationship between assets, liabilities and owners’ equity is discussed with the fundamental accounting equation (assets=liabilities+owners’ equity) written across the top of the blackboard. the students then engage in transactions using the poker chips as the medium of exchange and to perform the record keeping function. for example, the first transaction has the owner contributing $10,000 to the business and receiving stock in return. each owner gives each business ten white poker chips representing $10,000 in cash.3 the student in charge of assets stacks the assets on a common area designated for keeping track of assets, liabilities and owners’ equity. the student responsible for the blue chips also stacks ten blue poker chips indicating that owners’ equity increased. after each transaction, the students are asked to verify that the accounting equation remains in balance by stacking their liabilities (red chips) on top of their owners’ equity (blue chips) and noting that the stack is the same height as the firm’s assets (white chips).4 several accounting concepts are introduced with the twelve transactions in the exercise. 3each poker chip represents $1,000. 4the poker chips are reintroduced later in the course to illustrate the relationship between the income statement and the balance sheet by recording the portion of transactions relating to the income statement with blue chips on one side of the classroom with the balance sheet portion on the other. when the income statement poker chips return to the balance sheet via retained earnings, the balance sheet balances. (this is illustrated by verifying that assets (white chips) are equal to liabilities (red chips) and owners’ equity (blue chips). 8 these include, but are not limited to: the accounting entity concept, revenue recognition, the matching principle, operating activities, investing activities and financing activities. a complete description of the concepts illustrated and discussed with each transaction is provided in appendix 1. 9 assessment experimental design six sections of an introductory accounting course at a medium-sized midwest public university participated in an experiment over three semesters to assess the effectiveness of using role play simulation to teach fundamental accounting concepts. some of the students participated in the role play simulation exercise as described above; the remainder discussed the identical twelve transactions in a lecture-oriented format. the concepts and material presented in the classroom experience in the lecture-oriented format were identical in content to those of the role play simulation. the instructor had notes for each transaction which specified what items needed to be addressed in each class. similar to the role play simulation experience, the fundamental accounting equation (assets=liabilities + owners’ equity) was written across the blackboard in the classroom, and the students participated in figuring out how each transaction affected the accounting equation. these classes, however, did not have poker chips and did not have students actively assuming roles. immediately following the classroom activity, each student was asked to answer several questions using the assessment instrument in appendix 2. the students were not told that they would be answering questions based on their classroom experience until after it was completed. to increase the assessment’s salience, the students were told that their score on these questions would replace their lowest homework grade if their score was better than the lowest homework. all sections participating in the experiment had the same instructor, assessment instrument and performance incentives. both the lecture-oriented exercise and role-play simulation consumed approximately one hour of class time. the remaining 15 minutes of the 75 minute class were used for assessment and classroom housekeeping. the effect of alternative classroom 10 environments on retained learning was also assessed. approximately four weeks after the classroom presentation, the students answered a question on their first exam in the course that built upon the material.5 all six sections had the same intervening classroom material with all using the same teaching methods from the day after the experiment to the day before the first exam. analysis assessment instrument results the assessment instrument administered immediately following each classroom presentation has two types of questions. the first five questions assess transfer problem solving involving the fundamental accounting equation. the second five questions assess conceptual recall. 5this assessment was conducted over three semesters. to mitigate the effects of subsequent sections learning from prior exams, the exam question was altered in terms of order and the specific assets, liabilities, owners’ equity, revenues and expenses that were affected by each transaction. in addition, both the lecture-oriented and simulation formats were spread over the three semesters. empirical assessment indicates that there is no reason to believe that the carryover effect from one exam to the next is greater for students in one classroom presentation over another. 11 table 1 displays the mean number of correct student responses for each classroom presentation by the type of assessment question. in all cases, the mean number of correct responses is higher for the students who were able to participate in the role play simulation class than for those that experienced a more lecture-oriented format. statistical analysis reveals that the students participating in the role play simulation exercise answered significantly more questions correctly than those that participated in a more lecture-oriented format. when the questions are disaggregated into those that focus on problem solving versus those that focus on conceptual recall, however, only the problem solving questions show a statistically significant higher number of correct responses. students’ retention is also assessed with an exam four weeks after the material was presented. the retention instrument (see appendix 3) focuses on problem solving since that is the only learning outcome that showed significant differences in learning outcomes initially. the mean score is higher for the role play simulation learning environment at a mildly statistically significant level. insert table 1 here further analyses were conducted to mitigate the potential confounding factor that students in the simulation sections perhaps by chance have a higher aptitude for accounting. to control for this possibility, analysis of covariance (ancova) is performed with the number of questions answered correctly as the dependent variable. the independent factor is the classroom learning environment (role play simulation v. lecture). the covariate is the student’s aggregate number of points for the course excluding the points for the exam question pertinent to the subject material in the experiment. use of overall performance in the course not only helps mitigate the effects of differing levels of accounting aptitude across the two classroom 12 environments, but it also subsumes other factors which may influence student performance.6 for example, wooten (1998) finds differences in traditional and nontraditional students. mutchler et. al. (1987) find that gender may be related to student performance. lipe (1989) finds that interaction between the gender of the instructor and that of the student affects student test scores. buckless et. al. (1991), however, find no evidence of a gender effect once they controlled for academic aptitude. thus, use of a measure of course performance may help mitigate the multiple factors which can vary across groups of students that may influence student performance.7 the results displayed in table 2 are similar to those shown in table 1. the learning environment (role play simulation v. lecture) is statistically significant even after controlling for overall accounting aptitude. this is predominately true for the problem solving questions administered using an assessment immediately following the classroom experience; there remains no evidence of enhanced conceptual recall. these results mirror those found without attempting to control for overall student aptitude. the results for assessing retention show a statistically significant learning environment effect after controlling for overall student aptitude. the simulation learning environment was able to help students of similar accounting aptitude retain the information in a way that allowed them to perform better on an exam administered four weeks later. 6gpa and sat scores were also used as covariates with no material impact on the results. if the role-play simulation led to better overall classroom performance (and therefore better test scores) for the rest of the material in the course, that would tend to bias the results toward not finding significance. the test scores, however, may be a better measure of accounting aptitude with the caveat that they may be biasing the results away from significance. 7additional factors such as gender and traditional v. non-traditional students were also added to the model to assess if there was an effect over and above what was captured by the covariate. these factors were consistently not significant echoing the conclusions reached in buckless et. al. (1991). 13 insert table 2 here student self-assessment students were asked if they perceived either the problem solving or conceptual questions on the assessment instrument to be more difficult. results from chi-square tests are displayed in table 3. they indicate that the students perceived that the conceptual questions were more difficult. further partitioning of the sample by classroom learning environment (role play simulation v. lecture), gender, and traditional v. non-traditional students does not reveal evidence that this perception varies across these factors. in addition, their perceptions reflect their actual performance on the assessment instrument in that they answered 81.08% of the problem solving questions correctly in comparison to answering 75.26% of the conceptual questions correctly. the difference is statistically significant at less than the .01 level. insert table 3 here concluding remarks this study indicates that an active learning environment using role play simulation along with hands-on student learning materials can enhance students’ transfer problem solving using fundamental accounting concepts. students’ learning was assessed both immediately after the classroom learning experience and several weeks following introduction of fundamental concepts. assessment indicates that the active learning environment in which students simulated performing transactions helped them understand the material better than when the same material was presented in a lecture-oriented environment. in addition, when one controls for student accounting aptitude, the students who participated in role play simulation performed better on an 14 exam four weeks following the class than those who learned the material with a lecture-oriented format. these results do not, however, extend to questions that assess conceptual recall. much more work remains to understand the circumstances under which more active learning environments lead to enhanced student learning. in this setting, evidence indicates that the role play simulation along with hands-on learning materials enhanced students’ transfer problem solving learning. further research is needed to assess if this result holds for other topics in the first accounting course. additional research is needed to assess if more advanced accounting students can benefit from the use of active learning environments to enhance their problem solving abilities. the results do not show evidence that the more active learning environment enhanced conceptual recall. the students performed more poorly on the conceptual recall questions than the problem solving questions for both environments. it may be that role play simulations offer little advantage over lecture-oriented environments which stress the key components of and relationships between accounting concepts. alternatively, another learning environment may be superior to either of those explored in this study to enhance conceptual recall. further work exploring the classroom environments and methods that would enhance this type of knowledge is needed. 15 16 table 1 differences in mean performance between students who participated in a role play simulation exercise versus those who participated in a more lecture-oriented classroom presentation mean number of correct responses performance measure simulation lectureoriented tstatistic all questions 8.05 7.46 2.08** (maximum correct =10) (91) (59) problem solving 4.25 3.76 2.44** questions (91) (59) (maximum correct = 5) conceptual recall 3.81 3.69 .73 questions (91) (59) (maximum correct = 5) retention assessment 20.09 19.27 1.67* (maximum points = 25) (80) (55) the values in the parentheses are the number of students in each group. four students dropped the course before the first exam in the lecture-oriented group during the experiment. in the simulation environment, two students did not take the first exam; nine dropped the course. **difference in the means is significant at less than the .05 level with a two-tailed t-test. *difference in the means is significant at less than the .10 level with a two-tailed t-test. 17 table 2 analysis of covariance assessing the efficacy of role play simulation with hands-on learning material versus traditional lecture panel a: dependent variable is all questions degrees effects ss of freedom f-statistic p-value within group 266.47 114 covariate (course score)a 27.16 1 11.62 .001 learning environmentb 6.64 1 2.84 .095 panel b: dependent variable is transfer problem solving questions degrees effects ss of freedom f-statistic p-value within group 130.28 114 covariate (course score)a 5.85 1 5.12 .027 learning environmentb 5.92 1 5.18 .025 panel c: dependent variable is conceptual recall questions degrees effects ss of freedom f-statistic p-value within group 97.43 114 covariate (course score)a 7.80 1 18 9.13 .003 learning environmentb .02 1 .02 .877 panel d: dependent variable is problem solving exam question after four weeks degrees effects ss of freedom f-statistic p-value within group 624.33 114 covariate (course score)a 111.35 1 20.33 .000 learning environmentb 17.62 1 3.22 .076 athe covariate is the student’s aggregate grade excluding the points allocable to the exam question pertaining to this material. (the maximum number of points is 435). blearning environment was one of two classroom experiences: (1) learning the material through role playing simulation or (2) learning identical material via a more lecture-oriented format. 19 table 3 student perceptions of whether problem solving or conceptual recall assessment questions are more difficult panel a: tabulation of student perceptions: entire sample perception of which is more difficult problem solving conceptual recall perception of which is more difficult 36 111 χ2 : 38.27 (p=.000) panel b: tabulation of student perceptions: by classroom learning environment perception of which is more difficult problem solving conceptual recall learning environment role play simulation 23 66 lecture-oriented 13 45 pearson χ2: .22 (p=.698) panel d: tabulation of student perceptions: by gender perception of which is more difficult problem solving conceptual recall gender female 15 20 54 male 21 57 pearson χ2: .53 (p=.565 panel c: tabulation of student perceptions: by traditional v. non-traditional students perception of which is more difficult problem solving conceptual recall student typea traditional 28 70 non-traditional 8 41 pearson χ2: 2.65 (p=.154) anon-traditional students are those who are age 25 or older. this definition is used by the university in the experiment and also by wooten (1998). 21 appendix 1 classroom material used as a basis for simulation roleplay exercise starting a business: business transactions ima hoosier is starting a new consulting business. the following events occurred when she started her new business: before the exercise begins, students volunteer to assume the roles of banker, supplier, customer, landlord and owners. one student is assigned to each role with the exception of owners. the number of students assigned as owners is equal to the number of firms. each one of these parties receives a cupful of white poker chips. each white chip represents $1,000 of assets. the remaining students are divided into groups of three. one is assigned the task of record-keeping. the other two are given either red or blue poker chips. red poker chips represent liabilities; blue chips represent owners’ equity. before engaging in transactions, the groups are asked to indicate what resources each business has. since the firms are beginning operations, no firm has any resources. they are then asked to look about the classroom to locate potential resources. all resources (white chips) at this point are held by parties external to the firm. the students are then told that they will engage in transactions which will allow them to acquire and/or use their firm’s resources. the fundamental accounting equation is introduced. the words, assets, liabilities and owners’ equity are written across the top of a blackboard. the concept that every resource has a source and that source can originate from two potential places is introduced. the fundamental accounting equation (assets=liabilities+owners’ equity) is introduced. the effect of each transaction on assets, liabilities and owners’ equity is written under the headings for each transaction. 1) she contributed $10,000 cash to the business and the corporation distributed stock worth $10,000 to the owner. in addition to being the first example of how transactions affect the accounting equation, the entity concept is introduced and highlighted with this transaction. the owner’s personal assets not invested in the firm remain with each owner. it is noted that some owners have more resources (white chips) than others. this does not affect the amount of resources that are allocated to the firm because of the entity concept. 2) the firm bought $3,000 of supplies with cash. students realize upon visiting the supplier that they are merely exchanging one asset for another. there is no net change in assets. liabilities and owners’ equity remains unaffected. 3) the firm borrowed $5,000 cash from the bank. the concept of liabilities is further refined. 4) the firm bought a truck for $4,000 paying for it with cash. after the students engage in this transaction, the similarities between (4) and (2) are discussed. 5) the business realized it did not need as much cash as she thought she would, so it paid off $2,000 of the bank debt. this transaction highlights the effect of paying liabilities has on the firm’s overall resources. at this point, the following question is posed. if the firm were to cease being a business, how much in resources would the owner receive? the students realize that the owner would only receive her original investment back. the concept of owners having a residual interest is introduced. the students are asked if owners would be 22 satisfied over the long-run if they were only to receive their original investment. the notion that the owners are bearing risk to engage in this business is discussed. in return for bearing the risk, the owners are hoping to achieve a higher return for their investment than would be attainable through other investment vehicles. then, the students are asked how businesses generate returns for their owners. the notion of profit is introduced and that a business must perform a service or sell a product to generate profits. the concepts of operating, investing and financing activities are also introduced. it is noted that up to this point the firm has only engaged in financing and investing activities. to earn a return for its owners, the firm must engage in operating activities. 6) the firm performed services for customers and received $5,000 cash. students have difficulty deciding how balance the accounting equation. they are asked who would receive the $5,000 if the firm were to liquidate. the concept that owners’ equity can be divided into two components, contributed capital and retained earnings, is introduced. revenues are introduced. 7) the firm paid $1,000 rent in cash. the concept of expenses is introduced. 8) the firm performed $3,000 of services for customers and billed them for them. revenue recognition and the difference between cash and accrual basis is introduced. 9) the firm collected $2,000 of the amounts owed from its customers. the class discusses why the collection from a customer is not a revenue. the impact of this transaction on the accounting equation is emphasized. 10) the firm used $1,000 of the supplies. expenses and the matching principle are discussed. 11) the firm purchased a $4,000 computer on credit. liabilities are reinforced. the term, accounts payable, is introduced. ===================================================================== 12) the firm paid ima a $2,000 dividend. there is a discussion focusing on why dividends are not considered expenses. the notion that a firm has no legal obligation to pay dividends is introduced. in addition, the students are asked how investors are able to achieve a return on a stock investment. dividends and stock appreciation are discussed. growth stocks are also discussed at this point. note: the italicized descriptions below each transaction indicate what concepts were discussed in class for each transaction. 23 appendix 2 assessment instrument 1. red company borrowed $10,000 from a bank. a. assets decreased by $10,000 and liabilities decreased by $10,000. b. assets increased by $10,000 and liabilities increased by $10,000. c. assets increase by $10,000 and owners’ equity increased by $10,000. d. there was no net change in total assets. e. none of the above. 2. red company bought a computer for $5,000 in cash. a. assets increased by $5,000 and liabilities decreased by $5,000. b. assets increased by $5,000 and liabilities increased by $5,000. c. assets increased by $5,000 and owners’ equity increased by $5,000. d. there was no net change in total assets. e. none of the above. 3. red company performed services for customers and received $6,000 in cash. a. assets increased by $6,000 and liabilities increased by $6,000. b. assets increased by $6,000 and owners’ equity increased by $6,000. c. there was no net change in total assets. d. assets decreased by $6,000 and liabilities decreased by $6,000. e. none of the above. 4. red company performed $10,000 of services for a joe blue and billed him for it. payment is expected in a few weeks. a. this transaction is not recorded. b. assets increased by $10,000 and liabilities increased by $10,000. c. assets increased by $10,000 and owners’ equity increased by $10,000. d. there was no net changed in total assets. e. none of the above. 5. red company received $10,000 from joe blue for the services that they had performed for him in number 4. a. assets increased by $10,000 and owners’ equity increased by $10,000. b. assets increased by $10,000 and liabilities increased by $10,000. c. there was no net change in total assets. d. this transaction is not recorded. e. none of the above. 24 6. expenses are recognized: a. when they are paid in cash. b. when management cares to do so. c. in the time period in which the costs are used up. d. none of the above. 7. revenues are recognized: a. when the cash is received. b. when management cares to do so. c. when the service is performed or the product is sold. d. none of the above. 8. the entity concept in accounting: a. ensures that an owner’s personal assets and liabilities will be recorded separately from a firm’s assets and liabilities. b. requires that transactions that involve an exchange of value be kept separately from those that do not. c. requires that tax records be kept separately from accounting records. d. none of the above. 9. transactions are recorded in an accounting system: a. when there is an exchange between the firm and an outside party. b. when management feels it is appropriate to do so. c. only when it will cause a net change in either assets, liabilities or owners’ equity. d. none of the above. 10. with regard to owners’ rights: a. a corporation has a legal obligation to pay them a dividend every year. b. they have a residual claim to the assets of the business. c. should the firm go bankrupt, the owners’ initial investment in the firm is paid prior to that of the creditors. d. none of the above. 11. which questions do you feel were easier to answer? a. 1-5. b. 6-10. 12. have you taken an accounting class before? a. no. b. yes, in high school only. c. yes, i tried once before in college. 13. how old are you? a. 25 or older. b. less than 25. 25 appendix 3 instrument assessing concept retention for each of the transactions given below, indicate the effect on assets, liabilities, stockholders’ equity, revenues, expenses, and net income by entering a plus (+) for increase, a minus (-) for decrease, and a blank for no effect. transactions assets liabilities stockholders’ equity revenues expenses net income a issued capital stock for cash. b performed services for a customer who will pay us later. c received a telephone bill for telephone service for this month. d received cash from the customer in (b) e purchased truck on credit f paid the telephone bill mentioned in (c). g recognized the depreciation on equipment h declared and paid cash dividends. i paid next year’s insurance in advance. 26 bibliography accounting education change commission (aecc). 1990. objectives of education for accountants: position statement number one issues in accounting education (fall): 307312. accounting education change commission (aecc). 1992. the first course in accounting: position statement number two. issues in accounting education (fall): 249-252. albrecht, w. d. 1995. a financial accounting and investment simulation game. issues in accounting education (spring): 127-141. bonwell, c.c. and j.a. eison. 1991. active learning: creating excitement in the classroom. ashe-eric higher education report no. 1. washington, d.c.: the george washington university, school of education and human development. buckless, f., m. lipe, and s. ravenscroft. 1991. do gender effects on accounting course performance persist after controlling for general academic aptitude. issues in accounting education (fall): 248-261. ciccotello, c. s., d’amico,r. j. and c. t. grant. 1997. an empirical examination of cooperative learning and student performance in managerial accounting. accounting education (spring): 27-43. craig, r. and j. amernic. 1994. roleplaying in a conflict resolution setting: description and some implications for accounting. issues in accounting education (spring): 2844. deneve, k. m. and m. j. heppner. 1997. role play simulations: the assessment of an active learning technique and comparisons with traditional lectures. innovative higher education (spring): 231-246. ferguson, e.l. and m.h. hegarty. 1995. learning with real machines or diagrams: application of knowledge to real-world problems. cognition and instruction 13(1): 129-160. haskins, m.e. and r.p. crum. 1985. cost allocations: a classroom role-play in managerial behavior and accounting choices. issues in accounting education: 109-130. hite, p. 1996. an experimental study of the effectiveness of group exams in an individual income tax class. issues in accounting education (spring): 61-76. knechel, w. r. and r. s. rand. 1994. will the aecc’s course delivery recommendations work in the introductory accounting course? some preliminary evidence. journal of accounting education (summer): 175-192. 27 lindquist, t. m. and r. j. abraham. 1996. whitepeak corporation: a case analysis of a jigsaw ii applicaton of cooperative learning. accounting education: a journal of theory, practice and research 1: 113-125. lipe, m. 1989. further evidence on the performance of female versus male accounting students. issues in accounting education (spring): 144-152. mayer, r. e. 1989. models for understanding. review of educational research (spring): 43-64. mckeachie, w. j. 1986. teaching tips: strategies, research, and theory for college and university teachers (9th edition). lexington, massachusetts: d.c. heath and company. mutchler, j., j. turner, and d. williams. 1987. the performance of female versus male accounting students. issues in accounting education (spring): 103-111. peek, l.e., winking, c. and g. s. peek. 1995. cooperative learning activities: managerial accounting. issues in accounting education (spring): 111-126. ravenscroft, s. p., buckless, f. a., mccombs, g. b. and g. j. zuckerman. 1995. incentives in student team learning: an experiment in cooperative group learning. issues in accounting education (spring): 97-110. scofield, b.w. and l. combes. 1993. designing and managing meaningful writing assignments. issues in accounting education (spring): 71-85. tomasini, l.a. 1974. the use of role play simulation in accounting education. in accounting education: problems and prospects, ed. j.d. edwards, american accounting association: 376-382. wooton, t. c. 1998. factors influencing student learning in introductory accounting classes: a comparison of traditional and nontraditional students. issues in accounting education (may): 357-373. 503 service temporarily unavailable 503 service temporarily unavailable nginx/1.14.1 ten years ago ernest boyer and the carnegie foundation for the advancement of teaching released the book scholarship reconsidered: priorities of the professoriate (1990). a small book of less than a hundred pages, it was a manuscript with very big ideas that would begin what some would see as a revolutionary change in how those in the academy would view the role of the professoriate. in campuses across the country, from level i research universities to community colleges, colleagues began to discuss the boyer model and its implications for their scholarly endeavors, their teaching, and their professional lives. for the first time in fifty years our profession began to reflect seriously on the potential for new roles and rewards within the institutions. the academic revolution of the 1940's and 1950's (jencks and riesman, 1968) which had established a climate based on research as legal tender was now being challenged by a new revolution which suggested a broadened view of the professorate. for a generation of academics who had been raised on the centrality of discovery research the possibilities of a broadened view of scholarship raised questions and sparked debate. on some campuses this was a civil intellectual exchange welcomed by many in the academic community: these discussions often led to changes in the climate of the college or university. other campuses experienced greater resistance and, perhaps, less civility in the discussion. but in the past decade few campuses in america have escaped the impact of the boyer model. a brief history of the scholarship of teaching and learning and josotl ernest boyer and his colleague gene rice started the fire and kept it going at the national level. others at the carnegie foundation for the advancement of teaching ( http://www.carnegiefoundation.org/ )and the american association of higher education (http://www.aahe.org/) expanded the conversation with publications, conferences, and new initiatives. gene rice’s work at aahe with faculty roles and rewards and his working paper “the new american scholar” framed the agenda for faculty and administrators in higher education. as the conversation proceeded questions were raised about the legitimacy of teaching in terms of scholarship: what criteria should be used to assess valid scholarship? scholarship assessed (glassick, huber, and maeroff, 1997) helped answer these concerns by offering six criteria for evaluating all types of scholarship. in 1997 lee shulman became president of the carnegie foundation and in 1998 the carnegie academy for the scholarship of teaching and learning (castl) was launched. castl is a project which focuses on the development and encouragement of programs in the scholarship of teaching and learning in institutions of higher education and k-12 programs. the goals of the program are to:  foster significant, long-lasting learning for all students  enhance the practice and profession of teaching  bring to faculty members’ work as teachers the recognition and reward afforded to other forms of scholarly work to meet these objectives castl has launched three programs designed to: recognize and foster the work of outstanding faculty who are involved in the scholarship of teaching and learning; work with the scholarly and professional societies to advance the scholarship of teaching within the professional disciplines; and support campus initiatives in the scholarship of teaching and learning. the “pew national fellowship program for carnegie scholars” selects and brings together outstanding faculty to create a community of scholars in teaching and learning. the purpose of the pew scholars program is to support, foster, and deepen the profession of teaching and the learning of students. the “work with scholarly and professional societies” program is designed to provide networking opportunities for scholarly and professional organizations to support teaching and learning. castl will also afford these organizations grants to share findings in the scholarship of teaching and learning, encourage extensive peer review, and support graduate programs to be more responsive to the scholarship of teaching and learning. the “teaching academy campus program” was the first castl initiative and also the impetus to create the journal of scholarship of teaching and learning (josotl). in the fall of 1998 the carnegie foundation and the american association for higher education invited institutions of all types from higher education to join in a discussion on the scholarship of teaching. the first step, level i: campus conversation, invited any interested campus to draft a definition of the scholarship of teaching and identify supports for and barriers to the scholarship of teaching and learning. level ii: going public asked campuses to widen their circle by sharing their work including peer review and collaboration. level ii initiatives include seed grants to help campuses communicate their outcomes; this level of the program will continue through december 2001. in 2002 the campus program will begin level iii: national teaching academy by inviting selected campuses to become initial members of the national teaching academy. the journal of scholarship of teaching and learning started as an idea spawned at a conference for the faculty colloquium on excellence in teaching (facet) ( http://www.iusb.edu/~facet/ ). a number of colleagues from indiana university south bend were discussing how our campus could make the scholarship of teaching and learning more public on our campus. before long the discussants became more ambitious and faculty suggested that we widen our scope to communicate our findings to our facet colleagues from across the state. a number of faculty in the discussion had web pages and when gary kern, a management information systems professor, suggested we could put a journal on the web, josotl was conceived. the birthing process has taken a little more time and effort. the initial issue of josotl has been created in the past six months out of an interest in creating a forum that would encourage faculty to share their knowledge and inquiry into the teaching learning process. when we first began this endeavor our focus was the faculty on the eight campuses of indiana university. we quickly learned the world wide web offered us the opportunity to reach out beyond indiana university and our state to include colleagues from around the world. as soon as josotl was put on the web (before any articles were included) inquiries arrived from across the nation and around the world. i like to think the creative touch of our web-master (gary kern) inspired colleagues to contact us but the overwhelming response rate is also indicative of the widespread interest in the scholarship of teaching and learning. clearly, a vehicle is needed to allow scholars of teaching and learning to share their work in a public forum to build a knowledge base for a growing scholarly community. the revolution that began ten years ago with scholarship reconsidered has already had a dramatic impact on higher education. however, there is still a great deal of confusion and misperception surrounding what is meant by the scholarship of teaching and learning. some colleagues have argued excellent teaching qualifies as the scholarship of teaching. others claim any professor who is an expert in the pedagogy of their discipline is really demonstrating the scholarship of teaching. while there is still much to be learned about the scholarship of teaching and learning we at the josotl believe we should offer a general definition of the scholarship of teaching and learning to begin the discussion. this is not intended to thwart a discussion of definitions, or other debates, about sotl but rather as a starting point and we invite colleagues to send us an essay or challenge us on our threaded discussion page. to begin the discussion we turn to lee shulman for a definition. defining the scholarship of teaching and learning and making it public as we begin the new century we have an opportunity to promote the scholarship of teaching and learning in an environment which is more receptive to the significant contributions of the study of effective instruction, student learning, and the expertise of the pedagogues of the disciplines. to take advantage of the opportunity to define the role of the scholarship of teaching and learning it is critical to start with a general definition of scholarship. lee shulman, the president of the carnegie foundation, describes scholarship as disciplined inquiry and invention that has clear characteristics whether the scholarship be discovery, integration, application, or teaching (1998): “for an activity to be designated as scholarship, it should manifest at least three key characteristics: it should be public, susceptible to critical review and evaluation, and accessible for exchange and use by other members of one’s scholarly community. we thus observe, with respect to all forms of scholarship, that they are acts of mind or spirit that have been made public in some manner, have been subjected to peer review by members of one’s intellectual or professional community, and can be cited, refuted, built upon, and shared among members of that community. scholarship properly communicated and critiqued serves as the building blocks for knowledge growth in a field.” (p. 5) in the last analysis, however, the scholarship of teaching is best defined through actual practice. the purpose of the journal of scholarship of teaching and learning is to serve the professional community by giving individuals within our professional community an opportunity to make their own work public, open to critical review and evaluation, and accessible to a wide spectrum of colleagues from many disciplines. as we discussed the creation of josotl we realized the technology of the electronic journal could make a unique contribution to further the development of the scholarship of teaching and learning. paper journals invariably lead to a time lag of months, if not years; an electronic journal allows knowledge to be disseminated almost immediately. paper journals are extremely costly and tend to be discipline specific; an electronic journal is practically free and is accessible by anyone. paper journals advance the knowledge base through the dissemination of information but do little to directly facilitate communications between colleagues; while electronic journal allows colleagues from distant settings and diverse fields to have real time communications in an open forum. thus, the goal of the journal of scholarship of teaching and learning is to create a forum without boundaries for colleagues from any discipline to join the discussion of the teaching-learning process. we believe it is extremely important to invite all of our colleagues to join the discussion. this would include teachers from any discipline; there is much to learn across disciplines. we have noted an increasing number of colleagues have begun serious discussion on effective teaching within their own disciplines and they have much to teach those of us outside their field of study. we also believe it is critical to invite colleagues from all levels of “expertise” into the discussion. teaching has always been viewed as a private activity that is seldom discussed in an open forum. this has created an army of experts who, ironically enough, seldom share their teaching knowledge and may not realize the value of their own private learning. we need to invite these “silent” experts to share what they have learned about the teaching-learning process. even if this knowledge is more intuitive than empirical it can be instructive as part of the larger collegial exchange. our goal is to create a forum that invites colleagues to share their work through their writing but also share their expertise and experience through their critique of, and commentary on, other’s work. as information on the world wide web expands it is important for the academic community to carefully evaluate and screen information for rigor as we have always done through the peer review process. the advantages of the electronic medium (speed and availability) must not be undermined by the potential weaknesses (flood of data without peer review.) because we at josotl believe it is extremely important to uphold the standards of scholarship, we use double-blind reviews for all the articles we publish. at the same time, we also want to take advantage of the strengths of speed and availability which an electron journal affords by including our readers in the peer review process. the technology of the web allows for interactions to be held on-line through a threaded discussion that invites comments and questions from any reader. these critiques will add a new dimension to the peer review process. we believe the scholarship of teaching and learning can have an enormous impact on our understanding of the teaching-learning process, how we deliver instruction, and the learning which occurs within and outside our classes. we also believe the electronic format of the journal of scholarship of teaching and learning can facilitate this impact by creating a wider circle of sotl. we thus extend an invitation to you, as readers of this inaugural issue, to discuss these papers, to share your own ideas about and critiques of the findings, and to consider submitting your own contributions to this challenging ongoing conversation. journal of scholarship of teaching and learning: our initial public offering in the first chapter of scholarship reconsidered boyer describes the changes which occurred in the united states over the past two centuries which brought about the modern university. one of the most dramatic changes was the “conviction that knowledge was most attainable through research and experimentation.” (1990, p. 9) as we enter the 21st century the problems that need to be addressed are clearly different. in the late 1800's outreach became the mission of many universities through applied research in areas such as agriculture. in the mid-1900's universities served a nation at war and through scientific collaboration changed a country and the workforce. as we enter a new millennium the most pressing challenges to our universities may be occurring on our own campuses: how can we better serve the country by increasing the learning of our students? knowledge is still the focus of our studies but it is time to focus our research and experimentation on the teaching and learning which occurs within the university. the purpose of josotl is to advance the scholarship of teaching and learning by creating a vehicle for colleagues to make their studies of teaching and learning public, facilitating critical review and evaluation, and helping to build a knowledge base for exchange and use by members of a diverse scholarly community. in our first issue we have three papers to begin the discussion. each paper is from a different discipline and each raises a different question but all demonstrate an important reason why the scholarship of teaching is so important. scholarly teachers are reflective practitioners who have many questions they wish to answer about the practice of teaching. sometimes our questions are brought to the fore by outsiders (e.g., accrediting agencies); some questions arise out of institutional needs (e.g., student retention); other questions originate in our own curiosity or feelings of professional need (e.g., need to adapt new instructional strategies.) each of these questions can be addressed by the scholarship or teaching and learning although there are a variety of strategies which can be used to begin to answer the questions. the articles by kern and osborne we have described as research reports and the article by milner-bolotin and svinicki we are describing as classroom action research. as you read these articles consider the questions the article raises, including questions from your own teaching. and remember that josotl is very different from paper journals in that you can “talk” with the author and other colleagues who have read the article. we like to think of josotl as a cross between a paper journal and a conference. you don’t have to rush to front table to get a copy of the paper: just “click” on print the paper for your own copy from adobe acrobat reader. you can also get involved in a discussion with the author and other readers: just “click” the discuss the paper and you can read the remarks of other readers and post you own comments. we hope you will take full advantage of the opportunities available with our web journal. as we tackle the difficult questions of higher education in the 21st century we need a venue to communicate. as shulman says, “scholarship properly communicated and critiqued serves as the building blocks for knowledge growth in a field.” the research article by beth kern, “using role play simulation and hands-on models to enhance students’ learning fundamental accounting concepts”, is a controlled examination of a pedagogical approach to addressing a specific classroom challenge. building on the work of colleagues who have studied active learning strategies in higher education, dr. kern explores the effectiveness of active learning and the specific learning outcomes in her accounting class. many in the academic community are asked to make curricular changes in response to accreditation agencies and many programs make the “required” changes simply to meet the agency demand. dr. kern has taken the accounting education change commission’s call for active learning strategies in the accounting curriculum as an opportunity to study how active learning effects specific types of learning in her introductory students. she reports both significant and non-significant learning results which may assist instructors in choosing when to use active learning or more traditional approaches. the research article by randall osborne, “a model for student success: critical thinking and ‘at-risk’ students” addresses a growing institutional problem. as universities open their doors to students with less academic preparation they also invite increasing student failure in introductory classes which demand higher order thinking skills. professors often express displeasure at the number of students who are unable to think critically and it is not uncommon for faculty to believe these skills cannot be taught. universities who are concerned with student retention, as well as faculty who are troubled by high drop-out and failure rates, will be encouraged to read dr. osborne’s study. in a study of short-term and long-term gains dr. osborne reports on a program teaching critical thinking to his introductory psychology student. in their classroom action research study, “teaching physics of everyday life: project-based instruction and collaborative work in undergraduate course for nonscience majors”, milner-bolotin and svinicki explore a taxing problem for math and science teachers in higher education. they point out that scientific literacy and critical thinking is a requirement for every educated citizen in the 21st century but students who do not believe they can learn science and do not see the need to learn science make motivation a serious challenge in science classes for nonscience majors. their detailed account of how they implemented a project based instruction (pbi) into their physic class for nonscience majors is a pilot for future research with qualitative outcome measures and reflective commentary. their article is also a gateway to many web sites on innovative approaches to the teaching of science and mathematics. we at the journal of scholarship of teaching and learning are excited to begin the adventure. we know there are university teachers from every possible discipline, with various levels of expertise in teaching and research, who are interested in adding to and learning from the growing knowledge base of the scholarship of teaching and learning. we hope each of you will get involved on some level. as you read the articles in our first edition please consider the questions each article raises and post them on our threaded discussion. whether you are doing traditional research on teaching and learning or classroom action research please feel free to send your work to us for review. if you have ideas you would like to share about the scholarship of teaching and learning put them into an essay and send them in for review. and please feel free to contact us with your feedback about our electronic journal. at the american association for higher education conference this winter on “scholarship reconsidered reconsidered” the theme for lee shulman’s key note address, “teaching among the scholarships” was pass it on. this is also the guiding principle of josotl. we are here to help facilitate the discussion and we hope you will join us in passing on what you have learned, and are learning, about teaching and learning in higher education. references 1. boyer, e. (1990). scholarship reconsidered: priorities of the professoriate. san francisco, jossey-bass. 2. glassick, c.e., huber, m.t. and g.i. maeroff (1997). scholarship assessed: evaluation of the professoriate. san francisco: jossey-bass. 3. jencks, c. and d. riesman (1968). the academic revolution. new york: doubleday. 4. rice, g. (1996). making a place for the new american scholar. american association for higher education. 5. shulman, l. (1998). teacher portfolios: a theoretical activity. in n. lyons (ed.), with portfolio in hand: validating the new teacher professionalism (pp. 23-37). new york: teachers college press. 6. shulman, l. (2000, february). teaching among the scholarships. paper presented at the meeting of the american association for higher education on faculty roles and rewards, new orleans, la. 503 service temporarily unavailable 503 service temporarily unavailable nginx/1.14.1 v2n2breedlove journal of scholarship of teaching and learning, vol. 4, no. 2, september, 2004, pp. 33 – 42. collaborative testing and test anxiety william breedlove tracy burkett idee winfield1 abstract. prior research concluded that collaborative learning reduces test anxiety. examination of the evidence used in that research, however, calls into question those conclusions. the present study used an empirical measure of test anxiety and an experimental design to provide an improved estimate of the effect of collaboration in an evaluative context on test anxiety. the findings show no significant difference in test anxiety between students who collaborate on their exam and students who work alone. the ability to organize information is found to have a significant effect on test anxiety and that effect differs between collaborating students and those working alone. keywords: test anxiety, collaboration, testing, cooperative learning. i. introduction. among the many areas of research in teaching and learning, the areas of collaborative learning and test anxiety may be among the most studied. johnson, johnson, and stanne (2000) identify over 900 research studies, over a 100 year period, validating cooperative learning and 194 separate comparisons of specific collaborative learning methods. even more numerous are studies of test anxiety. pekrun, goetz, titz, and perry (2002) report over 1200 studies of test anxiety for the period 1974-2000 alone. surprisingly perhaps, given this volume of studies, only a very few studies have examined the relationship between collaboration in an evaluative context and test anxiety. even fewer attempt an empirical assessment of this association. rather, conclusions about a collaboration-test anxiety effect are based on student and teacher impressions. in this paper, we address this gap in the research literature. we examine the effect of collaboration in an evaluative situation on levels and changes in a quantitative measure of test anxiety among two groups of undergraduate college students. ii. literature review. a. collaborative learning. studies of collaborative learning have documented a range of beneficial outcomes across diverse populations and disciplines. researchers have documented learning gains among elementary school children (billington 1994; fuchs, fuchs, karns, hamlett, katzaroff, and dutka 1998), developmental students (ley, hodges and young 1995), and college students (clark 1994; giraud and enders 2000; gokhale 1995; grzelkowski 1987; guest and murphy 2000; hanshaw 1982; harris 1993; helmericks 1993; muir and tracy 1999; nowak, miller, and washburn 1996; 1 department of sociology and anthropology, college of charleston, charleston, sc 29424. direct correspondence to william breedlove, department of sociology and anthropology, college of charleston, charleston, sc 29424; breedlovew@cofc.edu. breedlove, w., burkett, t. and winfield, i. journal of scholarship of teaching and learning, vol. 4, no. 2., september, 2004. www.iupui.edu/~josotl 34 rau and heyl 1990; reinhart 1999; russo and warren 1999; sernau 1995). at the college level, collaborative learning studies have been conducted in courses in sociology (grzelkowski 1987; helmericks 1993; rau and heyl 1990; reinhart 1999; sernau 1995), psychology (guest and murphy 2000; ley et al 1995), business (nowak et al 1996), statistics (giraud and enders 2000), education (muir and tracy 1999), science (hanshaw 1982), industrial technology (gokhale 1995); and english composition (russo and warren 1999). among the learning outcomes identified by proponents of collaboration are increased complexity of thinking, increased motivation to learn, improved performance on oral, written, and multiple choice exams, and greater retention of information (gamson 1994; johnson, johnson, and stanne 2000). additionally, collaborative learning fosters cooperation and connections with others (muir and tracy 1999; rau and heyl 1990), develops skills critical workplace success such as team building and teamwork skills (nowak et al 1996; russo and warren 1999), humanizes the learning experiences (grzelkowski 1987), eliminates cheating (grzelkowski 1987; ley et al 1995), is associated with higher levels of student satisfaction (chickering and gamson 1991; fuchs et al 1998; giraud and enders 2000; sernau 1995; slavin 1980), and lowers test anxiety (grzelkowski 1987; hanshaw 1982; helmericks 1993; ley et al 1995; muir and tracy 1999; russo and warren 1999). the breadth and generalizability of collaborative learning effects across populations, disciplines, and methods of evaluation seems to make a very compelling case for adopting the collaborative learning format. closer scrutiny of the evidence, though, may temper enthusiasm for employing collaborative learning as a multi-purpose problem solver. consider the claim that collaborative learning reduces test anxiety. of the six studies we located that make such a claim, only hanshaw (1982) employs an instrument to measure test anxiety. the others base their conclusions about a collaboration-test anxiety effect on teachers’ and students’ impressions. while those impressions and conclusions may be valid, the absence of empirical evidence leaves them on less sure footing than empirical evidence would provide. further, the absence of a noncollaborative control group and a pre-collaboration test anxiety baseline against which test anxiety under collaboration can be compared make it all the more difficult to accept prior conclusions about the anxiety reducing effect of collaboration. b. test anxiety. test anxiety research has primarily focused on the association between test anxiety and academic achievement. reviews of that literature find strong consensus on the negative association between test anxiety and academic achievement (hembree 1988; seipp 1991). another large body of work has centered on the measurement of test anxiety. this research generally supports the idea that test anxiety is a two-dimensional construct with a cognitive and an emotional component. cognitive test anxiety refers to the inability to retrieve information in an evaluative setting. it is characterized by such conditions as task irrelevant thoughts, excessive fear of failure, worry about letting others down, and negative comparisons with others. emotional test anxiety refers to physiological reactions to evaluative situations. it includes reactions such as dizziness, nausea, and feelings of panic. meta-analyses and path analyses have concluded that cognitive test anxiety is the more important dimension of test anxiety for explaining difference in academic achievement. it is more strongly and more consistently associated with test performance (bandalos, yates and thorndike-christ 1995; williams 1991). breedlove, w., burkett, t. and winfield, i. journal of scholarship of teaching and learning, vol. 4, no. 2. 35 two models of the cognitive test anxiety-test performance association have received the most attention. the “interference” model argues that high levels of test anxiety inhibit the ability to recall learned information (sarason 1986; wine 1980). the problem is not one of learning, but one of interference with retrieval. wine (1980) writes that high test anxiety divides the students’ cognitive power between focusing on the task and attention to task-irrelevant thoughts. this both inhibits the power to recall and limits the ability to engage in higher order thinking. these factors may explain why students with high test anxiety generally do better on multiple choice exams than on essay exams where the former may require less recall. the “skills deficit” model claims that high test anxiety students have difficulty learning and organizing material, and that this results in poor test performance (birenbaum and pinku 1997; naveh-benjamin, mckeachie and lin 1987; tobias 1985). students with high test anxiety deal with anxiety through avoidance (appelhans and schmeck 2002). they minimize important differences and miss subtle cues about what is important to learn (cassady and johnson 2002). both avoidance of learning and minimization of attention lead to poor test performance. their test setting anxiety comes from their realization that they are not prepared. the problem is not one of recall, but rather, a lack of preparation due to poor study skills. they do poorly because there is little learned information to recall. rather than constituting alternative explanations, the interference and skills deficit models may be complementary (birenbaum and pinku 1997; tobias 1985). they suggest different types of students with different predicted performance levels. students with good study skills and good ability to organize information, along with low test anxiety should perform well since they have learned the test material and suffer from no inability to recall. other students are able to learn the material but suffer from retrieval problems in evaluative settings and when the task asks for more cognitive power than they have available. still others fail to learn and have difficulty organizing material so that they do poor regardless the test situation or test format. c. collaborative testing and test anxiety. collaborative testing seems to have implications for both interference and skills deficit on test performance. in the collaborative setting, students share their cognitive power and their archive of learned information. collaboration should boost the ability to retrieve both directly through lowered test anxiety and indirectly as students discuss, reflect on, debate questions and answers. collaboration should reduce anxiety due to skills deficit since students know they will have the knowledge of another student to aid them. on the other hand, collaboration may increase anxiety for these students if they become anxious over their lack of preparation being exposed to another student. we hypothesize that cognitive test anxiety will differ between students who collaborate on an exam and those who do not. collaborating students will have lower test anxiety than students working alone. we also hypothesize that the effect of the ability to organize information on test anxiety will differ between those who collaborate and those who do not. collaboration will reduce the effect of information organizing skill on test anxiety. iii. data and methods. our subjects were 131 undergraduate college students enrolled in four sections of an introductory sociology course. all participation was voluntary and only a few students chose not breedlove, w., burkett, t. and winfield, i. journal of scholarship of teaching and learning, vol. 4, no. 2., september, 2004. www.iupui.edu/~josotl 36 to participate. at the beginning of the course, students were told that they would have the opportunity to participate in a research study on learning that would be conducted in class. informed consent was gained from those who chose to participate. three sections of the course were designated as the experimental group where students would have the opportunity to work in same-sex, randomly assigned pairs on a multiple choice test. the fourth section served as the control group. basic demographic information on these groups is found in table 1. table 1. summary of student and group characteristics. all experimental group control group t-statistica sex female male 79 52 54 37 25 15 0.131 minority minority non-minority 10 121 3 88 7 33 -2.12* class rank freshman sophomore junior senior 70 45 10 6 52 29 6 4 18 16 4 2 -1.06 io test 1 score high moderate low 51 66 14 33 47 11 18 19 3 -1.54 io test 2 score high moderate low 53 51 27 36 40 15 17 11 12 0.734 minority refers to non-white students. io test 1 and io test 2 refer to self-reported ability to organize information a. t-statistics for the difference between the experimental and control group. *iindicates a significant difference at the 5% level. we divided the semester into three approximately equal sections and covered roughly the same amount of material in each section. a multiple choice test was administered at the conclusion of each section. our present interest is in the changes between tests 1 and 2. all students took the first test individually. students in the experimental group worked in pairs to completed test 2 while students in the control group continued to work alone. on the day of the test, and immediately prior to distributing the test forms, we administered a likert-type test anxiety scale. scale items are similar to those presented by cassady and johnson (2002). we were concerned that students have enough time to finish the test, especially since we anticipated that collaborating would increase completion time, so we used a smaller number of scale items than might otherwise be used.2 the final seven item scale has an alpha > .84. while the scale includes relatively few items, and that may be a cause of 2 test anxiety scales vary in number of scale items. pekrun et al (2002) report test anxiety scales ranging from 9 to 31 items. breedlove, w., burkett, t. and winfield, i. journal of scholarship of teaching and learning, vol. 4, no. 2. 37 concern, it captures variation sufficient for analysis and for distinguishing different levels of test anxiety. on our scale, the higher the score, the higher the level of test anxiety. we also collected self-reported information on subject’s sex (female = 1 male = 0), minority status (minority = 1 non-minority = 0), and class rank (freshmen = 1 sophomore = 2 junior = 3 senior = 4) to assess within group differences. we would like to have collected information on grade point average (gpa), in order to control for ability, but many of our students were entering freshmen who had not yet earned their first gpa. lastly, we measure a student’s ability to organization information (io) as their self reported agreement or disagreement to the two questions “i frequently feel that i have studied the ‘wrong’ things for the test” and “the harder i work at taking a test or studying for one, the more confused i get”. the data were reverse coded and summed so that the higher the sum of their scores on these questions, the higher their ability to organize information. iv. analysis and discussion. the summary data reported in table 1 show no significant differences between our experimental and control groups except for minority status. there was a significantly greater percentage of minorities in the control group. this difference should be kept in mind when evaluating any other group differences. most of our students were female, white, and predominantly either freshmen or sophomores. although there were no significant differences in the level of io between our groups, it is interesting to note that in both groups students are more likely to rate their information organization skills as moderate or high than as low. as shown in table 2, we find no significant difference in test anxiety between our groups at either test 1 or test 2. there is also no significant difference in the change in anxiety level between test 1 and test 2. our results do not support the argument that collaborative testing reduces test anxiety. rather than reduce test anxiety, anxiety is apparently higher at the time of the second test than the first, though not significantly so.3 while these tests fail to show a significant effect of collaboration on test anxiety, closer examination of the distribution of test anxiety change shows an interesting outcome. table 2. test anxiety levels and change. all experimental group control group t-statistica test 1: anxiety average 18.10 18.04 17.78 0.770 test 1: anxiety average 18.20 18.13 18.20 -.100 test 1: anxiety average 0.09 0.18 -0.01 0.632 a. t-value for the difference between the mean scores of the experimental and control groups. the evidence presented in table 3 shows that while most students in both groups experienced either an increase or a decrease in test anxiety between test 1 and test 2, students in the experimental group were less likely than those in the control group to experience an increase in test anxiety. whereas test anxiety increased 36 percent of experimental students, it increased for almost 43 percent of control group students. collaborative testing may not affect the overall amount of test anxiety, but it appears to affect the distribution of changes in anxiety. 3 t values are not reported. breedlove, w., burkett, t. and winfield, i. journal of scholarship of teaching and learning, vol. 4, no. 2., september, 2004. www.iupui.edu/~josotl 38 table 3. distribution of test anxiety score changes by group. experimental group control group cumulative changea percent percent -6 -5 3.3% 3.3% -4 4.4 7.7 -3 2.2 9.9 -2 11.0 20.9 -1 15.4 36.3 0 17.6 53.8 +1 17.6 71.4 +2 15.4 86.8 +3 6.6 93.4 +4 5.5 98.9 +5 1.1 100.0% cumulative change percent percent -6 2.5% 2.5% -5 2.5 5.0 -4 2.5 7.5 -3 10.0 17.5 -2 10.0 27.5 -1 15.0 42.5 0 20.0 62.5 +1 7.5 70.0 +2 12.5 82.5 +3 12.5 95.0 +4 2.5 97.5 +5 2.5 100.0% a calculated as anxiety at test 1 minus anxiety at test 2. negative values indicate an increase in anxiety from test 1 to test 2. the top half of table 4 presents the results of regressing test 1 anxiety level on four predictors for both groups. the bottom half of that table presents the corresponding results for test 2 anxiety level. in all four equations, io has a significant negative effect on test anxiety. it also has the largest effect on test anxiety except on test 1 for the control group where class rank has the largest effect. these results suggest that information organizing skills are important for improved test performance not only because they lead to better preparation for a test, but also indirectly because they reduce the cognitive test anxiety that others have shown to be detrimental to test performance. the effect for class rank is not consistently significant, but the effect in all equations is in the direction one might hypothesize. given their greater experience with examinations, their greater likelihood of having learned to cope with test anxiety, the greater probability that they have developed study skills, it is probably not surprising that more senior students have lower levels of test anxiety compared to newer students. the effect of minority status is interesting. again, although the effect is generally not significant, the sign of the effect is stable across equations. minority students express lower levels of test anxiety than majority students. this is perhaps surprising given the generally lower standardized test scores of minority students. to the extent that those scores reflect ability, one would expect their test anxiety scores to be higher since other research shows that high test anxiety is correlated with low ability. alternatively, minority students may receive more emotional support from family versus peers. this mode of support has been shown to significantly lower test anxiety (orpen 1996). whatever the explanation, it appears that collaboration may boost the effect of minority status on test anxiety, net of the other independent variables. additional research is needed to verify and explain this association. a final contrast of interest is the change in the size of the effect of io for the experimental group. while the change in io for the control group is minimal, there is a substantial decrease in the experimental group. collaboration may be responsible for this reduction. students with poor information organizing skills, who might otherwise be anxious breedlove, w., burkett, t. and winfield, i. journal of scholarship of teaching and learning, vol. 4, no. 2. 39 about the test, may feel less cognitive test anxiety knowing that they will have the knowledge of another student to call upon. further study on the reaction of students with poor io skills to collaboration could help us better understand how collaborative testing affects test anxiety. table 4: regression results: test anxiety on subject characteristics. experimental group control group test 1 anxiety sex class rank minority io test 1 adj. r2 +0.696 -0.279 -3.000 -2.120** 0.150*** +1.090 -2.090** -1.590 -2.570** 0.372*** test 2 anxiety sex class rank minority io test 2 adj. r2 +0.995 -0.321 -3.920* -1.610** 0.148*** -0.120 -1.855** -1.660 -2.760** 0.442*** the experimental group took test 2 in same-sex pairs. **significant at the 5% level. ***significant at the 1% level. v. conclusion. collaborative learning is one of the most commonly used and studied teaching techniques with researchers finding collaboration leading to increased complexity of thinking, increased motivation to learn, improved performance on oral, written, and multiple choice exams, and greater retention of information (gamson 1994; johnson, johnson, and stanne 2000). collaborative learning enhances behavioral outcomes such as fostering cooperation and connections with others (muir and tracy 1999; rau and heyl 1990), team building and teamwork skills (nowak et al 1996; russo and warren 1999), and eliminates cheating (grzelkowski 1987; ley et al 1995). in the affective realm, collaboration creates a more humane learning environment (grzelkowski 1987), is associated with higher levels of student satisfaction (chickering and gamson 1991; fuchs et al 1998; giraud and enders 2000; sernau 1995; slavin 1980), and lowers test anxiety (grzelkowski 1987; hanshaw 1982; helmericks 1993; ley et al 1995; muir and tracy 1999; russo and warren 1999). our review of the evidence for one of these outcomes, namely test anxiety, finds that conclusions about the effect of collaboration may be premature. of the collaboration-test anxiety breedlove, w., burkett, t. and winfield, i. journal of scholarship of teaching and learning, vol. 4, no. 2., september, 2004. www.iupui.edu/~josotl 40 studies we found, the evidence on which those conclusions were based is generally weak for reasons we outlined above. while no one study is adequate for drawing a final conclusion, we believe the present study employs a more rigorous design than earlier studies and begins to bring us closer to that conclusion. we find no significant difference in test anxiety between students who collaborate and those who do not. our findings are based on comparisons between two tests when students did not also engage in prior collaborative learning or get to know their test partners prior to the collaborative test. additional research comparing changes in anxiety across multiple exams, not just between two, that examines alternative testing formats, or that examines difference in test anxiety when students engage in collaborative learning in addition to collaborative testing may lead to different conclusions. interestingly, collaboration may have an effect on the distribution of test anxiety changes that is not apparent when comparing group means. understanding the distribution of change may help identify which students most benefit from collaborative testing. this is another area where additional research is needed. we also find that information organizing skills are important for reducing test anxiety and that the effect of those skills on anxiety depends on whether students collaborate or work alone on their test. students with poor information organizing skills, who might otherwise be anxious about the test, may feel less cognitive test anxiety knowing that they will have the knowledge of another student to call upon. further study of the reaction of students with poor io skills to collaboration could help us better understand how collaborative testing affects test anxiety. references appelhans, b. and r. schmeck. 2002. “learning styles and approach versus avoidant coping during academic exam performance.” college student journal. 36:157-161. bandalos, d., k. yates and t. thorndike-christ. 1995. “effects of math self-concept, perceived self-efficacy, and attributions of failure and success on test anxiety.” journal of educational psychology. 87:611-623. billington, r. 1994. “effects of collaborative test taking on retention in eight third-grade mathematics classes.” the elementary school journal. 95:23-32. birenbaum, m. and p. pinku. 1997. “effects of test anxiety, information organization, and testing situation on performance on two test formats.” contemporary educational psychology. 22:23-38. cassady, j. and r. johnson. 2002. “cognitive test anxiety and academic performance.” contemporary educational psychology. 27:270-295. chickering, a. and z. gamson. 1991. “applying the seven principles for good practice in undergraduate education.” in a. chickering and z. gamson, eds. new directions in teaching and learning. san francisco: jossey bass. clark, d. 1994. “high-risk teaching.” the teaching professor. 8:1-2. breedlove, w., burkett, t. and winfield, i. journal of scholarship of teaching and learning, vol. 4, no. 2. 41 fuchs, l., d. fuchs, k. karns, c. hamlett, m. katzaroff, and s. dutka. 1998. “comparisons among individual and cooperative performance assessments and other measures of mathematics competence.” the elementary school journal. 1998:23-52. gamson, z. 1994. “collaborative learning comes of age.” change. 26:44-50. giraud, g. and c. enders. 2000. “the effects of repeated cooperative testing in an introductory statistics course.” paper presented at the annual meeting of the american educational research association. gokhale, a. 1995. “collaborative learning enhances critical thinking”. journal of technology education. 7:1. grzelkowski, k. 1987. “a journey toward humanistic testing.” teaching sociology. 15:27-32. guest, k. and d. murphy. 2000. “in support of memory retention: a cooperative oral final exam.” education. 121:350-354. hanshaw, l. 1982. “test anxiety, self-concept, and the test performance of students paired for the same students working alone.” science education. 66:15-24. harris, t. 1993. “team testing produces dramatic results.” teaching for success. 5:5-6. helmericks, s. 1993. “collaborative testing in social statistics: toward gemeinstat.” teaching sociology. 21:287-297. hembree, r. 1988. “correlates, causes, and treatment of test anxiety.” review of educational research. 58:47-77. johnson, d., r. johnson and m. stanne. 2000. “cooperative learning methods: a metaanalysis.” http://www.clcrc.com/pages/cl-methods.html. ley, k., r. hodges and d. young. 1995. “partner testing.” research and teaching in developmental education. 12:23-30. muir, s. and d. tracy. 1999. “collaborative essay testing.” college teaching. 47:33-36. naveh-benjamin, m., w. mckeachie and y. lin. 1987. “two types of test-anxious students: support for an information processing model. journal of educational psychology. 79:131-136. nowak, l., s. miller and j. washburn. 1996. “team testing increases performance.” journal of education for business. 71:253-256. orpen, c. 1996. “the interactive effects of social support and test anxiety on student academic performance.” education. 116:464-466. pekrun, r., t. goetz and w. titz. 2002. “academic emotions in students’ self-regulated breedlove, w., burkett, t. and winfield, i. journal of scholarship of teaching and learning, vol. 4, no. 2., september, 2004. www.iupui.edu/~josotl 42 learning and achievement: a program of qualitative and quantitative research.” educational psychologist. 37:91-105. rau, w. and b. heyl. 1990. “humanizing the college classroom: collaborative learning and social organization among students.” teaching sociology. 18:141-155. rinehart, j. 1999. “turning theory into theorizing: collaborative learning in a sociology theory course.” teaching sociology. 27:216-232. russo, a., and s. warren. 1999. “collaborative test taking”. college teaching. 47:18-20. sarason, i. 1986. “stress, anxiety, and cognitive interference: reactions to tests.” journal of personality and social psychology. 46:929-938. seipp, b. 1991. “anxiety and academic performance: a meta-analysis of findings. anxiety research. 4:27-41. sernau, s. 1995. “using a collaborative problem-solving approach in teaching social stratification.” teaching sociology. 23:364-373. slavin, r. 1980. “cooperative learning.” review of educational research. 50:315-342. tobias, s. 1985. “test anxiety: interference, defective skills, and cognitive capacity. educational psychologist. 20:135-142. williams, j. 1991. “modeling test anxiety, self concept, and high school student’s academic achievement.” journal of research and development in education. 25:51-57. wine, j. 1980. “cognitive-attentional theory of test anxiety.” in i. sarason, ed. test anxiety: theory, research, and applications. pp. 349-385. hillsdale, nj: erlbaum. v4n1alessio journal of scholarship of teaching and learning, vol. 4., n. 1, may, 2004, pp. 23-34. student perceptions about and performance in problem-based learning helaine alessio1 abstract. many graduate and professional programs include problem based learning (pbl) as a mainstay in their curricula. for many undergraduate students, this is a change from a teacher-centered to a student-centered learning method. this study was undertaken to learn about perceptions and test performances of college students (n=116) enrolled in liberal education classes when pbl is used vs. traditional teaching methods. results indicated students perceived traditional teacher-centered learning more favorably than studentcentered pbl. nevertheless, test scores were similar. negative student perceptions about learning in pbl classes did not support either teacher observations of learning activity in the classroom or compromised test performances. keywords: problem based learning, teacher centered learning, student centered learning i. introduction. problem based learning (pbl) was initially implemented as an alternative higher education teaching method in the 1970’s at mcmaster university’s medical school (boud & feletti, 1997). the primary reason for implementing pbl into the curriculum was to better prepare students for real-world problem solving. medical professionals, when presented with a patient, do not always have all the information necessary to provide appropriate diagnosis and treatment. in particular, in emergency medicine, patients may not be able to communicate all pertinent information. the ability of professionals to distinguish what is known about a patient’s condition from what needs to be known, and how to go about gathering information that will lead to addressing the patient’s condition and solving a health-related problem, will determine the potential for a successful outcome. this type of learning process is not unique to medical school courses and scenarios. similar real-world problems are often presented in the field of accounting, where, for example, an accountant is presented with a stack of paperwork and is asked to prepare a balance sheet or journalize a transaction. in addition to what is evident before her or him, the accountant will have to sort out what is known, what needs to be known, and how to go about gathering necessary information to solve the problem. pbl has been successfully implemented in medical and professional education schools around the world (alleyne et al, 2002; sundglad et al., 2002; quinlan, 2000; albanese & mitchell, 1993; vernon & blake, 1993). pbl is also emerging in undergraduate curricula as many universities and colleges undergo curriculum reform and are exploring new ways to effectively engage and teach students (seaberry, 2002; barr & tagg, 1995). there is growing evidence that real-world type of questions and learning activities that are student-centered may be more valuable than traditional teacher-centered lectures in which most of the pertinent 1 helaine alessio is professor of exercise science, miami university, oxford, ohio. the author acknowledges statistical assistance from richard mcclure, and support from phil cottell, jr. larry greeson, greg wentzell, and milt cox. the author is grateful for funding from miami university’s center for the enhancement of learning and teaching. alessio, h. journal of scholarship of teaching and learning, vol. 4., n. 1, may, 2004. 24 information is presented by the teacher, followed by a case study or assignment (martin et al., 1998; norman & smidt, 1992). a goal of pbl is to prepare students for life-long learning by engaging them in active learning in which the students are responsible for discovering facts and uncovering key concepts. this approach contrasts with a traditional teacher-centered approach where key facts and concepts are presented to the students. three key features of pbl include: 1) learning in context, where real life problems are presented; 2) elaboration of knowledge through social interaction, where students work together in small groups; and 3) meta-cognitive reasoning and self-directed learning, where independent thinking and life long learning is encouraged (dahlgren & dahlgren, 2002). in pbl, students are presented with a realistic problem without prior traditional lectures or presentations (duch, groh, & allen, 2001). in the process of solving the problem, students develop knowledge of theory, practice, facts, concepts, and appropriate inquiry strategies related to the initial problem. success or failure of pbl depends upon students’ taking responsibility for their own learning (quinlan, 2000). solomon (2001) reported that her masters level physiotherapy students valued group process and work, as indicated by the following comment: “i learned that each member of the team has a role to play, not simply individuals working independently in a team setting, but that trust, patience, and respect for each member only makes the group stronger. that is what i learned—that i have a role to play to set up this environment.” nevertheless, pbl has been known to create tension in undergraduate students in studies where they reported dissatisfaction and fear. in particular, the transition from a traditional student to being a pbl student was described by biley (1999) as being associated with feelings of frustration and uncertainty. it is speculated that once students undertake the transition, then negative perceptions about pbl will be reduced and benefits of pbl will be realized. outcomes in the form of grades are important to the student in their perception of pbl. undergraduate nursing students acknowledged benefits of pbl such as the ability to “find things out that we had no previous knowledge of, to go back at the end of the program and work at it, or change direction and follow our own interests and educational needs.” (biley, 1999). although pbl has been implemented and assessed in many graduate and professional school programs, few studies have focused on the impact of pbl in liberal education graduate and undergraduate programs. many liberal education courses consist of primarily traditional teaching methods. traditional teaching methods have been described as pedagogues where the teacher transmits knowledge in a securely anchored way to students. primarily didactic teaching methods contribute to the student’s “comfort zone” (margetson, 1996). when roles are changed, for example, when the student is required to develop one’s own plan of direction, the comfort zone changes, and tension develops. a key intention of pbl is that such tension may lead to learning material in a different way, creating opportunities for self-directed, deep learning (dunkbase & penick, 1990). the initial response to pbl seems to relate to the lack of a “comfort zone” where traditional roles are supposed to be played out with the instructor responsible for the teaching and the student processing the information for learning. nardi & kremer (2003) reported a common occurrence found in pbl classesmost students reported discomfort when attempting to solve ambiguous problems in health care. a “comfort zone” is not limited to students as teachers tackle issues such as the extent to which a teacher can “let go” as a facilitator, deal with tensions that arise in student groups, manage time effectively, and balance the need to deliver problem solutions in real time while still meeting the learning needs of students (conway & little, 2000). alessio, h. journal of scholarship of teaching and learning, vol. 4., n. 1, may, 2004. 25 several studies have reported student and faculty concerns about pbl implemented in specific coursework and curricula in both pre-professional and professional schools (alleyne et al., 2002; sunblad et al., 2002). nardi and kremer (2003) used a naturalistic inquiry method (a combination of observation and questions) to learn about the degree of mindfulness undergraduate nursing students displayed about their own academic growth and the ability to link their learning activities to the degree of personal learning in the classroom. they were interested in determining how well students perceived they learned using pbl as well as how well test scores reflected learning. one factor leading up to this study is the expectations by many professional schools, including medicine, nursing, physical and occupational therapy, that students have experience in pbl. in my role as an academic advisor, i have seen an increasing number of inquiries on graduate and professional school applications about students’ experience with pbl. in my role as a teacher in higher education, i feel obliged to prepare students not only with content but also with appropriate tools needed to succeed in the process of learning. due in large part to the inquiries by graduate and professional schools on pbl experiences of undergraduates, i introduced pbl into my undergraduate and graduate courses several years ago and noted whether the academic performance differed from traditional teaching methods. i also noted perceptions of the students, in part because of a strong impression of frustration that pervaded the classroom. i felt a strange sense of incongruity between what students wanted and what students needed. the purpose of this study was to learn about the perceptions and test performances of college students enrolled in liberal education classes in which both pbl and traditional teaching methods are used. it was hypothesized that undergraduate and graduate students enrolled in a liberal education college would experience similar tensions as pre-professional students related to pbl teaching methods. it was further hypothesized that test performance would be inferior in students enrolled in courses which included pbl compared with students enrolled in classes that utilized mainly traditional teaching methods. the hypothesis that students in pbl would do worse on tests than students in traditional lecture was partly formed due to the high level of frustration among students that i perceived and the novelty of the pbl, student-centered approach to virtually all students. in this study, both quantitative and qualitative data are presented, as well as a naturalistic inquiry method by the instructor, that provide insight into undergraduate and graduate student learning, perceptions, and test performance associated with pbl in a liberal arts college setting. ii. method. this study was conducted at a midsized college, with a 15,000 student enrollment, located in a small town in a midwestern state in the usa. it is considered to be a liberal arts college and its mission has an undergraduate focus with a few graduate programs having national recognition. it has a selective admission policy. the community served by the school is mainly euro-caucasian with approximately 10% minorities enrolled. most of the undergraduate students reside on campus and range in age 18-24 years old. a majority of graduate students enroll as full time students and are supported by assistantships on campus. participants in these studies had similar characteristics as the typical undergraduate and graduate student enrolled in the college, with an average gpa range of 2.8-3.6 for undergraduates and 3.0-3.7 for graduates. all students were volunteers who agreed to allow their test scores to be used in calculations and comparisons. they also agreed to allow their anonymous feedback to be qualitatively analyzed. all students completed informed consent forms and complied with the rules and regulations of miami university’s human subjects internal research board. alessio, h. journal of scholarship of teaching and learning, vol. 4., n. 1, may, 2004. 26 two separate studies, one involving undergraduate and the other, graduate students, were conducted during the academic year. the same instructor taught all class sections, both graduate and undergraduate. a total of 93 undergraduate students, 64% female and 36% male, enrolled in three sections of exercise physiology, a senior level class taken by students as either a required or elective course. the age range was 21-28 years. not one student had experience with pbl methods in previous coursework. thirty-one students were enrolled in a course that included pbl. two other sections of the same course included mainly lecture presentations and enrolled 37 and 25 students. learning objectives included understanding and describing basic anatomy and physiology of muscle, muscle development, muscle atrophy, and theories related to muscle movement and muscle growth. traditional lectures included slide presentations and case studies. pbl activities included problems given to students prior to the presentation of information required to address the problems. the instructor consulted a pbl text book (duch et al, 2001) and web sites for examples of questions and brought to the classroom resources in the form of extra texts, journals, and video clips. pbl activities and questions were undertaken in small groups. small group formation followed a modification of models presented by biley (1999), millis and cottell (1998), and van der vleuten and weigne (1990), with four students, each having a specific role: leader, devil’s advocate, recorder, and reporter. roles were changed throughout the course so that every student was responsible for each role at some point during the course. organization of pbl was loosely structured around biley’s (1999) eight stages as shown in table 1. and finally, and importantly, motivational aspects of the problem, as described by chapman (2000) were taken into account: familiarity, relevance, dramatic appeal, significance, authenticity, and potential for group collaboration. table 1. biley’s (1999) pbl process. step 1. interpretation of the scenario, concept, and term clarification step 2. brainstorming step 3. organize the outcome of the brainstorm step 4. define problems and issues step 5. establish learning needs step 6. collect knowledge step 7 explore newly acquired knowledge in relation to established problems and i issues step 8 apply to scenario these stages were described to students and then reduced to three major questions that challenged students as they attempted, with their fellow group members, to solve the problem: 1. what do we know? 2. what do we need to know? 3. how do we proceed in finding out what we need to know? twenty-three graduate students, 83% female and 17% male, enrolled in two classes, and participated in a separate study. the age range was 22-42 years. one class of 15 participated in pbl and a second class of 8 students participated in a traditional lecture format. the same material on cardiovascular health and age-related disease was presented using either pbl or traditional lecture presentations. the class was a graduate level course titled: exercise, age, and health. this interdisciplinary class addresses physiological, social, and psychological forces that impact health and aging. learning objectives included understanding and describing a healthy and diseased cardiovascular system, studying causes for disease, and distinguishing between aging and disease processes that affect cardiovascular structure and function. alessio, h. journal of scholarship of teaching and learning, vol. 4., n. 1, may, 2004. 27 the same examinations were given to graduate students both in the pbl class and in the traditional lecture format class. similarly, undergraduates took the same examination. a one-way analysis of variance (anova) was used to compare undergraduate mean grades between the two teaching methods: pbl and traditional, followed by bonferroni post-hoc analyses tests, if significant differences were detected. a separate one-way anova was also used to compare graduate mean grades between pbl and traditional teaching methods. the chi square test for independence was used to compare grade distribution among the different classes. in both types of analyses, anova and chi square, the level of significance was set at 0.05. in addition to these two quantitative comparisons, qualitative information was recorded and categorized. the following open-ended question was posed to all students in the pbl class: “please comment about the teaching method-pbl-used in this class.” a similar question was posed to all students in the traditional lecture/presentation class: “please comment about the teaching methods used in this class”. subjective feedback was provided by students in all courses and analyzed by determining core categories, which captured major themes of student comments. the instructor also recorded perceptions of students as they worked together in both the traditional and pbl classes in a journal following each class. these perceptions included notes on student engagement and assessment of focus and efficacy of small group discussions in the different classes. iii. results. a. undergraduate course comparisons. the mean grade for undergraduate students in the pbl class was 82.3 + 1.3%. mean grades for the two traditional classes were 82.6 + 1.3 and 80.7 + 1.7%. grade distribution in the exams was similar and is shown in figure 1. anova indicated that there were no differences among the mean test scores for students in pbl versus traditional lecture classes (f2,89=0.53, p=0.59). chi square test for independence indicated a critical chi-square value=9.48, but the calculated chi-square value=2.90, therefore, we could not reject the null hypothesis-that grade distribution was similar among the three classes. a cursory view of the data indicates a difference in grade distribution between the two traditional lecture classes when comparing the percentage of students scoring in the 60’s, 70’s, and 80’s on the exam. upon further analysis of the c grade, when combining cells into the grade of c and non-c scores, once again, the calculated chi-square value of 2.59 did not reach the critical chi-square value of 5.99. therefore, at the 0.05 level of significance, we could not reject the null hypothesis and thus had to accept that the c grade distribution was similar among the three undergraduate classes. furthermore, in both traditional lecture formats, 80% vs. 76% of alessio, h. journal of scholarship of teaching and learning, vol. 4., n. 1, may, 2004. 28 0 10 20 30 40 50 60 70 60's 70's 80's 90's grades p e rc e n ta g e o f s tu d e n ts pbl traditional a traditional b figure 1. grade distributions in the undergraduate classes. students scored in the 70’s and 80’s grade range. that average compared closely to 77% of students in pbl who scored in the 70’s and 80’s grade range. subjective feedback from undergraduate students in the class in which pbl was used, included the following categorized by whether the feedback supported pbl or not. only two students provided positive feedback about pbl. most feedback on pbl was categorized as negative. the following include samples of undergraduate student feedback. undergraduate student feedback that was supportive of pbl: “i like interacting with other students in small groups.” “reading journal articles helped me learn more about the material in the book.” these two statements were distinctly categorized as “engaging with other learners” and “independent search for knowledge”. they represent two important learning processes associated with pbl. despite these positive signs, most undergraduate feedback was negative and several categories: partiality towards didactic or directed learning, perception of reduced learning efficiency, and feelings of uncertainty, emerged. b. graduate course comparisons. mean grades for graduate students in pbl and lecture format-class were 82.0 + 1.4 % and 82.5 + 2.2 %, respectively. grade distribution of the exams was similar between the two classes, with 26% versus 25% scoring in the 70’s, 60% versus 75% scoring in the 80’s, and 13% versus 0% scoring in the 90’s in pbl and traditional lecture, respectively. anova of the graduate student grades showed no significant difference between the pbl and traditional lecture classes. graduate level classes were smaller than undergraduate level classes. the traditional lecture class had no students earn a grade in the 90’s (a range), six alessio, h. journal of scholarship of teaching and learning, vol. 4., n. 1, may, 2004. 29 students earn a grade in the 80’s (b range) and 2 students earn a grade in the 70’s (c range) compared to pbl where two students scored in the 90’s nine scored in the 80’s, and four scored in the 70’s. due to the low number of students, a chi square analysis could not be performed. grade distributions in the graduate level classes are shown in figure 2. 0 10 20 30 40 50 60 70 80 70's 80's 90's grades p e rc e n ta g e o f s tu d e n ts pbl traditional a figure 2. grade distributions in the graduate classes. negative feedback toward pbl was given by all but two graduate students. graduate students specified no negative feedback toward traditional lecture-based learning. graduate student feedback that was not supportive of pbl was sorted into the same three categories as undergraduate negative feedback: partiality towards didactic or directed learning, perception of reduced learning efficiency, and feelings of uncertainty. in the present study, both undergraduate and graduate students reported a remarkably similar undercurrent of dissatisfaction, frustration, and uncertainty that biley’s (1999) undergraduate nursing students and seaberry’s undergraduate chemistry students reported (table 2). iv. discussion. as an academic advisor who completes several dozen recommendations for professional and graduate school-bound students, i have noted the increasing number of programs that request information about student experiences in pbl. many professional and graduate programs are grounded in pbl curricula. there are many reasons for this. advocates of pbl believe that students learn better when they actively engage in solving real world problems instead of passively receiving information from the teacher to solve a problem. deep learning can occur when students work together in small groups and in self-directed learning, where independent thinking is encouraged (dahlgren & dahlgren, 2002). studies on teaching and learning effectiveness have yielded mixed results comparing pbl with traditional lecture (martin et al., 1998; vernon & blake, 1993). despite the equivocal results, pbl, which has its roots in medical education in the 1970’s, appears to be gaining favor alessio, h. journal of scholarship of teaching and learning, vol. 4., n. 1, may, 2004. 30 with many other types of professional schools including nursing, law, social work, pharmacy, clinical psychology, computer engineering, and physical and occupational therapy (dahlgren & dahlgren, 2002). my first attempts in trying pbl in both undergraduate and graduate courses were prompted by the increased number of professional programs that currently implement pbl throughout the curricula and their desire for students with experience in pbl. unfortunately, my first experiences in using pbl included perceptions of student frustration and vexation that i had not encountered before with traditional teaching. despite student objections about the lack of learning they felt was occurring with pbl methods, i perceived that student discussions and knowledge acquisition was, in fact, happening. nevertheless, i could not dismiss negative reactions and perceptions by students about their learning and so i decided to learn more about table 2. similar student comments recorded by biley (1999), seaberry (2002), and alessio. comments by biley’s students (1999) comments by seaberry’s students (2002) comments by alessio’s students “it is so time consuming, there is such a large volume of knowledge to learn.” “it too way too much time.” “i think it would be effective if the teacher would go over essential information.” “i can see myself spending days in the library under piles of books…and having great difficulty getting any other work done.” “this method took around-about way to learn the concepts” “too much useless information.” “i like to be told things.” “you (the teachers) need to involve more of the class lecture material into the case work.” “i prefer getting the information straight from the professor than from working in small groups and learning from students.” “lunging into completely unstructured sessions was frightening “it was hard to understand how it was supposed to work.” i did not like the pbl exercise. i felt lost and did not feel that i learned as well as if i were taught by lecture. “there is such a lack of time and a lot of pressure to acquire knowledge.” “i need more explanation about how to work with the case material and the group itself.” students seem to frantically compete for the right answer, talking over and around us less knowledgeable folks.” more about undergraduate and graduate student perceptions of and performance in classes in which pbl methods were used. all students were enrolled in college programs in which traditional teaching methods predominated. it was hypothesized that these students would experience tensions related to the transition from mainly didactic (teacher centered) to pbl (student centered) methods. due to the novelty of pbl for virtually all students, it was also hypothesized that test performance may be inferior in students of pbl compared with traditional teaching methods. this hypothesis was not supported however, as test performance was similar between pbl and traditional teaching. alessio, h. journal of scholarship of teaching and learning, vol. 4., n. 1, may, 2004. 31 two main positive themes about pbl emerged in this study. 1) engaging with other learners, and 2) independent work. these two themes represent beneficial perspectives of pbl, but were clearly in the minority with only two students reporting positive comments related to working both independently and with others. the majority of feedback was negative towards pbl, and was categorized in three ways: 1. preference for directive or didactic learning, 2. perception of reduced learning efficiency, and 3. feelings of uncertainty. students stated that they preferred didactic learning, possibly because that was the main teaching method they had experienced prior to the present class. students also contended that pbl took too much time compared to traditional lectures. quinlan (2000) describes a literature database designed to address student concerns about making choices concerning resources and time management in order to guide veterinarian students towards the core material for learning. she found that the majority of veterinarian students preferred using their own database for gathering information, so in fact self-directed learning prevailed in these students. table 2 includes student feedback from two previous investigations and the current one on the use of pbl in the classroom. the critical statements were surprisingly similar in all three studies with time, information, learning style, and frustration emerging as significant issues for all students. students in the present study complained that pbl left too much uncertainty to sort out pertinent information. these negative perceptions about pbl by students in this study did not differ from perceptions of pbl by students in other studies, where feelings of insecurity (des marchais et al., 1992), uncertainty (west, 1998), frustration, fear (biley, 1999), reduced learning efficiency and confusion (walton & matthews, 1989) have been reported. concerns expressed by students in the present study were not unique from previous studies, especially concerns related to grade-anxiety (boud & feletti, 1997). in this study, mean grades did not differ in either undergraduate or graduate courses. in the undergraduate classes, the mean grades for traditional lecture classes (82.6 + 1.6 and 80.8 % + 1.4) were indistinguishable from the mean for pbl (82.5 % + 1.3). a similar result occurred comparing graduate student grades in traditional lecture (mean = 82.5%) and pbl (82.0%). an analysis of grade distribution also showed that a similar percentage of students scored in the a, b, and c range regardless of the teaching method. these results indicate that neither pbl nor traditional lecture was superior in preparing students for testing. an important finding is that learning was not compromised by either teaching method. in contrast to undergraduate and graduate students’ perceptions that learning was somehow compromised via pbl activities, my perception of the classes was that the students were, in fact, engaging in active learning. as i listened to group discussions, i noted that students were asking each other questions that reflected deep learning and abstract thinking. group interactions may have been facilitated in part by the “devil’s advocate” role of one member in each group, whose job it was to raise critical questions when statements by other group members were made. questions such as “are you sure that information is accurate or up-to-date? is there any evidence that would oppose that conclusion? what if…?” seaberry (2002) reported similar teacher observations of face-toface interactions and active discussions among the pbl student groups, during class time. i also observed students using resources other than the required text, to search for answers and information designed to fill in gaps and confirm statements made by group members. as i saw it, students in the small pbl groups engaged actively in the learning process and formed a type of learning community, with each making a meaningful contribution, as they tackled the problems. student perceptions of their learning did not correspond with my alessio, h. journal of scholarship of teaching and learning, vol. 4., n. 1, may, 2004. 32 observations of in-class learning activities, nor did they predict test performance in either undergraduate or graduate students. v. conclusions. a unique aspect of this study was that it focused on undergraduate and graduate students enrolled in liberal education classes, who unlike pre-professional and professional students are not usually accustomed to pbl. nevertheless, many of these students aspire to enroll in academic programs such as medicine, social work, and law where pbl courses and entire curricula are widespread. preparing students to cross the bridge from mainly teacher-centered to student-centered learning is important if they are going to succeed in academic programs in which pbl courses predominate. in the current study, negative feedback toward pbl was given by all but two undergraduate students, and was categorized as 1. partiality towards didactic or directed learning, 2. perceptions of reduced learning efficiency, and 3. feelings of uncertainty. similar results were obtained for graduate students with only two graduate students providing positive feedback about pbl, categorized as 1. engaging with other learners, and 2. independent work. nevertheless, pbl student test performances, analyzed by anova and chi square analysis for both mean and grade distribution, were no different compared to students from classes in which information was presented by traditional lectures. in conclusion, despite negative student perceptions about learning in pbl classes, based upon student test performances, learning was not compromised in the pbl method. based upon a combination of observation and questions, i judged the students to be actively engaged in the learning process. results from the present study demonstrate a need to convince students that learning can and in fact, does occur despite a sense of discomfort by students in the pbl classroom. furthermore, pbl curricula are ever-increasing in graduate and professional programs world-wide. therefore, experiences in pbl during undergraduate studies can help to prepare students for success in graduate and professional program pbl curricula and careers in which problem solving takes place. further research is necessary to learn about different ways in which pbl can be introduced and gradually implemented into both undergraduate and graduate curricula. a measured approach may address students’ initial negative perceptions of pbl, while still providing meaningful student-centered learning experiences. references albanese, m.a. and mitchell, s. (1993). problem-based learning: a review of literature on its outcome and implementation issues. academic medicine, 68, 52-81. alleyne t., shirley, a., bennett, c., addae, j., walrond, e., west, s., and pinto pereira, l. (2002). problem-based compared with traditional methods at the faculty of medical sciences, university of the west indies: a model study. medical teacher, 24, pp. 273-279. barr, r.b. and tagg, j. (1995). from teaching to learning: a new paradigm for undergraduate education. change, 27: 6. boud, d. and feletti, g. (1997). the challenge of problem based learning (2nd ed.) london: kogan page. biley, f. (1999). creating tension: undergraduate student nurses response to a pbl curriculum, nurse education today, 19, pp. 586-591. alessio, h. journal of scholarship of teaching and learning, vol. 4., n. 1, may, 2004. 33 chapman, d.w. (2000). designating problems for motivation and engagement in the pbl classroom. journal on excellence in college teaching, 11: 73-92. conway, j.f. and little, p.j. (2000). adopting pbl as the preferred institutional approach to teaching and learning: considerations and challenges. journal on excellence in college teaching, 11 (2&3), 11-26. dahlgren, m.a. and dahlgren, l.o. 2002. portraits of pbl: students’ experiences of the characteristics of problem based learning in physiotherapy, computer engineering, and psychology. instructional science, 30, pp.111-127. des marchais, j.e., bureau, m.a., dumais, b., pigeon, g. (1992). from traditional to problembased learning: a case report of complete curriculum reform. medical education, 26, 190-199. duch, b.j., groh, s.e., and allen, d.e. (2001). why problem-based learning? in b. duch, s.e. groh, and d.e. allen (eds.) the power of problem-based learning. sterling, virginia: stylus press, pp. 6-7. dunkbase j.a. and penick, e. (1990). problem-solving in the real world. journal of college science teaching, 19, 367-370. margetson, d. (1996). beginning with the essentials: why problem-based learning begins with problems. education for health, 9, 61-69. martin, k.j., chrispeels, j.h., and d’emidio-caton, m. (1998). exploring the use of pbl for developing collaborative leadership skill. journal of school leadership, 8, 470-500. millis, b.j. and cottell, p.g. (1999). cooperative learning for higher education faculty. westport connecticut: oryx press. nardi, d.a., and kremer, m.a. (2003). learning outcomes and self-assessments of baccalaureate students in an introduction to nursing course. the journal of scholarship of teaching and learning, 3: 43-56. norman, g.r. and smidt, h.g. (1992). the psychological basis of problem based learning: a review of evidence. academic medicine, 76, pp. 557-565. quinlan, k.m. (2000). an evaluation of a literature database to support problem-based learning. journal of excellence in college teaching, 11: 27-39. seaberry, j. (2002). introducing problem-based learning into quantitative analyses: a primer guide and literature review. journal of excellence in college teaching, 13: 19-39. solomon, p. (2001) perceptions of student peer tutors in a problem-based learning programme. medical teacher, 23, pp.181-186. solomon, p. (2001). perceptions of student peer tutors in a problem-based learning programme. medical teacher, 23, 181-187. alessio, h. journal of scholarship of teaching and learning, vol. 4., n. 1, may, 2004. 34 sundglad, g., sigrell, b., john, l.n., and lindkvist, c. (2002). students’ evaluation of a learning method: a comparison between problem based learning and more traditional methods in a specialist university training programme in psychotherapy. medical teacher, 24, pp. 268-272. vernon d. and blake, r. (1993). does problem based learning work? meta-analysis of evaluative research, academic medicine, 68, pp. 550-565. walton, h.j., matthews, m.b. (1989). essentials of problem-based learning. medical education, 23, 542-558. west, s. a. (1998). objectives in response to students’ uncertainty in a pre-clinical problembased learning curriculum. education for health, 11, 343-347. 503 service temporarily unavailable 503 service temporarily unavailable nginx/1.14.1 503 service temporarily unavailable 503 service temporarily unavailable nginx/1.14.1 503 service temporarily unavailable 503 service temporarily unavailable nginx/1.14.1 journal of the scholarship of teaching and learning, vol. 16, no. 6, december 2016, pp. 56-71. doi: 10.14434/josotl.v16i6.19843 the neglected case of the ‘w’: waste or wisdom? sridhar ramachandran1 and annette wyandotte2 abstract: institutional withdrawal policies allow students with academic and personal difficulties to avert or minimize damage to the gpa by dropping a course in which they may risk a grade of ‘d’ or ‘f,’ or to reduce credit hours to better manage those that remain. offsetting costs involve lost tuition, delay of progress, and perhaps wasted effort. yet in the pressures of the moment, how likely are students to weigh competing factors before acting? do they consider the implications for financial aid, satisfactory academic progress, admission to the major, or timely progress toward a degree? what duties may the institution share with students to promote informed judgments? the authors explore the rewards and risks of dropping a course in view of the literature on the ‘w.’ further, they explicate their own case study of withdrawal behaviors in an introductory course to the informatics major to address questions raised in light of the growing literature on student retention and persistence. “what ties may exist between dropping a course and dropping out of college?” “how may student success studies guide institutions and faculty to assist students facing ‘w’ decisions?” “of what benefit may this kind of intervention be for them?” for example, “how may proactive assistance in the ‘w’ process encourage students to pursue the competencies of a college degree amid discomfort and frustration likely having both cognitive and affective elements?” “how may students who persist through this challenge be better prepared for the complexities of their lives and work?” (kuh, kinzie, schuh, & whitt, 2010). keywords: withdrawal, ‘dfw’ attrition, college students, student success, retention, persistence to graduation, drop-out prevention, drop-out research introduction the authors first learned of studies on student success as part of a chancellor-appointed task force to inform the campus in its effective principles and practices for strategic planning. in the last weeks of this multi-year project, each participant was invited to conduct a short-term inquiry to benefit a specific academic unit.3 unaware then of studies on the ‘w,’ the authors took a student-success perspective in the small case study presented below, to examine withdrawal across a core introductory course in a major within the school of natural sciences. though realizing that the design would not permit generalization, they expected it to provide helpful 1 associate professor, informatics department, indiana university southeast, indiana, usa, sriramac@ius.edu 2 assistant vice chancellor for academic affairs and professor of english, indiana university southeast, indiana, usa, awyandot@ius.edu 3 the work of the task force occurred at the authors’ four-year, public comprehensive university in 2013-2015; for demographics, see n. 6 infra. mailto:sriramac@ius.edu mailto:awyandot@ius.edu ramachandran and wyandotte journal of the scholarship of teaching and learning, vol. 16, no. 6, december 2016. josotl.indiana.edu 57 material for the informatics program coordinator and faculty to consider.4 but the results raised other questions, including, “what help may the student success literature give to address ‘w’ behaviors—whether taking a ‘w’ (dropping a class) or taking a ‘w’ (dropping out of college)?” 5 also, “what insights do studies of withdrawal offer? in seeking answers to these and related issues, the authors have since reviewed the literature of withdrawal, revisited ‘w’-relevant topics within studies of retention and persistence, and looked more deeply into the potential implications of their case study, each resource shaping the present article. they encourage readers to consider the ideas and practices reported here, as well as the opportunities they suggest to assist students to make wise decisions about withdrawals. the ‘w’: reward, risk, and other insights the choice to drop a course or to leave college lies with the student, who may lack the necessary counsel to assess the situation fully. while some students may check with an advisor or instructor, many may just disappear. even if they have viewed their schedule planner for options, how likely are they to gather the right kind of information and to weigh their choices— at a time often given to an urgency for closure in the moment? yet for educators to assist them effectively, they must know more about the shaping influences that may underlie such pressure. in terms of withdrawal from college, tinto (1994) found that 20% of these acts were involuntary. of the 80% remaining, poor grades were not a common factor. students were more likely to flee for non-academic reasons: lacking commitment, making a poor adjustment, having unclear goals, finding a course irrelevant, or feeling isolated as a mismatch to the institution or as not belonging in college. could factors behind course withdrawal be similar? in the authors’ case study for the student success task force, they were surprised to find that many of the informatics course drop-outs had also left college within the five semesters reviewed. they wondered what cognitive and non-cognitive influences may have motivated their decisions. how did they feel about themselves at the time? did newer students perhaps bring inhibiting preconceptions to college, such as family issues, societal conditions, or negative messages experienced in school (kramer, 2007)? although the local design did not include a student follow-up, these questions prompted the authors to study both the literature on withdrawal and on student success.6 4 informatics was chosen due to a co-author’s access as program coordinator and course instructor of the study’s five sections of info-i101 that provided the data. 5 textual use of the ‘w’ refers to a course withdrawal, when contrasted with a ‘w’ as withdrawal from college. otherwise, ‘w’ refers generically to either kind of withdrawal. 6 the authors’ university serves 5750 undergraduates: traditional/non 71%/29%; male/female 40%/60%; caucasian/non 85%/14%; in-state out 67%/33% (representing 16 in-state counties, 14 states, and 36 foreign countries); commuters/non 70%/30%; student-faculty ratio 16:1; full/part time faculty 46%/54%; with class sizes <20 =29%; <20-49= 69%; >49 = 2%. ramachandran and wyandotte journal of the scholarship of teaching and learning, vol. 16, no. 6, december 2016. josotl.indiana.edu 58 withdrawal behaviors in a local case study methodology as part of their inquiry on the ‘w in a stem field, the authors used final grade rosters from five sections of introduction to informatics, info-i101, which served as a gateway for program admission to the informatics major on their campus. pre-majors had to attain a grade of c or higher. they took the course between fall 2010 and spring 2014, each section averaging 20 students. transcript review of the sample indicated whether each participant had re-taken the course, withdrawn from one or more other courses, and had either graduated, remained actively enrolled, or left the university. the student information system provided demographic information on age, gender, full or part-time status, and class year. findings drill-down analysis of these data yielded the results reported in table 1: table 1 w data for info-i101, fall 2010-spring 2014 sample size (total unique ‘w’ grades) 16 average age of students who took a ‘w’ 27 yrs. gender composition of those students (m/f) 63% / 37% combined percentage of juniors and seniors 19% combined percentage of freshmen and sophomores 81% percentage of students with a prior ‘w’ history 63% percentage of students who later retook the course after once withdrawing from it 12% percentage of students who did not retake the course after once withdrawing from it 88% discussion data analyses raised several questions for the authors. for instance, what precursors of course withdrawal could have been discernible to the instructor? what responsibilities may faculty have to intervene in such cases? how did campus ‘w’ policy help or hinder a chance to do so, prior to withdrawal? of the 16 students who withdrew from info-i101, 88% had not re-taken the course, thus never gaining admission to the major. over 90% of them had been attending college full time. yet a records check for persistence revealed that all had left the university, whereas all students who had re-taken the course had become majors and progressed to being seniors by fall 2014. regarding other attributes of the withdrawn students, 43% had no prior history with taking a ‘w’ and over half had a gpa of 2.0 or higher, which suggests that non-academic factors may have had some effect. but given the average age of 27, maturity appears not to have been an influence in opting for the ‘w.’ on the other hand, perhaps taking a course in a college field not ramachandran and wyandotte journal of the scholarship of teaching and learning, vol. 16, no. 6, december 2016. josotl.indiana.edu 59 previously studied in high school led some students to lack confidence or to find the subject matter unappealing. performance anxiety may have been involved—whether directly or in association with the baggage of anxious-making prior experiences. on the other hand, gender results raised no flag, as the 2:1 ratio of male to female students is not unusual in stem fields. however, two demographics warrant further inquiry: the 2:1 ratio of withdrawals by students with a ‘w’ history and the 4:1 ratio of freshmen and sophomores to juniors and seniors. implications hindsight has suggested additional ways for the authors to have informed their case study: 1) comparing the demographic categories through cross-tabulated ratios between the sample of withdrawn students and the general population of all students in the five course sections and 2) using an interview/focus group process to probe student decisions. “what were their considerations for retaking the course or not?” what lapse of time was involved in deciding to withdraw and/or to sign up again for the course, if at all?” the case also indicated prospects for future study of ‘w’ behaviors: 1) a more robust design (e.g. to look for significant correlations), 2) a comparison of gateway courses to several key majors for ‘w’ and/or ‘w’ patterns, and 3) examining ‘w’ and/or ‘w’ behaviors in pre-requisites to gateway courses or in others that block timely progress to enrollment in college-level math or writing. moreover, important work could be involved using the insights and shortcomings of the literature on withdrawal. withdrawal literature: insights and shortcomings the reward of dropping a course is well known to students, as university policy on the ‘w’ allows some flexibility to accommodate extenuating circumstances while in college. nelson (2009) has applied the term “wisdom” to this type of withdrawal decision, finding ample historical evidence in which students maintained their plans to graduate, notwithstanding course withdrawal. since the ‘progressive” era of the earlier 20th century, pre-college institutions have generally acknowledged that learning should take account of the student’s psychological needs as a factor of instruction (clasen and bowman, 1974). but in higher education, it was only in the later part of the century that institutions moved away from a “subject-centered” focus—in which taking a ‘w’ was perceived as “wasteful”—to a “student-centered” approach that now recognizes some course withdrawals as potentially wise (fahy,1973; rump & greet, 1975). contemporary perceptions of course withdrawal may still be similarly unfavorable, as marks on a transcript, a ‘w,’ ‘wu,’ or ‘wd’ remain visible. if recurring, these symbols may give pause to committees for hiring, program admission, or scholarships. though interviewees may offer valid reasons for them, the chance of progressing to that point may be slim.7 nor does the withdrawal literature alleviate this concern. wheland, butler, qammar, katz, and harris (2012) have found university systems who conducted a mandatory review of grade distributions in which excessive w’s were considered a fault of faculty; in other cases, universities have held the student responsible and restricted withdrawal credit hours. still other institutions have taken a no-tolerance stance: students who cross the permissible ‘w’ threshold are subject to automatic program dismissal (gscis, 2014; sobi, 2008; con, 2008; and hssp, 2015). it would also be useful 7 instructors sometimes suggest course withdrawal to low-performing students (osborn, 2001). yet their alertness to how taking a w’ may affect students’ immediate and long-term future, would position faculty to render more fully informed assistance. ramachandran and wyandotte journal of the scholarship of teaching and learning, vol. 16, no. 6, december 2016. josotl.indiana.edu 60 to know that, to some students, ‘w’ policies may seem like a form of punishment, which may impact their self-esteem, lead them to struggle with coping tactics, or in more extreme cases, induce them to resort to a self-defeating behavior, like perfectionism. to explain further, pritchard, wilson, and yamnitz (2007) have found a relationship between negative psychological repercussions and taking a ‘w’ that may also delay students’ academic progress. one correlation points to gender as a function. for example, males are more likely to reject or ignore negative feedback than to consider it a statement about themselves. although females’ self-concepts associate favorably with positive performance feedback, the opposite effect tends to occur when females encounter the negative (mickelson, 1989; correll, 2001; wigfield and eccles, 2002). as a case in point, crosnoe, riegle-crumb, and muller (2007) have discovered that females’ taking a ‘w’ associated it sufficiently with self-doubt to have reported a reduced self-concept. notwithstanding research on the potentially negative psychology of course withdrawal, studies in the ‘w’ literature have primarily addressed poor performance, generally indicated by the number of ‘d, f, w’ results for a given course. typically, institutions have focused on those with the highest rates. according to guskey (1994), this approach commonly identifies the relevant courses, compares their non-passing rates to a pre-set value, discerns patterns, and— where rates have increased—and reviews attempted strategies to enhance performance and their outcomes, to suggest future directions for inquiry. another limitation of the ‘w’ literature is that its studies tend to be older, dating back as much as 40 years: deren, 1969; fahy, 1973; lunneborg, lunneborg, & wolf, 1974; and rump & greet, 1975.8 more recent ‘w’ studies are problematic in other ways. in one instance, hall, smith, brockman, ramachandran, and jasin (2003) surveyed students immediately after they left their courses. but aiming to inform the institution on policy, the researchers gave the results to administrators, with a recommendation to offer tutoring. in the case of single-focused studies, such as looking at student personality traits or at socioeconomic issues, the results failed to produce significant outcomes (lounsbury, saudargas, & gibson, 2004; clague, 2014). further, richie & hargrove (2005) and stucky (2008) have yielded contrary outcomes. finally, michalski (2011) has characterized recent studies of the ‘w’ as largely unhelpful, due to the changing demographics of today’s college student populations. consequently, to the most-asked question—“why do students withdraw?”—many answers seem viable and several appear to be multi-faceted. in this context, the writers turn to the literature on student success. student-success and student-centeredness progressive means, practical ends major changes in primary and secondary education in the last century emphasized the needs of the learner, shaped by progressives like john dewey, maria montessori, and carl rogers (richards, 2014). but since the 80s and 90s, growing societal and economic pressures to retain and graduate college students have pushed pragmatism to the fore in higher education. “bolstered by federal incentives to encourage innovation in the classroom,” student-centeredness and whole-learning interventions have become prevalent in recent studies, according to the 8 for an exception, see andres & carpenter (1997). ramachandran and wyandotte journal of the scholarship of teaching and learning, vol. 16, no. 6, december 2016. josotl.indiana.edu 61 stanford center for opportunity policy in education (in richards, 2014). as a result, many tested strategies now exist whose application may assist faculty, advisors, and administrators to re-dress poor academic performance and psychological setbacks, both in their own right and in order to help students to make informed course withdrawal decisions. this work becomes the more urgent in the contemporary context in which public colleges and universities rely increasingly on tuition to supplement shrinking revenues in the face of legislative cutbacks to higher education, as well as shifts in funding formulas, based on degrees awarded rather than enrollment. institutions are avidly pursuing more impactful ways to retain students and to accelerate time to a degree through innovations such as centers for writing, math, science, mentoring, tutoring, testing and student development. non-academic measures have included counseling centers, as well as centers for adults and veterans and even for child-care. some institutions are collaborating with others to enhance student-achievement programming. as a case in point, the american association of colleges and universities has recently launched a project for “re-imagining the first-year experience,” bringing together faculty and administrators of 44 institutions around the country. following a master syllabus and working both individually and cooperatively, member teams are pursuing goals, objectives, and outcomes over a three-year period to enhance the freshman year.9 to make best use of institutional resources to improve retention and graduation rates, colleges and universities are relying on data-informed assessment of initiatives to learn which of them may make the most difference. for example, evaluative measures have compared campuswide rates of retention and graduation, transfer success, time to degree, improvement of grade point averages, and reduction of gateway courses with high ‘d/f/w’ rates (kramer, 2007). in this context, the writers next consider the integrative manner in which literature on student success considers both the academic and the affective, both student achievement and satisfaction, and both short-term and long-term strategies—without presuming to know what students think or feel (levitz, noel, & richter, 1999, p. 37). whole-student strategies kuh, kinzie, schuh, and white (2005) have investigated the attributes of public and private institutions known for higher-than-predicted graduation rates and levels of student satisfaction. their results address cognitive and non-cognitive factors shown to be effective to promote student success in different ways. from a classroom perspective, outcomes have pointed to high performance, sufficient time on task, and support of students through their academic challenges. over 18 months and using on-site visits and cross-site analyses of 20 colleges and universities, these researchers have found that these commonly identified educational components also fared well on measures associated with the national survey of student engagement (nsse, 2015): level of academic challenge, active and collaborative learning, student-faculty interactions, enriching educational experiences, and offering a supportive campus environment. other studies cohere with these findings in recommending pedagogies that consider students’ potential academic and affective concerns, being more apt to evoke a will to work through challenges (rothman, 2011). for instance, to foster understanding of course content and patience for an intensive learning environment, pascarella and terenzini (2005) have pointed to 9 the authors are part of the campus team effort under aac&u leadership working on multiple projects, such as removing policy barriers, examining gateway courses, investigating new technology, cultivating “growth mindsets,” and others. ramachandran and wyandotte journal of the scholarship of teaching and learning, vol. 16, no. 6, december 2016. josotl.indiana.edu 62 group problem-solving and peer teaching among the beneficial ways to engage students’ classroom participation and time on task. other methods for deeper learning have involved ongoing practice of course-essential skills, facilitating students to connect new knowledge with prior learning, giving them timely feedback, and framing their academic challenges as resolvable issues whose solutions are personally meaningful. high impact practices (hips) also offer faculty a set of content and methods for student learning which combine curricular and co-curricular motivations to improve student performance and persistence. they include collaborative learning, diversity (global learning), learning communities, first-year experiences, undergraduate research, applied learning (e.g. service learning and internship opportunities), common intellectual experiences, capstone courses and projects, as well as writing intensive courses (kuh, 2013; kuh & o’donnell, 2013; kuh, kinzie, schuh, and whitt, 2010). hips associate also with socio-psychological influences coherent with students’ self-reports of high satisfaction in a number of surveys.10 kuh (2013) has found correlations between hips practices and survey outcomes such as nsse, in which students have self-reported gains in gpa and progress toward a degree. kuh has credited them with engaging learners’ pursuit of deeper meaning. to illustrate, they seem to have generated a sense of fulfillment in connecting students to the subject, the instructor, and their peers, in constructing knowledge together. participation in collective knowing of this kind has correlated with a 25% rate of improved retention (tinto, 2012). student success literature has also found that collaborative learning appeals to the whole-person (kramer, 2007). farrington (2013) has similarly applauded generating student instructional experiences that appeal to the intellect and the affect. “mental contrasting” is an example that seems both to enhance achievement and to move students’ will to keep after a challenging goal. based on their study of meta-analyses, gollwitzer, oettingen, kirby, duckworth, and mayer (2011) have characterized this process as planning for success: 1) students first envision their goals and imagine themselves achieving them; 2) then they anticipate potential barriers and devise in advance strategies to overcome them—while keeping the desired outcome in view. mindset growth represents another line of thought on whole-person instruction. in the new psychology of student success, carol dweck (2007) posits that whether students may be willing to face a challenge depends in part on their having a positive or “growth” mindset in which they accept trial and error and work hard to succeed. in contrast, students with a “fixed” mindset may be reluctant to take risks, believing intelligence for particular knowledge or skill to be innately limited. tough (2012) speaks of this fear of failing even among students who have achieved success in other areas. dweck cautions that to promote a growth mindset is not about preaching, cheerleading, or raising false expectations. rather, it requires showing students how to learn through challenge by doing honest critique. facilitating this mindset includes student buy-in to revise and to stay for the long haul. blackwell, trzeniewski, & dweck (2007) and fensterwald (2015) have observed students patiently improving even as early as primary school. national and regional non-profit foundations also support the prospect of growth through learning. for example, since 2010, the william and flora hewlett foundation has funded projects for deeper learning especially featuring “children and youth in higher poverty communities” (farrington, 2013, p. 2). in connection with research for the university of chicago’s consortium, farrington’s inquiry has acknowledged academic mindset as a vital 10 see the nsse institute project deep related papers at http://nsse.iub.edu/institute/; also see the ruffalo noellevitz college student inventory (2015), and the gallup-purdue index (ray & kafka, 2014). ramachandran and wyandotte journal of the scholarship of teaching and learning, vol. 16, no. 6, december 2016. josotl.indiana.edu 63 whole-person element of motivating students’ “underlying self-perceptions of their own ability to succeed” (p. 7).11 oyserman (2009) characterizes motivation as choosing to act one way over another in relation to a sense of being in the world. she believes that human identities—even of children— are constructed in the dynamics of cumulative experience that move an individual to continue the same behaviors absent reasons to do otherwise. her recent book pathways to success through identity-based motivation (2015) theorizes and evidences these ideas. she recommends the kinds of interventions that evoke students’ trust in the prospect of their future achievement. for example, pathways ties early education to a young person’s natural drive to succeed. but problems in school and in life may dull and distance what once seemed feasible. recounting examples drawn from meta-analyses, oyserman lays fertile ground for scholarly teachers and teacher scholars to excavate, perhaps including action research in classroom to implement and test student success strategies.12 similarly, college students who develop confidence by seeing themselves doing better are more likely to be self-sustaining through a challenge, especially with timely support. in the past decade, the concept of grit has been correlated with an ability to persist or persevere to sustain a goal. duckworth’s grit studies operationally define the term as “perseverance and passion for long-term goals,” combining an interest in something with a passion to get it done, such as to pursue a challenging course and to finish a college degree (duckworth, peterson, matthews, and kelly, 2007, p. 1087). after developing a scale to measure grit,13 duckworth and colleagues gained access to 1200 west point freshmen ready for the institution’s rigorous course of summer training (duckworth & quinn, 2009). results established grit as a better predictor of drop-outs than the academy’s own “complex” test of the “whole cadet” (in tough, 2012, p. 75). duckworth considers it a way to explain “why some persons “accomplish more than others of equal intelligence”— in a way that goes beyond iq and “conscientiousness” (duckworth, peterson, matthews, & kelley, 2007, p. 1087). the many student-centered approaches considered thus far suggest that, under the right set of circumstances, college can provide a protected space in which to stretch the learner’s comfort zone in order to push beyond the familiar (hoerr, 2013). whether a quality like grit can be actually “taught” (schectman, debarger, dorsife, rosier, & yarnall, 2013), or is merely “caught” by example—in telling and hearing one’s own and others’ stories of perseverance— growth in resilience arguably merits a place in one’s toolkit for success in learning and living. two decades ago land and hannafin (1996) called on participants at the national convention of the association for educational communications and technology to create learning environments “rooted in psychological, pedagogical, technological, cultural, and pragmatic foundations” driven by new knowledge of how students “think and learn” (p. 396). on the other hand, nelson (2009) cautioned that little is known about how students cope with the struggles that lead them to decide one way or another. nonetheless, thanks to studies forming the literature of student success, educators may access a number of methods shown to enhance learning, retention, and persistence in college. 11 also see blackwell, trzensniewski, and dweck (2007) 12 “action research” refers to a qualitative form of study in which the actor (e.g. an instructor) investigates how to improve something (e.g. an instructional method) for the benefit of self or others (e.g. instructing students). for a case in point, see wyandotte, a. (2009). 13 the grit measure has since been refined to an 8-point scale called the “grit-s.” see duckworth and quinn (2009). ramachandran and wyandotte journal of the scholarship of teaching and learning, vol. 16, no. 6, december 2016. josotl.indiana.edu 64 classroom faculty spend the most time with college students and are the persons on campus whom they most respect. arguably, they should be first responders to observed or selfreported difficulties. 14 moreover, their timely instructional counsel could help to promote students’ making informed decisions that also factor in consequences that they otherwise not consider. however, this view presumes that faculty are in the know about what students may contemplate. yet concerning the prospect of course withdrawal, at the authors’ university it is strictly an online process, lacking a built-in intervention. students’ disappearance may be instructors’ first notice of course withdrawal, as was the case in the informatics study. in that light, as part of the university’s participation in the aac&u’s “reimagining” project, a policy audit has been conducted, with a new course withdrawal policy about to be heard by the academic policy committee. by its provisions, when students go to the withdrawal website to give notice of their intent to withdraw, a 72-hour hold is triggered, with notices to the instructor, advisor, and office of financial aid. they determine how and what to communicate to these students before the hold expires. at the very least, they will be given timely information to consider. when the hold is lifted, they may submit or retract their intent to withdraw. this approach recognizes that course withdrawals are sometimes warranted, or perhaps even unavoidable. suggested interventions surrounding withdrawal knowing that students will continue to withdraw from their courses and from college, an institution would benefit from strategies to bring students back again. below, the authors suggest some common sense interventions. providing a positive withdrawal experience  a positive course withdrawal experience15 for both student and institution could be part of the groundwork for retaking classes or returning to college. one can seldom underestimate the power of a word-of-mouth or of planting ideas for future action.  institutions could consider withdrawal as an act of the whole person—working from the outset to bring them back. but more often, students fill in forms, answer survey questions, scout for approval signatures, and perhaps attend a pro forma interview.  to expedite timing, the university bursar could perhaps offer the student a discounted rate toward tuition to re-take the class the next semester. 14 results of student self-reports in the purdue gallup index (ray & kafka, 2014) revealed factors impacting students in one of three ways: leading them to thrive, to struggle, or to suffer. meaningful engagement with faculty was high on the list for thriving, with twice the benefit when faculty also offered caring support for them as individuals, encouraged their pursuit of goals and dreams, and excited them to learn. www.insideindianabusiness.com/_newsitem.asp?id=65008 http://www.insideindianabusiness.com/_newsitem.asp?id=65008 ramachandran and wyandotte journal of the scholarship of teaching and learning, vol. 16, no. 6, december 2016. josotl.indiana.edu 65 dear student_name , we noticed that you withdrew from [course “x”] this term. to help you recover from this setback, we are offering you a 25% tuition discount to retake it next semester. when paying your next bill, present this official notice to the bursar to receive this courtesy. sincerely, university bursar xyz university re-engaging the withdrawn student  an academic advisor could automatically send students a revised plan for the next semester that includes re-taking one or more courses.  an instructor could send a short personal note to invite students back next term and perhaps encourage an office visit to discuss how to better prepare them. for example, a simple note may say, dear student_name , i have learned of your recent withdrawal from class. you will be missed, but i hope to see you back again next semester. retaking the class soon will benefit you because you already have some insight on which to build from your recent experience. i am saving your assignments in the hope that you will set up an appointment with me before the next term to discuss what might help you to better prepare for that experience. sincerely ~professor xyz [course x]/contact information ramachandran and wyandotte journal of the scholarship of teaching and learning, vol. 16, no. 6, december 2016. josotl.indiana.edu 66  a unit coordinator or dean in the students’ major or pre-major could send a personal note before the next registration period to layout and encourage steps to resume progress (e.g. retaking the course or returning to campus), perhaps offering a priority registration to ensure the chance for a seat in classes likely to fill quickly. employing communal strategies to bring students back  models from alumni recruitment could assist institutions to collect updated student information, find ways to develop ongoing campus commitment, be persistent in contacting those who leave, and join a social with an academic reason to come back to campus. 16  likewise, methods in business may help institutions to retain or to re-engage absent students—an approach used frequently by casinos.  additional motivation may derive from institutions who employ so-called “social proof” techniques, such as testimonies for withdrawn students that many like themselves have successfully returned to college when the time is right.  also, gamification techniques like those reported in huang and soman (2013) may offer plausible tactics for motivating desired behaviors like returning to college. conclusion this research has explored the limitations of the literature on the ‘w’ in juxtaposition to the promises that contemporary research has associated with student success. it has also presented the authors’ limited case study in which students who withdrew from and failed to retake the 16 interested readers will find detailed research in shannon & bylsma (2006). dear student_name , we noticed that you have withdrawn from course x this semester. it is offered next semester, and we are hopeful that you will retake it and get back on track with your graduation progress with us. course x can fill during semester pre-registration. so to help you stay on track with your graduation plan, we are willing to extend to you a priority registration into course x if you are interested. to request it, just bring this letter and your university id card to the departmental advising office before 4pm on xx/xx/xxxx. sincerely~ [contact information] ramachandran and wyandotte journal of the scholarship of teaching and learning, vol. 16, no. 6, december 2016. josotl.indiana.edu 67 introductory course to informatics were also found to have withdrawn from college. more particularly, the essay has presented a foundation for educators to experiment with student success strategies in their own classrooms as scholarly teachers, or in their own scholarship of teaching. indeed, these are the acts of concerned educators, representing the kind of work that caring teachers and scholars already do. they raise questions and pursue answers. they read, apply, and test the literature in their fields. they raise consciousness of the wasteful acts and promote the wise. they also advocate for positive change so that learning and learners may thrive. in short, they understand that “the success of an institution and its students are inseparable” (levitz, noel, & richter, 1999, p. 31). references andres, l., & carpenter, s. (1997). today's higher education students: issues of admission, retention, transfer, and attrition in relation to changing student demographics. british columbia council on admissions and transfer. vancouver: education resources information center. blackwell, l. s., trzeniewski, k.h., & dweck, c.s. (2007). implicit theories of intelligence predict achievement across an adolescent transition: a longitudinal study and intervention. child development, 78(1), 246-263. clague, p. (2014). why trade students withdraw from their courses: students’ perspectives. new zealand: unitec institute of technology. clasen, r. e., & bowman, w. e. (september 1974). toward a student-centered learning focus inventory for junior high and middle school teachers. journal of educational research, 68(1), 9–11. con. (2008). undergraduate nursing student handbook. purdue university-calumet. retrieved from http://academics.pnw.edu/nursing/undergraduate-handbook-toc/ correll, s. j. (2001). gender and the career choice process: the role of biased self-assessments. american journal of sociology, 106, 1691–1730. crosnoe, r., riegle-crumb, c., & muller, c. (2007). gender, self-perception, and academic problems in high school. social problems, 54(1), 118–138. deren, s. (1969). students who withdraw from courses: a descriptive analysis. hofstra university, center for the study of higher education. hempstead, n.y.: education resources information center. duckworth, a. l., kirby, a., gollwitzer, p. m., & oettingen, g. (2013). from fantasy to action: mental contrasting with implementation intentions (mcii) improves academic performance in children. social psychological and personality science, 4, 745-753. duckworth, a. l., & quinn, p. d. (2009). development and validation of the short grit scale (grit–s). journal of personality assessment, 91(2), 166-174. http://academics.pnw.edu/nursing/undergraduate-handbook-toc/ ramachandran and wyandotte journal of the scholarship of teaching and learning, vol. 16, no. 6, december 2016. josotl.indiana.edu 68 dweck, c. s. (2006). mindset: the new psychology of success. new york: ballantine. fahy, p. (1973). psychological aspects of student wastage. studies: an irish quarterly review, 62 (247/248), 195-204. farrington, c.a. (2013). academic mindsets as a critical component of deeper learning. retrieved from http://www.howyouthlearn.org/pdf/white_paper_academic_mindsets_as_a_critical_compone nt_of_deeper_learning_camille_farrington_april_20_2013.pdf. fensterwald, j. (2015, november 23). there’s more to a growth mindset than assuming you have it. retrieved from http://edsource.org/2015/theres-more-to-a-growth-mindset-thanassumingyou-have-it gollwitzer, a., oettingen, g., kirby, t. a., duckworth, a. l., & mayer, d. (2011). mental contrasting facilitates academic performance in school children. motivation and emotion, 35, 403-412. gscis. (2014). graduate catalog. nova southeastern university. retrieved from http://www.scis.nova.edu/documents/catalog.pdf guskey, t. r. (1994). making the grade: what benefits students? educational leadership, 52(2), 14-20. hall, m., smith, k., boeckman, d., ramachandran, v., & jasin, j. (2003). why do students withdraw from courses? southern association for institutional research, 01-11. san antonio, tx. hoerr, t. r. (2013). fostering grit: how do i prepare my students for the real world? retrieved from ascd publications at www.ascd.org hssp. (2015). policies and procedures. drexel university. retrieved from http://drexel.edu/pennoni/visitingscholars/policies_procedures/ huang, w. h. y., & soman, d. (2013). gamification of education. research report series: behavioural economics in action. retrieved from http://inside.rotman.utoronto.ca/behaviouraleconomicsinaction/files/2013/09/guidegamificatio neducationdec2013.pdf kinzie, j., & kuh, g. d. (2007). creating a student-centered culture. in g. l. kramer (ed.), fostering student success in the campus community. san francisco, ca: jossey-bass. kramer, g. l. (ed.). (2007). fostering student success in the campus community. san francisco, ca: jossey-bass. kuh, g. d. (2013). assessing underserved students’ engagement in high impact practices. in finley & mcnair, (eds.) retrieved from http://www.aacu.org/assessinghips kuh, g. d., & o’donnell, k., & reed, s. (2013). ensuring quality and taking high-impact practices to scale. (washington, dc: association of american colleges and universities.) http://www.howyouthlearn.org/pdf/white_paper_academic_mindsets_as_a_critical_component_of_deeper_learning_camille_farrington_april_20_2013.pdf http://www.howyouthlearn.org/pdf/white_paper_academic_mindsets_as_a_critical_component_of_deeper_learning_camille_farrington_april_20_2013.pdf http://edsource.org/2015/theres-more-to-a-growth-mindset-than-assuming-%09you-have http://edsource.org/2015/theres-more-to-a-growth-mindset-than-assuming-%09you-have http://www.scis.nova.edu/documents/catalog.pdf http://www.ascd.org/ http://drexel.edu/pennoni/visitingscholars/policies_procedures/ http://inside.rotman.utoronto.ca/behaviouraleconomicsinaction/files/2013/09/guidegamificationeducationdec2013.pdf http://inside.rotman.utoronto.ca/behaviouraleconomicsinaction/files/2013/09/guidegamificationeducationdec2013.pdf http://www.aacu.org/assessinghips ramachandran and wyandotte journal of the scholarship of teaching and learning, vol. 16, no. 6, december 2016. josotl.indiana.edu 69 kuh, g.d., kinzie, j., schuh, j.h., & whitt, e. j. (2010). student success in college: creating conditions that matter. san francisco, ca: jossey-bass. land, s. m., & hannafin, m. j. (1996). student-centered learning and interactive multimedia: status, issues, and implications. contemporary education, 68(2), 94-97. levitz, r. s., noel, l., & richter, b. j. (1999). strategic moves for retention success. in g. h. gaither (ed.). promising practices in recruitment, remediation, and retention). new directions for higher education, n. 108. san francisco, ca: jossey-bass 31-48. lounsbury, j. w., saudargas, r. a., & gibson, l. w. (2004). an investigation of personality traits in relation to intention to withdraw from college. journal of college student development, 45(5), 517-534. lunneborg, c. e., lunneborg, p. e., & wolf, v. d. (1974). why do university of washington students withdraw from classes? seattle: education resources information center. michalski, g. v. (2011). a text mining analysis of college course withdrawals. association for institutional research annual forum, 01-66, toronto. mickelson, r. a. (1989). why does jane read and write so well? the anomaly of women’s achievement. sociology of education, 62, 47–63. national survey of student engagement (2015). nsse survey instruments. retrieved from http://nsse.indiana.edu/html/survey_instruments.cfm nelson, v. (2009). should my college student consider withdrawing? college parent central. retrieved from http://www.collegeparentcentral.com/2009/10/-should -my-college-studentconsider-withdrawing-from-a-class/ noel-levitz (2015). college student inventory. retrieved from https://www.noellevitz.com/student-retention-solutions/retention-management-systemplus/college-student-inventory osborn, e. (2001). when should a student withdraw? the chronicle of higher education. retrieved from http://chronicle.com/blogs/onhiring/when-should-a-student-withdraw/29424 oyserman, d. (2015). pathways to success through identity-based motivation, cary, n.c.: oxford university press. oyserman, d. (2009). identity-based motivation. journal of consumer psychology, 19(3), 250260. pascarella, e. t., & terenzini, p. t. (2005) how college affects students: a third decade of research, (2), san francisco, ca: jossey-bass. pritchard, m. e., wilson, g. s., & yamnitz, b. (2007). what predicts adjustment among college students? a longitudinal panel study. journal of american college health, 56, 15-21. ray, j. & kafka, s. (2014). life in college matters for life after college. retrieved from http://www.gallup.com/poll/168848/life-college-matters-life-college.aspx?version=print http://nsse.indiana.edu/html/survey_instruments.cfm https://www.noellevitz.com/student-retention-solutions/retention-management-system-plus/college-student-inventory https://www.noellevitz.com/student-retention-solutions/retention-management-system-plus/college-student-inventory http://chronicle.com/blogs/onhiring/when-should-a-student-withdraw/29424 http://www.gallup.com/poll/168848/life-college-matters-life-college.aspx?version=print ramachandran and wyandotte journal of the scholarship of teaching and learning, vol. 16, no. 6, december 2016. josotl.indiana.edu 70 richie, s. d., & hargrove, d. s. (2005). an analysis of the effectiveness of telephone intervention in reducing absences and improving grades of college freshman. journal of college student retention: research, theory & practice, 6(4), 395-412. richards, e. (2014, april 2). student-centered learning. retrieved from https://edpolicy.stanford.edu/news/articles/1193 rump, e. e., & greet, n. s. (1975). the characteristics and motivations of students who withdraw without failing. vestes: australian universities' review, 8(2), 150-160. schectman, n., debarger, a. h., dorsife, c., rosier, s., & yarnall l. (2013). promoting grit, tenacity and perseverance. critical factors for success in the 21st century. washington, d.c.: u.s. department of education. shannon, s. g., & bylsma, p. (2006). helping students finish school: why students drop out and how to help them graduate. olympia, wa: office of superintendent of public instruction. sobi. (2008). academic procedures. the university of texas. retrieved from https://sbmi.uth.edu/current-students/student-handbook/academic-procedures.htm stucky, t. d. (2008). investigating whether contacting absent studets increases course success. journal of the scholarship of teaching and learning, 8(1), 61-71. sung, h. k. (2014, february 26). evidence-based (simple but effective) advice for college students: microaction and macrochange. retrieved from https://dus.psu.edu/mentor/2014/02/evidence-based-simple-but-effective-advice-for-collegestudents-microaction-and-macrochange/ tinto, v. (2012). completing college: rethinking institutional action. chicago: university of chicago press. tinto, v. (1994). leaving college: rethinking the causes and cures of student attrition. chicago: university of chicago press. tough, p. (2012). how children succeed: grit, curiosity, and the hidden power of character. boston: mariner books. wheland, e. r., butler, k. a., qammar, h., katz, k. b., & harris, r. (2012). what are they thinking? students’ affective reasoning and attitudes about course withdrawal. national academic advising association journal, 32(2), 17-25. wigfield, a., & eccles, j. s. (2002). the development of competence beliefs, expectancies for success, and achievement values from childhood through adolescence. in a. wigfield, & j. s. eccles, (eds.), development of achievement motivation, pp. 173–195. san diego, ca: academic press. willging, p. a., & johnson, s. d. (2009). factors that influence students’ decision to drop out of online courses. journal of asynchronous learning networks, 13(3), 115-127. https://edpolicy.stanford.edu/news/articles/1193 https://sbmi.uth.edu/current-students/student-handbook/academic-procedures.htm https://dus.psu.edu/mentor/2014/02/evidence-based-simple-but-effective-advice-for-college-students-microaction-and-macrochange/ https://dus.psu.edu/mentor/2014/02/evidence-based-simple-but-effective-advice-for-college-students-microaction-and-macrochange/ ramachandran and wyandotte journal of the scholarship of teaching and learning, vol. 16, no. 6, december 2016. josotl.indiana.edu 71 wyandotte, a. (2009). reciprocal gains in higher order thinking and course content in teaching students to argue and think critically. journal on excellence in college teaching, 20(4), 31-53. 503 service temporarily unavailable 503 service temporarily unavailable nginx/1.14.1 503 service temporarily unavailable 503 service temporarily unavailable nginx/1.14.1 503 service temporarily unavailable 503 service temporarily unavailable nginx/1.14.1 503 service temporarily unavailable 503 service temporarily unavailable nginx/1.14.1 seamon journal of scholarship of teaching and learning, vol. 5, no. 1, pp. 46 – 53. the communication triad: a participatory model for the scholarship of teaching and learning in communication marc seamon1 i. introduction as an area of research, instructional communication has struggled to map out the role of communication in the teaching process (sprague, 1992). more recently, the scholarship of teaching and learning (sotl) movement has emphasized the importance of applying the same systematic, informed rigor in our teaching that we do in our research pursuits (cohen, 1997; cohen, barton & fast, 1999). both concepts share considerable overlap and stem from the desire to improve education. but how can one advance the sotl beyond buzzword status and put it into substantive practice? for communication educators, the key to applying scholarly rigor to the improvement of their teaching lies first in embracing all sides of the communication discipline and then in capitalizing on the many similarities between what they are teaching (communication) and the act of good teaching itself. these prerequisites are vital to a full understanding of the discipline and to generating thoughtful questions about how its teaching might be improved. this paper proposes a model that facilitates discipline-wide understanding for communication educators by illustrating the connections between three components of the discipline—professional practice, the classroom, and the research academy. the model suggests that familiarity with each of the three components increases our purview of the discipline and enables a more thoughtful inquiry into the teaching of communication. figure 1 three elements of the communication discipline joined visually. 1 assistant professor of journalism, marshall university, 121 communications building, huntington, wv 25755. seamon@marshall.edu teaching community research community professional community m. seamon journal of the scholarship of teaching and learning, vol. 5, no. 1, january 2005. 47 ii. the communication discipline as a whole composed of its parts much has been written about the supposed divide between the research academy and the newsroom (pew center, 2000; riffe, hedgepeth & ziesenis, 1992), and arguably, the communication classroom is out of touch with both (duhe & zukowski, 1997; bolding, 1996). even among communication educators, there are those who embrace the industry by reading only trade publications, while a separate group embraces academia by reading only research journals (weaver & wilhoit, 1998). it is clear that in some ways, the discipline has become a segregated lot with distinctly separate points of focus. so where are these different perspectives on communication located? what are the various vantage points from which the discipline can be observed, and what insights do they offer for this quest of self-improvement from within the craft? obviously, one is the media. without professional journalists and their product, mass communication would not resemble what we know it as today. j-schools are surely another point. they may be the only place on earth where you’ll be given the definition of a nut graph or a news peg without asking for it. scholarly research, too, is a home for the discipline of communication. it’s there that the theories of our discipline are incubated. it seems that interest in communication has arisen and evolved simultaneously in newsrooms, classrooms and academic research circles, but these three domains do not always overlap in practice or in the exchange of ideas. these “points” of the communication model—the teaching community, professional community, and research community—all have something vital to offer us in our quest to become better teachers through the sotl. only in knowing each of them can communication educators achieve full disciplinary understanding, and only then are they prepared to systematically and empirically approach the improvement of their teaching. for communication educators, this way of thinking about their subject matter and their teaching practices should be a useful examination of familiar ideas from a combined perspective that they may not have considered before. it is the similarity between the teaching process (pedagogy) and what is being taught (communication) that allows the three-way model of intradisciplinary improvement to be uniquely useful for the teaching of communication. iii. the teaching community “those who can, do. those who can’t, teach” is a jab that working professionals often take at teachers and researchers. lee shulman (1986) provided academics with a strong defense to such banter when he wrote. “those who can, do. those who understand, teach.” the comeback means that effective communication teachers are not failed practitioners who took up another line of work, but rather holistic communication experts who maintain and integrate skills from the discipline’s professional, pedagogical, and research communities. it is only through such discipline-wide synthesis that real understanding and the sotl can be achieved. for communication faculty who “understand” (as shulman defines it), it is clear that the act of teaching and the act of communicating share many similarities. that’s why the “teaching point” of this model belongs here as a legitimate member of the communication discipline. the field of instructional communication affirms that fact by recognizing that teaching is itself a specialized form of communication. whether it’s a course in documentary film or an introductory newswriting class, bringing students to the “same page” as the instructor— establishing common ground—will itself emphasize many of the higher-order goals and m. seamon journal of the scholarship of teaching and learning, vol. 5, no. 1, january 2005. 48 objectives of the curriculum. good teaching implements some of the same observational and synthesis skills that are listed among the desired outcomes of communication courses. the best communication is clear, straightforward communication that tells people what they need to know without confusing them. it anticipates and pre-empts uncertainty and, where possible, provides an avenue for feedback. this simplicity is not necessarily the key to excellence in mathematics, biology, engineering, or other disciplines, but for communication, it is the stock of the trade. it just so happens that the methods of clear communication also make some of the best teaching strategies. instructors face a complex task in teaching communication skills and concepts to their students. to do so, they must synthesize their insight of the discipline just as professional practitioners do when faced with a novel communication situation. in general, both must work diligently to be sure that the correct message is meaningfully understood in the correct context. these parallels between teaching and communication become especially relevant when the subject being taught is communication. shulman (1997) asserts that a profession requires the ability to navigate a complex variety of circumstances and that to do so necessitates a “deep understanding” of the discipline and possession of higher-order skills in its areas of specialty. these skills are what communication students are in school to learn. to best handle the task, communication faculty should polish their teaching skills while staying on top of their discipline by maintaining professional skills at the same level they would if working in the industry. it has been argued that teaching communication requires a special set of skills that mirror the metacognitive, self-monitoring skills of communication itself (book, 1989). the idea that communication educators with experience in the industry would be best equipped to capitalize on these similarities of process between communication and pedagogy is supported in the education literature. for example, sarah dinham (1996) asserts that in addition to knowledge about teaching and knowledge about the discipline, the best teachers must have “disciplinespecific teaching knowledge.” dinham explains that discipline-specific teaching knowledge goes beyond a working knowledge of the subject matter to include an ability to adapt the disciplinary concepts in an infinite number of ways to best suit the teaching needs at hand. such higher-order synthesis can only come from the mastery of the whole discipline—in the case of communication, its teaching, research, and professional communities. for communication educators, such mastery necessarily involves membership in all three communities. as communication educators seeking the sotl consume current education literature emphasizing collaborative, engaged, and student-centered learning (eggen & kauchak, 1999), they will begin to see many conceptual similarities to trends currently shaping the direction of communication and the media, including newsroom teams, civic journalism, and a new awareness of “audience.” conceptualizing those similarities is a big step toward both the sotl and the broader understanding described in dinham’s discipline-specific teaching skills. pursuing the sotl is about becoming an “expert” teacher. to borrow from schema theory, it can be said that when a communication educator’s teaching schema and professional practice schema are each rich and strong, a considerable amount of generalization will occur between them as knowledge and skills are exchanged (walls, 1999). the result will be even more connections in the schema network, which, according to schema theory, is what distinguishes expert, higher-order proficiency from lower levels of ability. the more connections that are made between the teaching schema and the professional practice schema, the more thorough, creative, and effective the instructor will be in facilitating meaningful understanding among students. m. seamon journal of the scholarship of teaching and learning, vol. 5, no. 1, january 2005. 49 so for communication professionals heading into the classroom as teachers, it’s worth noting that the skills gained in the work world are not simply what they should teach but a strong blueprint for how they should teach as well. the skills owned by the best reporters—finding the truth, extracting the essence of a complex process or situation and helping others to understand it, perhaps even better than they ever could have on their own—are the same skills that make great communication teachers. in short, this teaching point of the model can be said to show that “to communicate is to teach; to teach is to communicate.” iv. theprofessional community after interviewing hundreds of reporters, researchers at the pew center for the people and the press concluded that for most journalists, being able to communicate for a living was the most compelling influence guiding them to their chosen careers (pew center, 1999). the sincere desire to communicate thoroughly on the part of professional journalists includes making sure that the message they’re sending is clearly and fully understood (burgoon, bernstein & burgoon, 1983). for journalists, this drive to fully enlighten can overshadow other commonly touted roles of the media. in a survey of journalists and news consumers, burgoon et al. (1983) found that reporters rated the goal of “explaining how important events and issues relate to the community” highest of eight possible functions of the media, including “uncovering wrongdoings,” “providing a thorough (historical) record of events,” and the “watchdog role” of the press. when the department of journalism at ball state university sought to identify differences in the news selection processes employed by student and professional journalists, they found that professional reporters were driven by the need to fully inform (teach) their audiences by explaining all possible aspects of the issue or topic being communicated (pitts, 1987). whereas the less-skilled student journalists in the experiment were content to provide a less-than-thorough explanation, professional reporters were frustrated if they were unable to provide their audiences with complete insight. just as good teachers want their students to have the most enriching educational experience possible, professional communicators are motivated to communicate in ways that result in learning and full understanding. all this would seem to suggest that communication educators who have worked as professional communicators have much to offer in the classroom, and indeed they do. there is evidence in the literature to suggest that everyone, even diehard academics, agrees on the value of realistic, job-specific instruction (duhe & zukowski, 1985; oregon report, 1987). but the need for association flows both ways. newsroom research by the freedom forum (1994) indicates that working journalists are starving for additional instruction about how to do their jobs. of 652 journalists surveyed at 123 daily newspapers in 1993, a consistent majority reported dissatisfaction with the quantity and quality of training, instruction and skills seminars offered by their employers. they also indicated that outside training programs were much more effective and popular than in-house programs. about 22 percent of outside training for reporters is offered by nearby colleges or universities, creating an ideal pathway through which to begin connecting the newsroom and the classroom. such a connection is not a new idea. phillip gaunt, who has researched in detail the history of journalism training throughout the world, writes that the united states has a long history of co-mingling between professional practitioners and j-schools. as early as 1912, major newspapers were throwing significant funding into j-schools at various public and private universities (gaunt, 1992). and internships, which are now seen as a mutually beneficial m. seamon journal of the scholarship of teaching and learning, vol. 5, no. 1, january 2005. 50 arrangement between the industry and j-schools, were soon to follow. by the late 20th century, 80 percent of graduates who found work in their field participated in an internship (1991 journalism career and scholarship guide, 1990). but there is still a rift between academics and working journalists. a survey of newspaper editors by the american society of newspaper editors suggests that media professionals believe one of the best ways to make j-schools better is to have more work-hardened journalists in teaching positions. when j-school faculty have previous professional experience in the media, it’s easier for them to serve as ambassadors to both the newsroom and the classroom. the idea of work-hardened faculty serving as a bridge between the newsroom and the classroom facilitates the model proposed here and advances the pursuit of the sotl for communication educators. journalism teachers can remain active in the newsroom by serving as a stringer or correspondent for a local newspaper. as a part-time contributor, it may be possible for a journalism teacher to cover the regular meetings of a city council or similar event. such meetings usually take place only twice a month on weeknights, so they would not interfere with the daytime duties of a faculty member. those with strong ties to a local paper often write a regular column or contribute to the op-ed page. whatever the arrangement, finding a way to simultaneously occupy all corners of the discipline is invaluable for communication educators in pursuit of the sotl. after 17 years of teaching, jan whitt (1995), an assistant professor of journalism at the university of colorado, returned to the newsroom to refresh the skills she teaches her students. whitt asserts that it is vital for educators to acquire professional experience before they teach and to maintain it throughout their careers to avoid stagnating and drifting out of touch with the changing industry. after returning to the newspaper, whitt wrote, “perhaps i have a clearer answer to the student question, ‘if the media are such exciting places to work, why did you leave?’ i now say, ‘two answers really. i love to teach, and i never really left the newsroom.’” v. the research community as fascinating as this interplay between the practice and teaching of communication might be, there’s more to the story. the model includes a third layer— research. for the sotl, the research component is important because, not coincidentally, the sotl concept is built on the idea of emulating the rigor of empiricism and research in one’s teaching pursuits. if the communication discipline is going to thrive, its research must be connected to the other parts of the discipline. but communication research doesn’t make it into the classroom nearly as often as it should (book, 1989), and it receives an even chillier reception in the professional world (pew center, 2000). what that means is that few people with a good understanding of the communication research literature are involved successfully with the other points of the model, and vice-versa. the scarcity of people with a comfortable foothold in all three points of the model means that few are truly able to claim complete understanding of the discipline. book (1989) argues that “translators” are needed to make the fruits of research more accessible. but who has the ability to do this translating and a reason for wanting to? the most feasible candidate seems to be the communication educator who knows the importance of true disciplinary understanding. in september 2000, 17 journalists and editors met for a symposium sponsored by the pew center for civic journalism. their mission was to “create new lifelines between journalists and academics.” titled “cracking the code,” the symposium generated plenty of dialogue that fits this paper’s model well. the ideas put forth during the “cracking the code” symposium centered m. seamon journal of the scholarship of teaching and learning, vol. 5, no. 1, january 2005. 51 around trying to foster relationships between the two camps. however, most such relations have proved to be contrived and short-lived, perhaps because none was conceived around the one motive for interconnection that lasts—true mastery of all points within the discipline. phil meyer, professor of journalism at the university of north carolina, asserts that academics and journalists don’t connect well because they have different priorities. “the newspaper business is now a business that doesn’t want to go to much trouble to do stuff that doesn’t have an immediate payoff. that’s why they don’t connect with academics, because we think in the very long term and they think in the very near term, and our horizons are just too different” (pew center, 2000). meyer is correct to point out that, individually, researchers and professional practitioners will have very different horizons if the extent of their focus is on their respective corners of the discipline. only someone whose goal is mastery of the entire discipline is likely to aspire to long-term membership in both camps, but that is exactly what is needed to fully inform one’s teaching as a communication educator. another way of winning acceptance for research in the newsroom is to begin with students who are studying communication. when students are familiarized with research, the benefits include a greater awareness of what academic research is all about and how communication theory is developed and tested. too many undergraduate students finish their academic careers without ever knowing that the research community exists. the idea that new knowledge is generated through scholarly inquiry is a foreign concept to them. they see “research” as looking something up in the encyclopedia, not as the production and testing of novel hypotheses. if communication educators were plugged into the discipline’s research community, such an oversight could be avoided. when exposed to the research mindset by an educator who embraces the sotl, communication students benefit from improved critical thinking skills, a better awareness of how data can improve certain news stories and how to separate scientifically valid research from junk polls and bad science. these basic research skills are important to working journalists as well, but the value of theory-based academic research for the newsroom is a tougher sell, in part because it is greatly misunderstood (pew center, 2000). the value of complex, theory-based communication research does not lie in some utilitarian application that a copy editor can make use of in a jam, but rather in exploring the state of the discipline, making predictions about it, and, to some extent, shaping its future. communication educators who understand and embrace this component of the sotl will be far better able to impart the benefits of academic research in their classrooms than will their colleagues who do not. this is because they have a more thorough conceptual grasp of the entire discipline and thus can better synthesize it for their students. it allows the development of sophisticated repertoires for engaging the subject matter (walls, 1999). knowing the research literature will enable communication faculty to synthesize the discipline, making rich inferences and connections that would be impossible for those without such a familiarity. only those teachers who have this mastery of the research literature and who can make it accessible to students will truly have the full view of what communication education should be. it is they who are equipped to implement the sotl. vi. connecting the model if drawn as a model, the three elements of the communication discipline (education, research, and professional practice) form a triangle with a series of two-way connections among each of its points. the term “triangulation” describes using two other known fixed points to put m. seamon journal of the scholarship of teaching and learning, vol. 5, no. 1, january 2005. 52 one’s own position into context. when thought of in this way, the model proposed here allows communication educators to put their teaching into proper disciplinary context because they are aware of how the professional and research communities relate to their classroom. dinham (1996) describes this pedagogical view of “context” by saying that teaching must not be conducted in a vacuum. instead, it must be placed in the context of a “larger curriculum.” in this case, that larger curriculum can be seen as the communication discipline’s three communities. it is possible to be a teacher, a practitioner, and a researcher all separately, but perhaps communication education is best served when these roles are blended with one another in mind. only someone with an intimate, working knowledge of each can claim true disciplinary understanding. a communication educator who pursues that goal will surely find the sotl along the way. references bolding, j. (1996). research skills instruction in undergraduate research programs. journalism educator, 51, 15-22. book, c.l. (1989). communication education: pedagogical content knowledge needed. communication education, 38, 315-321. burgoon, j.k., bernstein, j.m. & burgoon, m. (1983). public and journalist perceptions of newspaper functions. newspaper research journal, 9, 41-47. cohen, j. (1997). learning the scholarship of teaching in doctorate-granting institutions. journalism & mass communication educator, 51, 27-38. cohen, j., barton, r. & fast, a. (1999). the growth of the scholarship of teaching in doctoral programs. journalism & mass communication educator, 54, 4-13. dinham, s. (1996). what college teachers need to know. in menges & weimer (eds.), teaching on solid ground. pp. 297-313. san francisco: jossey-bass publishers. duhe, s.f., & zukowski, l.a. (1997). radio-tv journalism curriculum: first jobs and career preparation. journalism educator, 52, 4-15 eggen. p. & kauchak, d. (1999). educational psychology, windows on classrooms. upper saddle river, nj: prentice hall. freedom forum, (1994). no train, no gain. continuing training for newspaper journalists in the 1990s. freedom forum: arlington, va. gaunt, p. (1992). making the newsmakers. international handbook on journalism training. greenwood press: westport, ct. 1991 journalism career and scholarship guide. (1990). princeton, n.j.: the dow jones newspapers fund inc. as cited in gaunt, making the newsmakers. oregon report (1987). planning for curricular change in journalism education: project on the future of journalism and mass communication education. eugene: university of oregon pew research center for the people and the press. (1999). striking the balance: audience interests, business pressures, and journalists’ values. views of the press on their performance and the people. pew press: washington, d.c. m. seamon journal of the scholarship of teaching and learning, vol. 5, no. 1, january 2005. 53 pew center for civic journalism & reilly center for media and public affairs at the louisiana state university manship school of mass communication. (2000). cracking the code. creating news lifelines between journalists and academics pitts, b.j. (1987). the news selection process of student and professional journalists. newspaper research journal. riffe, d., hedgepeth, j.k. & ziesenis, e.b. (1992). the influence of journals on curriculum and instruction. journalism educator, 47, 54-60. shulman, l.s. (1986). those who understand: knowledge growth in teaching. educational researcher, 15, 4-14. shulman, l.s. (1997). professing the liberal arts. in r. orill (ed.), education and democracy. pp. 151-173. new york: the college board. sprague, j. (1992). expanding the research agenda for instructional communication: raising some unasked questions. communication education, 41, 1-25. walls, r.t. (1999). psychological foundations of learning. morgantown, wv: west virginia university. weaver, d. & wilhoit, c.g. (1998). a profile of jmc educators: traits, attitudes and values. journalism educator, 43, 4-41. whitt, j. (1995). exiting the ivory tower to revisit the newsroom. journalism educator, 50, 86-91. 2037-8264-5-ce journal of the scholarship of teaching and learning, vol. 13, no. 3, august 2013, pp. 134 – 155. who says: “no fair!”? what personality and an experiment in educational value tell us about perceptions of costs and benefits of research pool requirements lisa demarni cromer1, shannon m. reynolds, and mitchell d. johnson abstract: human subject pools (hsps) are the basis for much psychological research. there is an explicit assumption that participants receive benefits from their participation, however there is little empirical research about the costs/benefits of participation. we conducted two studies with undergraduate psychology students to evaluate factors that can affect the cost/benefit ratio. study 1 (n=46) examined big five personality characteristics and number of psychology courses taken, in relation to perceived benefits. there were depreciating returns for on-going participation but no personality differences in ratings. study 2 (n=50) used a quasi-experimental design to manipulate educational value. half of the participants completed an educational assignment that integrated their hsp research experience into course material. students who completed the educational assignment had a strong sense of contributing to scientific knowledge whereas students who had no such assignment did not. implications for increasing educational value in hsps are discussed. keywords: human subject pool, educational value, integrative assignment, student perceptions of research i. introduction. human subject pools (hsps) are a valued resource for psychological research. they are used by three quarters of universities (miller, 1981; sieber & saks, 1989) and one third of four-year institutions (landrum & chastain, 1999). approximately 70% of personality and social psychology studies and 90% of perception studies are conducted with college students in hsps (kulich, seldon, richardson, & servies, 1978). given the widespread use of hsps, studies about the costs and benefits of hsp research participants’ experiences have widespread applicability to psychological research and ethics in the united states. the relatively small body of research about hsps has focused on researchers’ concerns. hsp meta-research has sought to identify idiographic differences among participation characteristics. for example, meta-research has focused on individual differences that relate to predictors of early or late participation in the semester, finding that women tend to participate earlier in the semester than do men (aviv et al., 2002; cooper, baumgardner, & strathman, 1991; roman et al., 1995; witt, donnellen, & orlando, 2011). research into personality aspects of participation has found that participants higher in conscientiousness, agreeableness (witt, donnellen, & orlando, 2011) and introversion (aviv et al., 2002) participate earlier in the semester. some studies have examined pragmatic obstacles to research participation from participants’ perspectives. other research has focused on factors related to volunteerism, suggesting that there are individual differences in who may be more likely to see the benefit of 1 department of psychology, the university of psychology, 800 s. tucker drive, tulsa, ok 74104, lisa-cromer@utulsa.edu cromer, l.d., reynolds, s.m., and johnson, m.d. journal of the scholarship of teaching and learning, vol. 13, no. 3, august 2013. josotl.indiana.edu 135 research participation. we summarize the literature relevant to united states research pools and then focus on the issue of costs and benefits for research participants. a. individual differences in willingness to volunteer for research. presumably, volunteer research participants experience some value or benefit from their volunteerism. when we examine research on volunteerism in hsps, there are some individual differences between volunteers and nonvolunteers. this may illuminate who is more likely to perceive benefits from hsp participation. martin and marcuse (1958) prescreened 400 psychology undergraduates. these students were categorized into “volunteers”, i.e., they volunteered to do additional studies and “nonvolunteers,” i.e., they did not respond to subsequent research invitations. volunteers were higher on intelligence (as measured by self-report act scores) than nonvolunteers, and female volunteers were more sociable than female nonvolunteers (martin & marcuse, 1958). numerous other traits have been observed in individuals who elect to participate in research. jews are more likely to participate in interviews than are protestants and catholics (fischer & winer, 1969). willingness to volunteer in an hsp has been related to having more left-wing political views and being less conventional (rosen, 1951; rosenthal & rosnow, 1975) and to being lower on authoritarianism (rosnow & rosenthal, 1976). volunteers are more aggressive and higher in need for achievement (rosenthal & rosnow, 1975), and they are more agreeable and open to experience than are nonvolunteers (dollinger & leong, 1993). amongst volunteers, extraverts are more likely to participate in face-to face research than internet-based research (aviv et al., 2002). b. beneficence in research. the belmont report (u.s. department of health, education, and welfare: the national commission for the protection of human subjects of biomedical and behavioral research, 1979) articulated an ethical obligation of beneficence in research. in other words, researchers should work to maximize the possible benefit to research participants. this standard is also found in the american psychological association’s ethics code (american psychological association, 2002). benefits, however, may be in the eye of the beholder, and as such, perception of benefits is an important consideration for institutional review boards (irb) and others (e.g., professors) who are responsible for assuring participants’ welfare. all studies must weigh the participants’ costs and benefits; this balance is explicitly stated in irb applications. in the present research, we consider costs and benefits as they relate to participation in hsps, over and above the usual consideration of cost and benefits for any particular study. at the level of hsp participation, we consider cost to relate to factors such as possibly feeling coerced to participate, as well as negative feelings and experiences such as boredom, stress, perceptions of fairness (or unfairness) with regard to a hsp requirement or offer for extra credit. meanwhile, benefit relates to learning from the experience, satisfying curiosity, intellectual stimulation and interest, altruism, and the opportunity to earn extra credit. importantly, these are not objective measures of cost or benefit. rather, these are individuals’ subjective perceptions of their experiences. we review this scant literature on costs and benefits below. cromer, l.d., reynolds, s.m., and johnson, m.d. journal of the scholarship of teaching and learning, vol. 13, no. 3, august 2013. josotl.indiana.edu 136 c. perceptions of cost and coercion in hsp participation. some authors have suggested that coercion may be a concern about research conducted with hsps (miller & kreiner, 2008; miller, kreiner, ryan, & smith, 2010). in a 2005 survey of irbs, wille (2005 as cited in miller et al., 2010), found that a third of irbs had explicit procedural concerns in order to prevent coercion. the most recent edition of the american psychological association ethics code, standard 8.04b, addresses coerciveness in requirements for course credit, noting that it is essential that participants be given choices that include equitable alternatives to research participation (apa, 2002). in short, instructors can require students to be directly involved in research and can require their participation in person, as long as there is an equitable alternative such as reading and summarizing empirical papers (smith, 2003). implicit in this assumption is that reading about research and writing a summary is equitable in time and effort to the research requirement. some investigators have attempted to document whether the assumption of equitability is consistent with students’ experiences. foot and stanford (2004) found that students viewed these alternatives as either boring or too time-intensive compared to the research sessions. another study found that students thought research participation and summarizing articles were equally coercive (trafimow, madson, & gwizdowski, 2006). miller and kreiner (2008) found that students reported feeling coerced to participate and viewed offers of extra credit from instructors to also be coercive, but then reported the overall participation was worthwhile. additionally, the participants reported that they did not object to receiving extra credit or being paid for their research participation (miller & kreiner, 2008). the authors noted the apparent contradiction and speculated that participants may not have fully understood the word “coercion” used in the query. previous research that examined participants’ personality primarily focused on whether personality predicted differences in participants’ behavior. researchers found that personality type was related to when participants signed up for studies (early or late in the semester). only two studies examined personality as it related to perceptions of one’s hsp experience. carver (1980) found that type a (high strung) personality types perceived higher levels of coercion than did type b (easy going) personality types. the idea that perceived costs of research participation could vary by personality type was later pursued by miller and colleagues. miller et al., (2010) used a common five factor model (ffm) of personality called the revised neo personality inventory (neo pi-r). the neo pi-r measures five broad domains of personality: neuroticism, the tendency to experience negative emotions, extraversion, the tendency to be social and assertive, openness, creativeness and be willing to try new experiences, agreeableness, the ability to get along with others and be friendly, and conscientiousness, being thorough and careful (costa & mccrae, 1985). miller and colleagues (2010) used different vignettes that depicted five different recruiting strategies to research studies and had participants rate perceived level of coercion. the authors found that personality did not impact views of coerciveness. nonetheless, miller and colleagues did find a within subjects effect for type of recruitment depicted in the vignettes. the highest level of coercion was perceived in a vignette in which a professor simply asked students to participate in research. there was a lower level of perceived coercion when a professor was said to offer monetary incentives, and even less perceived coercion when a professor was said to ask students to stay after class to participate. the vignette rated least coercive was a professor offering extra credit to those who participated. from this research, it appears participants view coercion as being related to how direct and personal the requests for participation were. cromer, l.d., reynolds, s.m., and johnson, m.d. journal of the scholarship of teaching and learning, vol. 13, no. 3, august 2013. josotl.indiana.edu 137 taxing students’ time is another potential cost to an hsp requirement. when queried for criticisms of hsps, participants complained about studies being difficult to schedule (elicker, mcconnell, & hall, 2010). elicker and colleagues pointed out that the most common reason students do not participate in research was having limited time to participate followed by general scheduling conflicts. this research pointed to important considerations for reducing costs to students, including having a reasonable number of study selection options and a variety of study session days and times. d. perceptions of benefits to hsp participation. objective measures of educational benefits have shown that research participants learn from their research experiences. students’ knowledge about research methods was greater when they participated in experiments than when they studied from a textbook (darling et al., 2007; thieman, clary, olson, dauner, & ring, 2009). objective tests of knowledge-based questions also showed a better understanding of ethics after participating in research (rosell, beck, luther, goedert, shore, & anderson, 2005). nonetheless, there is no work that queries parameters for learning. in particular, it is unknown whether participants gain more knowledge with more hours of research participation (miller, 1981). perception of learning is also an important parameter to measure. the educational value of participating in research is often used to justify an hsp requirement and is the purported benefit that balances the cost of participation. furthermore, an educational component to research participation is implicit in federal regulations about human subjects’ research, and is monitored by irbs. if participants perceive educational value, it would provide support for the view that research offers benefits to participants. similar arguments have been made regarding perceptions of coercion. leak (1981) and scott-jones (2000) noted that it is not whether researchers state there is coercion but rather participants’ perceptions of coercion that should determine whether or not coercion is present. regarding perceptions of educational benefits of research, participants’ perceptions present a generally positive picture (darling, goedert, ceynar, shore, & anderson, 2007). consistent with objective studies of knowledge, subjective studies evidenced that students reported having a better understanding of research methodology and of research ethics after participating in experiments. furthermore, students reported that research participation augmented their knowledge more than equivalent hours of classroom instruction (darling et al., 2007; elliott, rice, trafimow, madson, & hipshur, 2010). in contrast to these positive impressions about research participation, brody, gluck and aragon (2010) found an absence of positive views when they interviewed 65 students about their experiences. although brody and colleagues did not probe for positives, the authors also noted that positive responses were not spontaneously offered; this suggests that students’ first impression of research may not be that it is educational for them. these studies also did not evaluate whether or not there is an optimal amount of research participation that can provide benefits. if indeed research participation is beneficial and positive for students, it would be helpful to know how much participation provides the most benefit (e.g., number of hours or studies) and whether or not there are diminishing returns after some amount of research participation. cromer, l.d., reynolds, s.m., and johnson, m.d. journal of the scholarship of teaching and learning, vol. 13, no. 3, august 2013. josotl.indiana.edu 138 e. the current study. in the present research we sought to examine the individual benefits that may be gained from hsp participation, and if the perception of benefits is correlated to the amount of participation in research. we wanted to compare responses of students who were new to the hsp compared to those who had participated over multiple semesters. this would allow us to detect whether there is a limited value in hsp participation or whether there are cumulative benefits over repeated participation. we also were interested in bringing together the individual difference literature (i.e., conservatism, and personality) with perceived level of costs (e.g., coerciveness) and benefits (e.g., educational value) of research. hsp studies (reviewed above) have shown that individual differences do relate to many hsp variables, such as when during the semester a student will participate (witt, donnellen, & orlando, 2011), which personality types are more likely to participate in any kind of research (dollinger & leong, 1993; martin & marcuse, 1958; rosen, 1951; rosenthal & rosnow, 1975), and which personalities are more likely to participate in faceto-face studies (aviv et al., 2002). however only two studies (carver, 1980; miller, 2010) have examined personality differences as they relate to perceptions of research participation. unfortunately, carver (1980) used broad type “a” and “b” personality traits, which are not well-defined personality constructs (friedman & booth-kewley, 1988; ganster, schaubroeck, sime, & mayes, 1991). although miller found no differences for personality type, the study did not unpack these analyses into specific hypotheses. because personality has been examined and found to be related to other hsp variables, there may be a possible “file drawer” problem. the “file drawer” problem was first described by rosenthal (1979). rosenthal asserted that the literature was biased because null results generally did not get published; hence information about the absence of group differences or the absence of associations between variables was not disseminated. this issue remains prevalent today, yet null findings do need to be communicated to the scientific community (howard, lau, maxwell, venter, lundy, & sweeny, 2009). given that personality differences that may relate to hsp participation is a relatively new area of inquiry, and a previous “file drawer” problem may have existed, we sought to examine personality variables as they related to different perceptions of research experiences. in particular, we were interested in whether conscientiousness and openness to experience would relate to positive or negative views of research participation, and if agreeableness would relate to the perceived educational value and fairness of required participation. to our knowledge, this is the first study that examines differences in personality as it relates to perceived educational value and fairness. personality, by definition, characterizes the way one thinks, behaves, and feels (allport, 1961), so it is extremely relevant to perceptions of costs and benefits. we were interested in extending earlier research by rosen (1951) and rosenthal and rosnow (1975) that found research volunteers held more liberal political views than nonvolunteers. both of these former studies were conducted in traditionally liberal states (authors were in california, massachusetts, and pennsylvania). given that the current research is conducted in a conservative state, we wanted to investigate the generalizability of these earlier findings; in short, we wondered if liberalism would also equate to attitudes about research when in a predominantly conservative state. hypotheses 1 and 2. as an extension of our discussion about miller’s (1981) observation that little is known about whether participants believe that they gain more knowledge with more hours of research participation, we sought to examine this relationship. our prediction was that cromer, l.d., reynolds, s.m., and johnson, m.d. journal of the scholarship of teaching and learning, vol. 13, no. 3, august 2013. josotl.indiana.edu 139 (1) perceptions of benefits would decline as research hour requirements increase. to test this, we correlated the number of psychology courses taken prior to the most recent participation, and the number of positive and negative comments that each student generated in response to openended questions about the value and experience of hsp participation. (2) we predicted that students who had more negative views of the research and hsp would have been required to complete more total research hours in the current semester. hypotheses 3 and 4. we sought to examine whether personality variables would account for individual differences in perceptions of research experiences. because only one published study (miller, 2010) sought to examine hsp experiences and personality (and failed to find differences), we thought that it would be useful to determine whether in fact individual differences could explain positive or negative experiences with research. specifically, we predicted that: (3) individuals who were higher on personality variables of openness to experience, conscientiousness, and political conservatism would be more likely to be positive (versus negative) in their views about their hsp experiences. (4) we also predicted that individuals who were higher on big five agreeableness would be more likely to rate the research requirement as being fair (versus unfair). ii. methods study 1. data were collected during one semester. students participated either for a research requirement or extra credit, as determined by their course instructor. participation was optional; this study was offered amongst several other studies. data collection was anonymous. a. procedure. participants who had completed at least two hours of research in the semester were eligible to participate. this was tracked by an online system (sonatm). sonatm is an online software product that is licensed by the psychology department. all research (online and on site) is administered through this system. two hours of research was chosen as a baseline for all participants to ensure that they had at least minimal exposure to participation before answering questions about how they viewed those experiences. they were contacted by mass email notification through the sonatm system. they were given a research assistant’s email to contact if they wished to receive a password for the online study. responses were completed online on the sonatm system affording anonymity and automatic credit granting. the survey took fewer than 30 minutes to complete. debriefing was provided online, and participants were encouraged to print this out, or contact the researcher for a paper copy. b. participants. the study was conducted at a private midwestern university. at this university, 128 students had registered in the hsp (as determined by number of accounts in the online administration system, sonatm). seventy-four participants responded to the invitation to the complete the survey and received a password. of these, 62% subsequently took the online survey (and all completed once started). one participant was excluded from analyses due to selecting an option to not be included in any publications. the final sample (n = 46) consisted of 40 women and 6 men. the mean age was 21.43 (sd = 5.60). table 1 details additional demographic information. cromer, l.d., reynolds, s.m., and johnson, m.d. journal of the scholarship of teaching and learning, vol. 13, no. 3, august 2013. josotl.indiana.edu 140 *gpa compares to a mean of 3.22 for the college of arts and science at this university, per dean’s office report. c. measures. demographic questions included gender, gpa, academic classification, academic major, and employment status. participants also were asked about the number of psychology courses they had taken, number of required research hours that semester, intention to complete extra credit research hours, and how many research hours they planned to complete as compared to number of hours reading and summarizing articles. political conservatism. political affiliation was measured with a single subjective question: “please indicate which option best describes your political affiliation” and responses were provided on an 8 point likert scale where liberal = 1, moderate = 4, and conservative = 8. big five inventory. personality was measured with the 44-item big five inventory (bfi; john, donahue, & kentle, 1991) self-report questionnaire. the bfi has five scales consistent with the five-factor model: extraversion, agreeableness, conscientiousness, neuroticism, and openness. internal consistencies for the five scales range from .79 to .87, with a mean .83 (john, donahue, & kentle, 1991; john, naumann, & soto, 2008). the internal consistencies in the current data set ranged from .69 (openness and neuroticism) to .85 (extraversion). the bfi also has a mean convergent validity correlation of .95 with the neo pi-r scales (john, naumann & soto, 2008). the bfi means for a sample (n = 6076) of 21 year-olds are: extraversion = 3.25 (sd =.90), agreeableness = 3.64 (sd = .72), conscientiousness = 3.45 (sd = .73), neuroticism = 3.32 (sd = .82), and openness = 3.92 (sd = .66; john & srivastava, 1999). five one-sample ttable 1. study 1 demographic information for the hsp (n = 46). minimum maximum mean sd age 18 53 21.43 5.60 political affiliation 1 (liberal) 7 (conservative) 3.98 1.57 year in school 1 5 2.61 1.26 gpa* 2.3 4.0 3.41 0.47 total psychology classes taken 1 36 6.43 6.29 extra credit hours 0 10 1.87 2.71 required research hours 2 8 5.87 1.98 bfi extraversion 2 5 3.32 0.78 bfi agreeableness 3 5 4.09 0.60 bfi conscientiousness 3 5 3.79 0.51 bfi neuroticism 2 4 2.91 0.63 bfi openness 2 5 3.47 0.59 female (%) 87 view hsp experience positively (%) 72 view hsp experience as fair (%) 89 view sonatm positively (%) 100 cromer, l.d., reynolds, s.m., and johnson, m.d. journal of the scholarship of teaching and learning, vol. 13, no. 3, august 2013. josotl.indiana.edu 141 tests showed our sample means statistically differed from the normative sample because our sample was higher on agreeableness (t(45) = 5.09, p = .0001, d = -.68), and conscientiousness (t(45) = 4.52, p = .0001, d = -.54), but lower on neuroticism (t(45) = 4.41, p = .0001, d = .56), and openness (t(45) = 5.17, p = .0001, d = .72). see table 1. hsp experiences questionnaire (hsp-eq). the hsp experiences questionnaire was developed by the first author in order to obtain feedback regarding students’ experiences in the hsp. the original intention for this measure was to provide evaluative information in order to learn about students’ experiences as participants and to potentially guide changes to the hsp. all questions were open-ended to allow students to express as much as they wanted about their hsp experiences. coding categories were developed by evaluating participants’ responses. coding of hsp-eq. question 1, “why did you choose to do the experiments instead of the alternative assignment? (or why did you do the alternative assignment instead of the experiments, if that applies to you)?” open-ended responses were coded into five categories: (1) it was easier, (e.g., “for me, spending hours participating in a study was easier than doing the alternative assignment” (2) contribute to research, (e.g., “i think it is important to support the research that graduate students are doing.”) (3) for educational or learning purposes (e.g., “better experience—allows you to get a different perspective,” “it's also a nice deviation from "normal," written schoolwork,” “i wanted first hand experience at what participating in psychology research was about.”), (4) it was more interesting, fun, or they were curious (e.g., “i thought participating would be more interesting,” “it’s fun to get the experience,” “i chose to participate in experience due to the curiosity.”), or (5) felt coerced (e.g., “i thought that the alternate assignment would result in a lower grade.”) a distinction we made between categories 3 and 4, was that category 3 was coded if the implied goal was to learn something and to contribute to an academic knowledge base, as compared to category 4 that suggested more of a novel component with a shorter term goal. each participant’s response could have multiple components, each of which would be coded although no single component was ever double coded. participants’ opinions of the research requirement were assessed in question 2, “what do you think of the experiment requirement?” negative responses included statements suggesting (1) fewer required hours or (2) unfair, “it’s difficult for athletes to find spare time to do it.” positive views were coded as (1) motivating the participant to become involved in research (e.g., “it’s helpful, otherwise i wouldn’t be motivated to get involved at all”), (2) educational (e.g., “it's nice to learn the research side of psychology, and to get to see the tests that we talk about in class.”), or (3) generally positive, such as “it was okay” and “i liked it.” coders only provided one code for each participant because nearly all of the participants only provided a single response. for the few cases in which more than one response was given, we coded the first response on this question. thus, for this question, response category frequencies are independent and dichotomous. these scores were included in the overall summed score of responses, and question 2 was also used as an stand alone dependent variable in the logistic regression used to test hypothesis 2. question 3, “what do you think is valuable about participating in research studies?” was coded as (1) no value, (2) educational (e.g., “learning how the process works, e.g. informed consent, debriefing, etc.” or (3) contribute to research (e.g., “it provides data for professor's research.”). frequencies for these items do not sum to 100% because of the double coding and therefore are not independent, thus only descriptive statistics (frequencies) are provided. no cromer, l.d., reynolds, s.m., and johnson, m.d. journal of the scholarship of teaching and learning, vol. 13, no. 3, august 2013. josotl.indiana.edu 142 single portion of a statement was ever double coded, however a compound sentence that touched on two categories, would have each part of the sentence coded separately. question 4 queried dislikes of the research requirement: “is there anything about the research/experiment requirement that you do not [sic] like?” responses included: (1) too few experiment choices, (2) too many hour requirements, (3) experiments took too long or were difficult to schedule, or (4) general complaints (e.g., “some of the studies are lengthy and monotonous”).the modal response was (5) no dislikes; when this question was left blank it was also coded as “no dislikes.” question 5a, “is the experiment requirement fair?” was coded yes or no. question 5b “why or why not?” was coded (1) educational, (2) contributes to research, or (3) that it was a reasonable requirement, (e.g., “because it's just like a participation credit in any other class, and students are given ample time to complete it”). responses explaining why the experiment requirement was unfair included statements that (1) it was educational for others but not me, (2) the participation should be extra credit, (3) unfair study exclusion factors or too few options, and (4) exploitative or too demanding (e.g., “other departments don’t require it and as a undergrad student we don’t get any results therefore for us it seems pointless”). question 6 assessed the participant’s opinion of his/her debriefing experience. responses towards the debriefing were coded as (1) positive/helpful/necessary, (2) negative/not helpful/unnecessary, (3) no debriefing experience, or (4) a neutral statement (e.g., “it was brief and to the point.”) question 7 assessed the participants’ experience with the electronic sign-up system. this was coded as either positive (e.g., “i liked it a lot more than signing up on paper, it is fast, easy, and instant”) or negative. however, no negative responses were given. the faculty investigator, graduate student, and undergraduate student were all trained on the coding method. the graduate and undergraduate student independently coded all the participants’ responses. coding meetings were held weekly to discuss and resolve discrepancies. the primary investigator served as judge for discrepancies, so the final code reflected her agreement with one of the two coders. the two independent raters agreed 96% of the time. across all of the open ended questions, we summed the number of codeable positive responses in order to obtain a measure of how positive respondents were. the range was 2 to 9 for each participant. we also summed the number of codeable negative responses in order to obtain a measure of how negative respondents were. the range of values was 0 to 5 for each participant. all respondents had a summed positive and a summed negative score. these scores were negatively correlated (pearson’s r = -.684, p = .0001), indicating a strong relationship with the variables, yet not so correlated as to be redundant. the more positive respondents were about their hsp experience, the fewer negative things they had to say. iii. results from study 1. a. descriptive statistics. seventy-two percent of participants thought the research experience was educational or positive. most respondents (89%) thought the research requirement was fair. although 41% reported liking everything about the research requirements, the most common complaint was that the sessions were hard to fit into their schedules (26%), followed by 11% of the sample not liking an aspect of one of the experiments or feeling coerced into participating. while 52.2% of the sample cromer, l.d., reynolds, s.m., and johnson, m.d. journal of the scholarship of teaching and learning, vol. 13, no. 3, august 2013. josotl.indiana.edu 143 thought the debriefing was helpful, 17.4% said it was unhelpful, and 30.4% did not report on debriefing or said they had never experienced one. to further examine how previous hsp experience and personality may influence these views of the hsp experience, we tested hypotheses 1-4 below. because the sample was 87% female, we did not include gender in any of the analyses for lack of power which could have been misleading in the results. hypothesis 1. we predicted that perceptions of benefits would decline as research hour requirements increase. we examined the number of psychology classes participants had taken at our institution and proportion of individuals who rated the experiment as educational and/or positive (see figure 1). negative comments per number of psychology classes are also shown in figure 1. figure 1. study 1: percentage of positive and negative responses across number of psychology classes taken by students. all the students who had taken only one psychology class provided a positive response to question 2, “what do you think of the experiment requirement?” in comparison, there were fewer positive responses from participants who had taken more than one class that required research participation. there were also more negative views about experiences for individuals who had taken two or more classes. as stated above, we computed a summed score of the number of positive responses for all of the questions and a summed score of the total number of negative responses for each respondent. these summed positive and negative values were correlated with the total number of reported psychology classes taken. number of psychology classes taken was used because at the institution in which this study was conducted, all psychology classes require students to complete between 1-4 hours of research experience as part of the course requirement (either reading and summarizing an article or participating in a research study). most students choose to participant in research over reading and summarizing an article. the more psychology classes a student takes, the more research experience s/he has. number of psychology classes was a better estimate than year in school because some senior students were not psychology majors, had only taken one psychology class, and only participated in one hour of research. both correlations were significant (one-tailed because we hypothesized more classes would result in more negativity and less positivity). more classes related to fewer 0   20   40   60   80   100   1  class     2-­‐4  classes     5-­‐  7  classes     8  or  more  classes     study  1:  percentages  of  pos4ve  and   nega4ve  response  across  psychology   classes  taken   posi5ve  response  (n  =  33)   nega5ve  response  (n  =  12)   cromer, l.d., reynolds, s.m., and johnson, m.d. journal of the scholarship of teaching and learning, vol. 13, no. 3, august 2013. josotl.indiana.edu 144 positive responses, pearson’s r = -.248, p = .048, and more classes related to more negative responses about the hsp, pearson’s r = .411, p = .005. hypothesis 2: to determine whether negative views of research were related to the number of required research hours across all psychology courses in which participants were currently enrolled, we performed a logistic regression, regressing participants’ views of the research experience as positive = 1 or not = 0 (based on each participant’s dichotomously coded answer to question 2), on a categorical predictor of research hours required. we coded research hours required into a categorical variable because examination of a histogram revealed that number of hours required was essentially a bimodal distribution. five or fewer hours were dummy coded 0, and 6 or more were dummy coded 1. the full model was statistically significant, χ2 (1, n = 46) = 4.70, p = .03, indicating that the model was able to distinguish between those who thought the research experience was positive and those who did not. the model as a whole explained between 9.7% (cox and snell r square) and 14% (nagelkerke r squared) of the variance in views of the research experience, and correctly classified 71.7% of the cases. the odds ratio of .193 indicated that respondents were about 20% more likely to report the research pool experience as negative if they were required to complete 6 or more hours of studies in a semester. hypothesis 3. to examine if individuals high on openness to experience, conscientiousness, and political conservatism were more likely to rate their research experience as positive (versus negative), we conducted a direct logistic regression. the model contained three independent continuous variables (openness, conscientiousness, and political conservatism). the dependent variable was a dichotomous 1 = positive, 0 = negative experience score (based on question 2). the full model was not statistically significant, χ2 (3, n = 46) = 5.66, p = .129, indicating that the model did not distinguish between individuals who found the research experience to be positive or not. see table 2 for means and standard deviations. these results may reflect the fact that the majority of the sample (72%) considered the research requirement a positive experience. hypothesis 4. to examine whether individuals who were higher on big five agreeableness were more likely to view the research requirement was fair, responses to question 5a were coded into 1 = fair, 0 = unfair. a logistic regression with agreeableness as the predictor was not statistically significant, χ2 (1, n = 46) = 1.024, p = .312. failure to find a significant difference may be due to the majority (89%) of the sample viewing the requirement as fair. table 2. study 1: means and standard deviations for positive and negative groups. group openness scale score conscientiousness scale score political affiliation mean (sd) mean (sd) mean (sd) positive group (n = 33) 3.42 (0.63) 3.72 (0.45) 3.79 (1.60) negative group (n = 12) 3.58 (0.45) 4.02 (0.60) 4.42 (1.51) cromer, l.d., reynolds, s.m., and johnson, m.d. journal of the scholarship of teaching and learning, vol. 13, no. 3, august 2013. josotl.indiana.edu 145 iv. discussion for study 1. the results indicate that participants of this hsp generally viewed the research experience as positive and fair. nonetheless, we observed depreciating returns to students as they participated in the hsp over multiple classes. it is important to note that 100% of participants who were partaking in the hsp for the first time had a positive experience in the hsp. the proportion of positive views of the hsp diminished for the other groups, with 69% in 2-4 classes, 75% in 5-7 classes, and 63% in 8 or more classes having positive ratings. there were increased negative views across more hsp participation, with 31% in 2-4 classes, 25% in 5-7 classes and 38% in 8 or more classes reporting negative views of hsp experiences. both trends (increasing negativity and decreasing positivity about hsp experiences) were significant when correlated to number of psychology classes taken. this provides evidence that there are educational benefits to students who participate for the first time. this statistically significant finding speaks to the strength of the effect because our sample was small, yet differences still emerged. regarding the question about whether students continue to receive benefits from higher levels of hsp participation in a given semester, some but not all students reported benefits and these benefits were reduced with increase exposure to research studies. when we dichotomously coded participants into requirements for 6 or more hours compared to 5 or fewer hours, we found that those with higher demands were less likely to view their experience as positive (hypothesis 2). from a cost/benefit perspective, this suggests that upper division classes that require hsp participation may need to augment the hsp experience in order to maintain a high level of educational benefit for students. alternatively, departments may consider setting a ceiling on the number of research hours that are required of students in a given semester. some of the variability in responses may have to do with the different types of research in which students were able to participate. at the university where this research was conducted, there are several different types of research, and so it is possible that some students’ benefits were augmented if they participated in different kinds of studies over time (e.g., personality, psycho-physiological, trauma), exposing participants to different methodologies and different content areas of psychology. despite the increase in negative views, it is important to note that overall participants continued to report positive views of their participation (that outweighed the negative views). as an anonymous reviewer pointed out, it may be that participants’ increasing experience in human subjects research and their additional courses in psychology may have taught them critical thinking skills that enabled them to see these experiences in a more balanced light. personality did not relate to attitudes about hsp participation. this finding was consistent with miller (2010) who also noted that personality did not relate to perceptions of research experience. this suggests that student reports about research experiences being positive, fair, or valuable, is not driven by a personality difference of being more agreeable or conscientious; rather, it suggests that there is a general positive experience for most students who participate in research. in contrast to previous work (rosen, 1951; rosenthal & rosnow, 1976), we did not find that political affiliation related to perceptions of hsp experiences. this may be related to the current sample being in the moderate range on a self-report measure of conservatism. rosen (1951) used a fascism scale, and in rosenthal and rosnow (1976) authoritarianism was the construct discussed. the measure in the current study may not have elicited as strong responses as a fascism or authoritarianism scale, which may have limited sample variance. the absence of significant differences is often regarded as a non-finding in research and can result in a “file drawer” problem. we include it here because it is noteworthy cromer, l.d., reynolds, s.m., and johnson, m.d. journal of the scholarship of teaching and learning, vol. 13, no. 3, august 2013. josotl.indiana.edu 146 that personality and individual differences may not have a large impact on attitudes toward hsps, suggesting that variance in attitudes has more to do with different experiences (under the control of hsp administrators and researchers) rather than individual differences. in effect, this can put the onus on the hsp administrators and experimenters to ensure that all participants receive benefits. for example, a department’s hsp may incorporate an educational assignment that encourages integration of the research experience into course material. the value of an integrative assignment that relates the hsp participant experience to course material is the focus of study 2 below. v. study 2. broadly speaking, when surveyed, hsp participants appear to feel satisfied with their experiences and they find participation to be educational. nonetheless, participants who are new to hsp experiences rate their experiences more positively than do participants who have repeatedly participated. possibly, the decrease in favorable perceptions could be explained by educational value, in that those with previous experience have less to learn from additional participation. it follows then, that if we could increase educational value of hsp participation, positive ratings would also increase. in the present study, we used a quasi-experimental design in order to test the relationship of attitudes toward hsp participation and educational value, by enhancing educational value of research participation for one group. a. intentionally educating about research. the question of whether hsp participation is of implicit educational value, or whether educational value can be explicitly enhanced, is empirical. one suggestion for increasing the educational value of hsp participation was to incorporate an integrative assignment into the research experience (richardson, pegalis, & britton, 1992). richardson and colleagues (1992) administered an integrative assignment after each research session. the assignment was five brief questions about the research that asked, “(a) the area of psychology relevant to the study, (b) data collection techniques used (e.g. survey, interview, observation), (c) potential practical applications of findings, and (e) relation of the study to course material” (p. 12). they then gave a questionnaire assessing all students’ perceptions of the overall education benefit of the research sessions, the benefit of the debriefing experience, how related the research was to the course, and an overall evaluation of the research experience. in this naturalistic research, they found that students who completed the integrative assignment rated the research sessions as more educational and relevant to course content. specifically, students who completed the integrative assignments thought the research sessions allowed them to better understand psychology experiments, generated more interest in the debriefing explanation, and had larger educational emphasis. this study was naturalistic and there was no experimental control for who took the integrative assignment. in the present study, we sought to replicate the richardson et al. (1992) findings using a quasi-experimental design. hypothesis 1. we predicted a significant effect for the integrative assignments, in that the group that completed the integrative assignments (group 1) would have a greater proportion of individuals who rated their experiences as positive/ educational and fair than group 2, which did not complete the integration assignments. cromer, l.d., reynolds, s.m., and johnson, m.d. journal of the scholarship of teaching and learning, vol. 13, no. 3, august 2013. josotl.indiana.edu 147 vi. methods – study 2. data were collected at the end of two semesters as part of an assessment of students’ research pool experience. because this was part of a regular educational assessment, irb approval was not sought prior to data collection (american psychological association, 2002). the university irb gave the authors permission to use these existing data after they were collected. a. procedure. data were collected in paper and pencil form, in class, for both groups. the professor gave the students the hsp-eq described in study 1, and asked them to compete it in order for her to understand their research experiences and to possibly help improve those experiences in the future. the professor then left the room while the students responded to the questions. this took about 10 minutes, and students did not receive any compensation or incentive for completing the questions. participation was voluntary as well as anonymous. b. participants. participants were enrolled in an introductory psychology class at the same private midwestern university in study 1. each student either participated in four hours of psychology research as part of their course requirement, or read and summarized four journal articles (of their choice). demographic information was not collected because it was not pertinent to the standard course evaluation. most students were white. it is a requirement that they be at least 18 years of age in order to participate in research studies (age of consent). students who were not of the age to consent read and summarized the four journal articles for their course requirement. for group 1 the class composition was: 35.4% freshmen, 38.2% sophomores, 17.6 % juniors, and 8.8% seniors. for group 2 the class composition was: 36.7% freshmen, 30% sophomores, 20 % juniors, and 13.3% seniors. c. research design. participants signed up for studies on the sonatm electronic system, which could be accessed on and off campus at any time of the participants’ choosing. this is a quasi-experimental research design with the assignment to group 1 or 2 being naturalistic because we collected evaluation data as part of regular course evaluations. the integration activity was initially incorporated into the course to enhance learning, based on previous research (richardson, pegalis, & britton, 1992). the choice to not give the integration activity to the second group was an intentional manipulation for this study. in addition to their research experience, students in group 1 completed two assignments that were designed to help them integrate their research participation into their psychology course. the assignment was to answer four questions about the research participation. these questions were adapted from richardson et al. (1992) and were designed to help students see a didactic value to research participation. specifically, students needed to identify the name of a study they had completed, a book chapter and section that would describe the study (e.g. personality), and to think critically about whether the validity of the study was compromised due to the college student sample. responses were graded on a 10-point scale by a graduate teaching cromer, l.d., reynolds, s.m., and johnson, m.d. journal of the scholarship of teaching and learning, vol. 13, no. 3, august 2013. josotl.indiana.edu 148 assistant. the points for these two assignments accounted for 5% of students’ grades. students who opted to do alternative assignments of reading empirical studies completed these assignments based on research articles they read and summarized. for the purposes of the current investigation, the integration assignment was eliminated from the course for group 2 participants. so, although group 2 completed the same number of study hours as group 1 (or read and summarized four articles), they did not have an integrative assignment, in which they would have reflected on their research experience and incorporated it into the course material. d. measures. hsp-eq. the hsp-eq that was used in study 1 was also used in study 2. we utilized the coding scheme described in study 1 and the same two coders. coding agreement was 92%. vii. results from study 2. there were a total of 50 participants in the two groups. all but one participant chose to participate in research rather than complete the alternative assignment. the most frequently reported reason for choosing the research option was curiosity (56.6%) about what the research session would be like, followed by the research session was easier (41.5%) than the alternative assignment. the majority (77.4 %) thought the hsp experience was educational or positive, and 71.7% of the sample reported viewing the research requirement as fair. reasons why students viewed the research requirement as fair included it was a reasonable class requirement (37.7%), it was educational (15.1%), or the participation contributed to research (18.9 %). for those who said the research requirement was not fair (22.6%) the reasons included it was too demanding or exploitative (18.9%), the studies were too few in options or had unfair exclusion factors (1.9%), and finally 3.8% thought the research should be extra credit rather than a class requirement. the most frequent response to the question asking what the participant did not like about the research requirement was no complaints (37.7%), followed by having scheduling difficulties (26.4%), and then 20.8% of the sample not liking an aspect of the experiment or viewing the research requirement as exploitative. about half (49.1%) of the participants thought the debriefing was helpful, 9.4% thought it was unhelpful, 13.2% did not receive a debriefing, and another 5.7% gave a neutral comment about the debriefing (e.g., “it was short”). all participants reported positive views about sonatm. hypothesis. we predicted that group 1, which completed the integrative assignments, would be more positive in their answers about whether or not the research experiences were educational or positive compared to group 2, which did not complete the integration assignment. figure 2 shows the proportions of students who viewed the experience as positive. overall, students thought that the requirement was educational, regardless of condition. in addition, for the entire sample, 71.7% thought the research requirement was “fair” with proportions as follows: 72% of group 1 and 71.4% of group 2. cromer, l.d., reynolds, s.m., and johnson, m.d. journal of the scholarship of teaching and learning, vol. 13, no. 3, august 2013. josotl.indiana.edu 149 figure 2. study 2: proportion of students who viewed hsp as educational/positive and negative. figure 3 shows the reasons why students thought the research requirement was fair or unfair. the effect of not doing the integrative assignment (group 2) appears to be that a greater proportion of students thought the research requirement was exploitative and the belief that they contributed to science was eliminated (i.e., a loss in this as an altruistic benefit of research participation). while group 2 (no assignment) did contain the most participants who endorsed negative views (26%) of the research requirement as compared to group 1 (17% participants reported negative views), this result was not statistically significant χ2(1, n = 50) = 0.527, p = .47. we were unable to conduct chi square analyses on between group differences for reasons why the research requirement was perceived to be fair or not fair, because assumptions of the analyses were not met (minimum n=5 in each cell). viii. discussion for study 2. the second study was designed to evaluate whether educational value and overall ratings of the hsp experience could be increased by changing aspects of how hsp requirements are incorporated into the departmental curriculum. by experimentally manipulating the educational component of the research requirement, we were able to test its relationship to students’ perceptions of educational value and fairness. based on these qualitative data, there was a consistent trend that overall participants viewed their hsp experiences as positive. nonetheless, we saw a change in responses to openended questions when there is no integrative assignment. that is, more students appeared to feel exploited and none of them thought they were contributing to science. the integrative assignment asked students to determine, “if this study was published in your introductory psychology textbook, what chapter would be the best choice?” it appears that the effect of this assignment, possibly attributed to this question, was that students not only integrated their experience into what they were learning, but they gained an additional benefit of feeling like they were contributing to science. this is an important aspect of the research experience, when 0.00   10.00   20.00   30.00   40.00   50.00   60.00   70.00   80.00   90.00   100.00   posi5ve  response   nega5ve  response   study  2:  percentages  of  posi4ve  and  nega4ve   responses  for  opinions  on  hsp  experience   group  1:  assignment   group  2:  no  assignment   cromer, l.d., reynolds, s.m., and johnson, m.d. journal of the scholarship of teaching and learning, vol. 13, no. 3, august 2013. josotl.indiana.edu 150 weighing the costs and benefits of research – that students may gain a sense of pride, altruism, or importance for their contribution. what specifically was driving this finding will be a question for future study. figure 3. study 2: reasons why student thought research requirement was fair. viii. summary. researchers and educators are ethically bound to ensure research requirements provide an educational experience for undergraduate student participants (apa, 2002). our findings support the notion that this is occurring, and these findings are consistent with the extant literature. students generally hold positive perceptions of their research participation and consider it an educational experience (darling et al., 2007; elliott et al., 2010). study 1 contributes to the current literature by assessing the relationship between specific individual differences and demographic characteristics and students’ perceptions of fairness and educational value. we found that as students take more psychology classes, and get more experience participating in research, they retain a stable level of positive views of the hsp. however, they also gain more negative views of the hsp, which on the whole could mean that their overall views are more balanced. on the face of it, it may be that the greatest educational benefit is for students in introductory psychology courses who are getting exposed to research for the first time. on the other hand, there could be continued benefits while more balanced (i.e., also seeing the draw backs of participation) as students develop more nuanced thinking and critical thinking skills. study 2 extends the literature by assessing methods designed to increase students’ positive perceptions of hsp experience. since participants report more negative views of hsp as they participate in more research session, we attempted to enhance the educational experience of research participation for students by giving an integrative assignment that allows them to incorporate their research experience with textbook knowledge. we found that overall positive 0.00   10.00   20.00   30.00   40.00   50.00   60.00   study  2:  reasons  why  par4cipants  thought   research  requirement  was  fair  or  not  (%)   group  1:  assignment   group  2:  no  assignment   cromer, l.d., reynolds, s.m., and johnson, m.d. journal of the scholarship of teaching and learning, vol. 13, no. 3, august 2013. josotl.indiana.edu 151 and educational views were not substantially different with the addition of the integrative assignment. thus, general ‘happiness’ with participation may not change when an assignment is added. nonetheless, it is noteworthy that the group which did an integrative assignment believed that they were contributing to science whereas the other group did not. also, when participants completed the integrative assignment, they less frequently expressed views of having been exploited in the research. ix. limitations and future directions. there are several strengths to the current set of studies. we were able to extend the literature by examining questions related to hsp experiences that have not yet been conducted. it is a strength that we were able to use a quasi-experimental design to examine the impact of an educational assignment. there are several limitations to the current work. because this research was conducted at a small, private college, we had limited power; interesting trends that were not statistically significant in this study may have been significant with a larger sample. furthermore, the sample in study 1 was predominantly female, and subsequent research could benefit for examining gender differences in research experiences. also, because this is a private college in a conservative state with incoming freshman act scores averaging 28 points and higher, we potentially have a limited sample in terms of variability of responses. additionally, our sample statistically differed from the normative sample on four of the five personality variables. our sample was more agreeable and conscientious and less neurotic and open to experience, which could contribute to the positive attitudes and perceptions of the hsp. it is possible that larger schools with a more diverse student body would have more variance in student responses. because evaluating students’ experiences of hsp is a relatively new area of research, we used free-response questions. the limitation here is that students may well have had additional views –both positive and negative – that would be expressed if prompted. we were limited in coding responses to non-leading questions that were designed to provide students an unrestricted opportunity to express their opinions of the research requirement. the results of our study illustrate the need for a measure that can accurately capture the fine nuances and changes in students’ opinions. given that the use of hsps is widespread in the united states and given the ethical obligation psychologists have to positive benefit/cost ratios for participants, studies of this nature are important. moyer and franklin (2011) noted that psychology departments typically do not evaluate the educational benefit to participants. the potential for educational benefits and good attitudes from participants is an area that could have pervasive benefits for psychological research in general. it is possible (although as yet, not empirically documented) that increasing the positive attitudes of participants could decrease the costs of administering hsps. there could be fewer email complaints, few no-shows, and generally easier administration if participants’ experiences are positive. furthermore, we hope that studies in the future will attempt to relate the feeling of contributing to science to the overall quality of data that is collected by researchers using the hsp. for example, we would predict that there would be fewer invalidated subjects’ data when participants feel responsibility of contributing to science and this perception could increase motivation to follow instructions and to do one’s best to meet study demands. cromer, l.d., reynolds, s.m., and johnson, m.d. journal of the scholarship of teaching and learning, vol. 13, no. 3, august 2013. josotl.indiana.edu 152 acknowledgements thank you to the dynamics lab at the university of oregon for feedback on a presentation of this data which resulted in helpful suggestions for data coding and analysis. we are grateful to dr. amy criss at syracuse university for helpful comments on an earlier version of this manuscript. references allport, g. w. (1961). patterns and growth in personality. new york, ny: hold, rinehart & winston. american psychological association. (2002). american psychological association ethical principles of psychologists and code of conduct. retrieved from http://www.apa.org/ethics/code2002.html aviv, a. l., zelenski, j. m., rallo, l., & larsen, r. j. (2002). whocomeswhen: personality differences in early and later participation in a university subject pool. personality & individual differences, 33, 487-496. brody, j. l., gluck, j. p., & aragon, a. s. (2000). participants' understanding of the process of psychological research: debriefing. ethics & behavior, 10(1), 13-25. doi: 10.1207/s15327019eb1001_2. cooper, h., baumgardner, a. h., & strathman, a. (1991). do students with different characteristics take part in psychology experiments at different times of the semester? journal of personality, 59, 109–127. costa, p.t., & mccrae, r. r. (1985). the neo personality inventory manual. odessa, fl: psychological assessment resources. darling, j., goedert, k., ceynar, m., shore, w., & anderson, d. (2007). learning about the means to the end: what us introductory psychology students report about experimental participation. psychology learning & teaching, 6(2), 91-97. doi: 10.2304/plat.2007.6.2.91 dollinger, s. j., & leong, f. t. l. (1993). volunteer bias and the five-factor model. journal of psychology, 127, 29-36. elicker, j. d., mcconnell, n. l., & hall, r. j. (2010). research participation for course credit in introduction to psychology: why don't people participate? teaching of psychology, 37(3), 183185. doi: 10.1080/00986283.2010.488521 elliott, l. j., rice, s., trafimow, d., madson, l., & hipshur, m. f. (2010). research participation versus classroom lecture: a comparison of student learning. teaching of psychology, 37(2), 129-131. doi: 10.1080/00986281003626862 cromer, l.d., reynolds, s.m., and johnson, m.d. journal of the scholarship of teaching and learning, vol. 13, no. 3, august 2013. josotl.indiana.edu 153 fischer, e. h., & winer, d. (1969). participation in psychological research: relation to birth order and demographic factors. journal of consulting and clinical psychology, 33(5), 610-613. doi: 10.1037/h0028112 foot, h., & sanford, a. (2004). the use and abuse of student participants. the psychologist, 17(5), 256-259. friedman, h. s., & booth-kewley, s. (1988). validity of the type a construct: a reprise. psychological bulletin, 104(3), 381-384. doi: 10.1037/0033-2909.104.3.381 ganster, d. c., schaubroeck, j., sime, w. e., & mayes, b. t. (1991). the nomological validity of the type a personality among employed adults. journal of applied psychology, 76(1), 143168. doi: 10.1037/0021-9010.76.1.143 howard, g. s., lau, m. y., maxwell, s. e., venter, a., lundy, r., & sweeny, r. m. (2009). do research literatures give correct answer? review of general psychology, 13, 116-121. john, o. p., naumann, l. p., & soto, c. j. (2008). paradigm shift to the integrative big-five trait taxonomy: history, measurement, and conceptual issues. in o. p. john, r. w. robins, & l. a. pervin (eds.), handbook of personality: theory and research (pp. 114-158). new york, ny: guilford press. john, o. p., & srivastava, s. (1999). the big five trait taxonomy: history, measurement, and theoretical perspectives. in l. a. pervin, o. p. john, l. a. pervin, o. p. john (eds.), handbook of personality: theory and research (2nd ed.) (pp. 102-138). new york, ny us: guilford press. kulich, r. j., seldon, j. w., richardson, k., & servies, s. (1978). frequency of employing undergraduate samples in psychological researchand subject reaction to forced participation. paper presented at the meeting of the midwest psychological association, chicago. landrum, r. e., & chastain, g. d. (1999). subject pool policies in undergraduate-only departments: results from a nationwide survey. in g. d. chastain & r. e. landrum (eds.), protecting human subjects: departmental subject pools and institutional review boards. (pp. 2542). washington, dc us: american psychological association. leak, g. k. (1981). student perception of coercion and value from participation in psychological research. teaching of psychology, 8(3), 147-149. doi: 10.1207/s15328023top0803_4 martin, r. m., & marcuse, f. l. (1958). characteristics of volunteers and nonvolunteers in psychological experimentation. journal of consulting psychology, 22(6), 475-479. doi: 10.1037/h0041496 miller, a. (1981). a survey of introductory psychology subject pool practices among leading universities. teaching of psychology, 8(4), 211-213. doi: 10.1207/s15328023top0804_4 cromer, l.d., reynolds, s.m., and johnson, m.d. journal of the scholarship of teaching and learning, vol. 13, no. 3, august 2013. josotl.indiana.edu 154 miller, w. e., & kreiner, d. s. (2008). student perception of coercion to participate in psychological research. north american journal of psychology, 10(1), 53-64. miller, w. e., kreiner, d. s., ryan, j. j., & smith, j. w. (2010). personality types and perceived coercion in psychological research participation. individual differences research, 8(3), 140-150. richardson, d. r., pegalis, l., & britton, b. (1992). a technique for enhancing the value of research participation. contemporary social psychology, 16 (1/2), 11-13. roman, r. j., moskowitz, g. b., stein, m. i., & eisenberg, r. f. (1995). individual differences in experiment participation: structure, autonomy, and the time of the semester. journal of personality, 63, 113–138. rosell, m. c., beck, d. m., luther, k. e., goedert, k. m., shore, w. j., & anderson, d. d. (2005). the pedagogical value of experimental participation paired with course content. teaching of psychology, 32(2), 95-99. doi: 10.1207/s15328023top3202_3 rosen, e. (1951). differences between volunteers and non-volunteers for psychological studies. journal of applied psychology, 35(3), 185-193. doi: 10.1037/h0061411 rosenthal, r., & rosnow, r. (1975). the volunteer subject. new york: wiley. rosnow, r. l., & rosenthal, r. (1976). the volunteer subject revisited. australian journal of psychology, 28(2), 97-108. doi: 10.1080/00049537608255268 scott-jones, d. (2000). recruitment of research participants. in b. d. sales & s. folkman (eds.), ethics in research with human participants. (pp. 27-34). washington, dc us: american psychological association. sieber, j. e., & saks, m. j. (1989). a census of subject pool characteristics and policies. american psychologist, 44(7), 1053-1061. doi: 10.1037/0003-066x.44.7.1053 smith, d. ( 2003). five principles for research ethics. monitor on psychology, 34. retrieved from http://apa.org/monitor/jan03/principles.aspx thieman, t. j., clary, e. g., olson, a. m., dauner, r. c., & ring, e. e. (2009). introducing students to psychological research: general psychology as a laboratory course. teaching of psychology, 36(3), 160-168. doi: 10.1080/00986280902959994 trafimow, d., madson, l., & gwizdowski, l. (2006). introductory psychology students' perceptions of alternatives to research participation. teaching of psychology, 33(4), 247-249. doi: 10.1207/s15328023top3304_7 cromer, l.d., reynolds, s.m., and johnson, m.d. journal of the scholarship of teaching and learning, vol. 13, no. 3, august 2013. josotl.indiana.edu 155 u.s. department of health, education, and welfare: the national commission for the protection of human subjects of biomedical and behavioral research. (1979, april). the belmont report: ethical principles and guidelines for the protection of human subjects of research. retrieved january 3, 2008, from http://www.hhs.gov/ohrp/humansubjects/guidance /belmont.htm witt, e. a., donnellan, m. b., & orlando, m. j. (2011). timing and selection effects within a psychology subject pool: personality and sex matter. personality and individual differences, 50(3), 355-359. doi: 10.1016/j.paid.2010.10.019 503 service temporarily unavailable 503 service temporarily unavailable nginx/1.14.1 503 service temporarily unavailable 503 service temporarily unavailable nginx/1.14.1 503 service temporarily unavailable 503 service temporarily unavailable nginx/1.14.1 v4n1fife journal of scholarship of teaching and learning, vol. 4, no. 1, may, 2004, pp. 1 – 12. reexamining carnegie research institutions: evidence from ipeds data brian l. fife and joseph losco1 abstract: officials at the carnegie foundation for the advancement of teaching changed the system that had been used to classify institutions of higher education in 2000. part of the redesign was a change in the criteria for placement of doctoral degree granting institutions. especially noteworthy was the removal of specific levels of research funding as a distinguishing feature of commitment to research. do the new criteria accurately capture differences in priorities as measured by expenditures across all educational categories? are there significant differences in overall patterns of spending that distinguish the types of graduate programs categorized by the new instrument? this research effort is an attempt to address these questions by examining expenditures in three major categories of university operations–research, instruction, and public service. keywords: research, instruction, public service, carnegie foundation i. introduction. the carnegie classification of institutions of higher education was developed in 1971 and initially published in 1973 under the leadership of clark kerr to provide a portrait of the diversity of higher education in america and to enable researchers to make meaningful comparisons about educational performance across a range of similar institutions. the classification was updated in 1976, 1987, and structurally revised in 1994. by the late 1990s, the foundation members identified significant problems with the categories utilized. these included the tendency by institutions to view the classification as a hierarchy in which they would compete to “move up” the ladder, and the more systemic concern that the classification matrix imposed an external framework that failed to capture the institutions’ own focus or mission. in 1997, lee shulman, president of the foundation, convened a group of scholars to re-conceptualize the scheme. reconfiguration was to be completed in two stages. first, a reclassification was advanced in 2000 to include updated data and a consolidation of some categories. second, re-conceptualization and additional data collection would continue until 2005 at which time the foundation would provide a more “sophisticated, adaptive set of tools that allows users to cluster information in several different ways...[and to]...provide a series of lenses through which to examine and analyze institutional mission and other important differences among institutions” (carnegie foundation for the advancement of teaching, 2001, p.viii). under the original classification, five broad categories were employed: doctoral-granting institutions, comprehensive universities and colleges, liberal arts colleges, two-year colleges and institutes, and professional schools and specialized institutions. the scheme was modified in 1994 to include doctorate-granting institutions, master’s (comprehensive) colleges and universities, baccalaureate colleges, associate of arts colleges, specialized institutions, and tribal colleges and universities (carnegie foundation for the advancement of teaching, 2001, p.10). in 1973, doctoral institutions were divided into four categories: research universities i and ii and doctoral-granting 1 brian l. fife (fifeb@ipfw.edu) is professor of public affairs at indiana university-purdue university fort wayne. joseph losco (jlosco@gw.bsu.edu) is professor of political science at ball state university. fife, b. l. and losco, j. journal of scholarship of teaching and learning, vol. 4., n. 1, may, 2004. 2 universities i and ii. research universities were ranked not only on the basis of degree conferral but also on the institution’s relative ranking in attracting federal research funds. the top 100 institutions receiving federal money were simply split with the top 50 inserted into category i and the remainder in category ii (carnegie foundation for the advancement of teaching, 2001, pp.10-12). in 1987, research universities were distinguished not on the basis of relative rank in funding received but on the basis of an assigned threshold of research support. category i institutions received at least $33.5 million and category ii institutions secured between $12.5 and $33.5 million annually. these thresholds were modestly raised in 1994 (carnegie foundation for the advancement of teaching, 2001, p.12). the 2000 classification scheme has undergone two substantial changes. first, the subcategories of doctoral-granting institutions was reduced from 4 to 2 and labeled doctoral/research universities–extensive and doctoral/research universities–intensive. second, a new category of baccalaureate colleges was established (baccalaureate/associate’s colleges), continuing in the mode of emphasizing degrees rather than broader functional categorization. importantly, however, foundation scholars changed the measures they utilized for inclusion in some categories. a comparison of carnegie classifications for 1973, 1994, and 2000 is provided in table 1. the doctoral/research category is the primary focus of this paper, and the two categories in the 2000 classification are doctoral/research extensive and doctoral/research intensive. the foundation evaluators deleted the criteria of federal funding for research because they found it to be variable and unreliable from year to year, and because the funding criteria tended to favor institutions specializing in scientific and technical research rather than the humanities and social sciences (carnegie foundation for the advancement of teaching, 2001, p.14). instead, the new categorization relies primarily on number of doctoral degrees awarded and number of disciplines represented (i.e., 50 or more doctorates were awarded per year across at least 15 disciplines in the institutions in the doctoral extensive category; at least 10 doctorates per year across 3 or more disciplines, or at least 20 doctorates were awarded per year in the institutions represented in the doctoral intensive category). the foundation officials determined that since doctoral education emphasizes research, that the number of degrees serves as an appropriate proxy for research spending (carnegie foundation for the advancement of teaching, 2001, p.27). the foundation analysts offer some evidence in this regard by comparing newly categorized institutions with national science foundation (nsf) survey data related to federal science and technology funding and to self-reports of federal and nonfederal research expenditures at a subset of institutions (carnegie foundation for the advancement of teaching, 2001, pp.28-29). as the foundation leaders hypothesized, research expenditures among the doctoral extensive institutions far exceed that of their doctoral intensive counterparts. nevertheless, foundation findings with regard to categorizing research institutions are not definitive. the classifications do not necessarily account for an institution’s own priorities with regard to its educational mission. better data does exist for analyzing an institution’s commitment to research. use of such data and the type of analysis provided below may move the carnegie foundation closer to its goal of incorporating the institution’s own sense of mission (as indicated by commitment to expenditure categories) in future classifications. ii. research justification. this investigation affords instructors a foundation of knowledge that is essential in understanding a complex array of institutions in the united states. how can an informed discourse of teaching and learning perpetuate without at least some rudimentary understanding of the importance of resource allocations to instruction, research, and service, the three primary missions within the academy? an enlightened discussion about values and priorities concerning teaching, research, and service will fife, b. l. and losco, j. journal of scholarship of teaching and learning, vol. 4., n. 1, may, 2004. 3 hopefully ensue. for example, should certain institutions spend more, or less, on these fundamental objectives? are instruction and research mutually exclusive or complementary, i.e., does more research investment result in lower spending on instruction or enhance the teaching mission as the knowledge base of the instructor/researcher has been increased by some exponent? higher education instructors should endorse the very ideal that many promote to their own students in the classroom. by way of illustration, public budgeting students are taught that understanding resource allocation in the public sector is akin to good citizenship. this is why students regularly scrutinize federal, state, and local budgets. after all, how can citizens hold elected officials accountable without knowing how they reallocate resources in a complex world? many in education, the social sciences, and humanities subscribe to the holmesian notion (holmes, jr., 1897) that the essence to understanding the world is theory. ideas have been much more central to social and political change than military power. yet policies have a definitive impact on what we can aspire to accomplish as academicians. a greater understanding of the policy realities of today will help us all work toward a more utopian ideal of the future when it pertains to defining and refining the process of evaluating excellence in teaching and learning. iii. the data. the national center for education statistics (nces) of the u.s. department of education receives annual data from all postsecondary institutions in the united states, covering a range of topics including student enrollments, institutional revenues, institutional expenditures, faculty salaries, completions of programs, and demographic characteristics. these surveys are encompassed by the integrated postsecondary education data system (ipeds) (u.s. department of education, 2003). for this evaluation, the finance survey is utilized to obtain a measure of policy commitment to instruction, research, and public service. the finance survey includes data on expenditures in fourteen categories: total current expenditures and transfers; instruction; research; public service; academic support; student services; institutional support; operation and maintenance of the physical plant; scholarships and fellowships; mandatory transfers; nonmandatory transfers; auxiliary enterprises; hospitals; and independent operations (u.s. department of education, 2003; losco and fife, 2000). the survey has cross-institutional comparative utility as it allows researchers to determine the level of spending (i.e., policy commitment) to each category. the 1999/2000 academic year finance survey is utilized in this evaluation. bear in mind that incrementalism is typically the dominant budgetary mode for most institutions of higher education (lindbloom, 1959; losco and fife, 2000). while the primary focus of this paper is the analysis of commitment to research (relative to teaching and service) among doctoral universities, we also utilize data on master’s colleges and universities i as a point of contrast. two categories of master’s colleges and universities (i and ii) are included in the 2000 classification. the master’s colleges and universities i institutions are utilized in this investigation in order to assess whether or not there are substantive differences in budgeting allocations between this group and the doctoral intensive group in particular. the institutions in this category offer a wide range of baccalaureate programs, and are committed to graduate education through the master’s degree (no doctoral programs exist). during the period in scrutiny, these institutions awarded 40 or more master’s degrees per year across 3 or more disciplines (carnegie foundation for the advancement of teaching, 2001, p.26). our analysis is guided by the following questions: are there important differences between doctoral extensive and doctoral intensive institutions? do spending patterns at doctoral intensive institutions differ significantly from master’s colleges and universities i? fife, b. l. and losco, j. journal of scholarship of teaching and learning, vol. 4., n. 1, may, 2004. 4 table 1: the carnegie classification of institutions of higher education: 1973, 1994, 2000. 1973 1994 2000 i doctoral-granting institutions i. doctoral-granting institutions i. doctoral/research universities research universities i research universities ii doctoral-granting universities i doctoral-granting universities ii research universities i research universities ii doctoral universities i doctoral universities ii doctoral/research extensive doctoral/research intensive ii. comprehensive universities and colleges ii. master’s (comprehensive) colleges and unversities ii. master’s colleges and universities comprehensive universities and colleges i comprehensive universities and colleges ii master’s (comprehensive) colleges and universities i master’s (comprehensive) colleges and universities ii master’s colleges and universities i master’s colleges and universities ii iii. liberal arts colleges iii. baccalaureate colleges iii. baccalaureate colleges liberal arts colleges i liberal arts colleges ii baccalaureate (liberal arts) colleges i baccalaureate (liberal arts) colleges ii baccalaureate colleges—liberal arts baccalaureate colleges—general baccalaureate/associate’s colleges iv. two-year colleges and institutes iv. associate of arts colleges iv. associate’s colleges v. professional schools/ other specialized institutions v. specialized institutions v. specialized institutions theological seminaries medical schools/centers other separate health professional schools schools of engineering and technology schools of business and management schools of art, music, and design schools of law teachers colleges other specialized institutions theological seminaries medical schools/centers other separate health professional schools schools of engineering and technology schools of business and management schools of art, music, and design schools of law teachers colleges other specialized institutions theological seminaries medical schools/centers other separate health professional schools schools of engineering and technology schools of business and management schools of art, music, and design schools of law teachers colleges other specialized institutions vi. tribal colleges and universities vi. tribal colleges and universities minor revisions from 1976 and 1987 are not included. source: adapted from carnegie foundation for the advancement of teaching, 2001. iv. simple descriptive statistics. simple descriptive statistics are provided for public institutions and private institutions in table 2. the mean represents the sum of the values in each category (instruction, research, and public service) divided by the number of values. the smaller the standard deviation, the more the data cluster about the mean (losco and fife, 2000, pp. 57-58). the standard deviation is the most common measure of dispersion for interval-level data, and reflects the dispersion of data points about the mean. clearly, doctoral institutions (extensive and intensive) spent more on research; for public institutions, they also spend more on public service. differences in budget allocations to teaching also exist, with doctoral institutions generally spending a smaller percentage of fife, b. l. and losco, j. journal of scholarship of teaching and learning, vol. 4., n. 1, may, 2004. 5 table 2: institutional expenditures, by carnegie classification scheme, 1999-2000 (percent of institutional spending. % spent on instruction % spent on research % spent on public service public institutions doctoral extensive (n = 101) doctoral intensive (n = 62) master’s i (n = 246) mean s.d. 27.7% 6.4% 33.1% 7.0% 34.6% 5.5% mean s.d. 15.9% 7.3% 10.6% 9.0% 2.0% 2.8% mean s.d. 6.2% 4.6% 3.7% 3.0% 3.1% 3.4% private institutions doctoral extensive (n = 49) doctoral intensive (n = 43) master’s i (n = 230) mean s.d. 34.5% 10.5% 39.5% 12.0% 38.2% 8.3% mean s.d. 15.8% 10.3% 6.7% 10.7% 1.0% 4.0% mean s.d. 1.3% 2.1% 2.6% 8.7% 1.2% 2.3% their budgets on instruction, the differences are also generally small. while these figures are illuminating, a more rigorous analysis follows which affords heightened insight concerning institutional similarities and differences in each of the spending categories in question. v. research hypothesis/appropriate statistical technique. it is hypothesized, in a general sense, that the carnegie classification scheme affects institutional spending on instruction, research, and public service: h1: carnegie classification scheme (x) impacts institutional spending (y) the null hypothesis is that there is no relationship between the carnegie classification scheme and institutional spending: ho:β=0 since the objective of this research effort is to identify a discernible impact, if any, of carnegie institutional categories (explanatory variables) on institutional spending in the three traditional focal areas in the academy (ultimate dependent variables), multiple regression is the optimum manner in which to empirically test h1 (see tufte, 1974, pp.135-163; fife and miller, 2002). vi. specification of the model. a multiple regression equation generally takes the following form: y = α + β1x1 + β2x2 + e where, y=ultimate dependent variable; α=intercept or constant; β=regression coefficients for the explanatory (x) variables; and e=error term. the dependent variables (y) in this evaluation are the percentages of overall spending allocated to instruction, research, and public service. the explanatory variables (x) are the three carnegie institutional categories of interest (doctoral extensive, doctoral intensive, and master’s i). the most plausible manner in which to operationalize them is by creating three dichotomous variables, otherwise known as binary, categorical, or dummy variables. a. using dummy variables. dummy variables are only assessed values of zero or one. in order to prevent perfect fife, b. l. and losco, j. journal of scholarship of teaching and learning, vol. 4., n. 1, may, 2004. 6 multicollinearity, one category is omitted. the β coefficients reflect the changes in the dependent variable with respect to the reference group (the group that is left out). the intercept reflects the value of the dependent variable for the reference group. the t ratio associated with the coefficient on a specific dummy variable is utilized to determine whether or not that group differs statistically from the reference group (schroeder, sjoquist, and stephan, 1986, pp.56-58). b. the t ratio. the null hypothesis that β=0 can be tested by computing the t ratio and comparing it to the appropriate t statistic. if the t ratio is greater than the appropriate t value, the null hypothesis can be rejected at a specified level of significance. in the social sciences, the most common level is the 95 percent confidence interval. if the value of the test statistic lies in the critical region, then it is statistically significant from the other categories (in this instance at the .05 level of significance) and means that the sample size may vary up to 5 percent or less from the population 95 times out of 100. according to bernstein and dyer (1992), “the requirement that findings be significant at the .05, or a more restrictive, level reflects the fact that the scientific community fears the acceptance of an untrue hypothesis much more than the failure to accept a true hypothesis. it also reflects the scientific community’s goal of establishing a body of knowledge by building on sets of confirmed hypotheses. if there is any reasonable doubt about the empirical support for a hypothesis, it is better to delay acceptance until it is retested than to accept and build on it. the requirement of such restrictive levels of significance may also stem from a concern that researchers may be advocates of the hypotheses they are testing, and advocates may be tempted to fudge findings ever so slightly to favor their hypotheses” (p.182). with this in mind, the .01 and .05 levels of significance are utilized in this evaluation. c. three equations. since a reference group has to be deleted when using dummy variable analysis, three equations are utilized in order to test h1 (for both the public and private institutions). each equation is tested using each of the three dependent variables. the models include the following: 1) pb = α + β1 (doctoral extensive) + β2 (doctoral intensive) + e where, pb=percent of total budget allocated for instruction, research, and public service; α=value of the dependent variable for the reference group (master’s i); doctoral extensive=dummy variable (1=doctoral extensive institutions; 0=otherwise); doctoral intensive=dummy variable (1=doctoral intensive institutions; 0=otherwise); and e=error term. 2) pb = α + β1 (doctoral extensive) + β2 (master’s i) + e where, pb=percent of total budget allocated for instruction, research, and public service; α=value of the dependent variable of the reference group (doctoral intensive); doctoral extensive=dummy variable (1=doctoral extensive institutions; 0=otherwise); master’s i=dummy variable (1=master’s i institutions; 0=otherwise); and e=error term. 3) pb = α + β1 (doctoral intensive) + β2 (master’s i) + e where, pb=percent of total budget allocated for instruction, research, and public service; α=value of the dependent variable of the reference group (doctoral extensive); doctoral intensive=dummy variable (1=doctoral intensive institutions; 0=otherwise); master’s i=dummy fife, b. l. and losco, j. journal of scholarship of teaching and learning, vol. 4., n. 1, may, 2004. 7 variable (1=master’s i institutions; 0=otherwise); and e=error term. the statistical results are presented for the public institutions in table 3. the first part of table 3 presents regression results for percent of budget allocated to instruction both for public institutions (panel a) and for private institutions (panel b); for percent of budget allocated to research both for public institutions (panel c) and for private table 3. regression results. model 1 model 2 model 3 panel a. percent of budget allocated to instruction public institutions constant doctoral extensive doctoral intensive masters r2 34.6** (0.4) -7.0** (0.7) -1.5 (0.8) 0.19 33.1** (0.8) -5.4** 1.5 (0.8) 0.19 27.6** (0.6) 5.4** (1.0) 7.0** (0.7) 0.19 panel b. percent of budget allocated to instruction private institutions constant doctoral extensive doctoral intensive masters r2 38.2** (0.6) -3.7* (1.5) 1.3 (1.5) 0.02 39.5** (0.6) 5.0* (1.9) -1.3 (1.5) 0.02 34.5** (1.3) 5.0* (1.9) 3.7* (1.5) 0.02 standard errors of the coefficients in parentheses. ** statistically significant at the 1% level. * statistically significant at the 1% level. fife, b. l. and losco, j. journal of scholarship of teaching and learning, vol. 4., n. 1, may, 2004. 8 institutions (panel d); for percent of budget allocated to public service both for public institutions (panel e) and for private institutions (panel f); for percent of budget allocated to research. d. interpreting the results2. instruction. the data in table 2 suggest that doctoral extensive do not dpend more on instruction than do other types of institutions, and may spend less. for public institutions, the constant term is always significantly different from zero at the 1% level. whether the reference group is master’s institutions or doctoral intensive institutions, the results indicate that doctoral extensive institutions spend a significantly smaller percentage of budgets on instruction. however, the difference between doctoral intensive and master’s institutions is not statistically significant. for private institutions, doctoral extensive institutions allocate a significantly smaller percentage of their budgets to instruction as well, whether compared to doctoral intensive institutions or to master’s institutions; however, the difference is significant only at the 5% level. once again, doctoral intensive and master’s institutions appear to spend essentially the same percentages of their budgets on instruction. both for public and for private institutions, the explanatory power of the regressions, as shown by the r2s is small (19% of the variation in instructional spending is explained for public institutions, and only 2% for private institutions). research. the raw numbers in table 2 indicate that doctoral institutions allocate a larger percentage of their budgets to research than do master’s institutions (between 6.7% and 15.9%, compared with 1-2%). the regression analysis demonstrates that these differences are statistically significant. in all formulations of the model, the constant terms and the coefficients are statistically significant from zero. all the coefficients estimated for doctoral institutions are significant at the 1% level, and those for master’s institutions are significant at the 1% level as well. in addition, the models have much larger r2s (0.55 for public institutions and 0.40 for private institutions) than do the models for instructional spending and for public service spending. the importance of doctoral institutions in explaining variation in spending on research is noteworthy, but not unexpected. public service. the pattern of spending shown in table 2 is not as clear-cut as is the pattern in instructional spending or in research. among public institutions, doctoral extensive institutions table 3 (continued). regression results. model 1 model 2 model 3 panel c. percent of budget allocated to 2 we have chosen to present regression results which show the effects of including or excluding each of the three institutional categories. readers not familiar with regression analysis incorporating dummy variables will note the following two features of the results. first, the choice of which category of institution to exclude has no effect on the explanatory power (r2) of the regression. second, the choice of which category to exclude has no effect on the structural properties of the regression. for example, consider these two formulations of the model: %research = α1 + β1,1*(doctoralextenvise) + β1,2*(doctoralextensive) (where masters is the excluded category) %research = α2 + β2,1*(doctoralextenvise) + β2,3*(masters) (where masters is the excluded category) the reader will note that α2 = α1 + β1,2 and β2,1 = β1,1 β1,2 and β2,3 = 0 β1,2. that is, model 2 is a linear transformation of model 1. this will be true for all model pairs in which the only change is from one excluded institutional category to another. fife, b. l. and losco, j. journal of scholarship of teaching and learning, vol. 4., n. 1, may, 2004. 9 research public institutions constant doctoral extensive doctoral intensive masters r2 2.0** (0.3) 14.0** (0.6) 8.6** (0.8) 0.55 10.6** (0.7) 5.4** (0.9) -8.6** (0.8) 0.55 15.9** (0.5) -2.5** (0.6) -3.1** (0.4) 0.55 panel d. percent of budget allocated to research private institutions constant doctoral extensive doctoral intensive masters r2 1.0** (0.4) 14.7** (1.0) 5.6** (1.1) 0.40 6.7** (1.0) 9.1** (1.4) -5.6** (1.1) 0.40 15.8* (0.9) -9.1** (1.4) -14.7** (1.0) 0.40 standard errors of the coefficients in parentheses. ** statistically significant at the 1% level. * statistically significant at the 1% level. fife, b. l. and losco, j. journal of scholarship of teaching and learning, vol. 4., n. 1, may, 2004. 10 table 3 (continued). regression results. model 1 model 2 model 3 panel e. percent of budget allocated to public service public institutions constant doctoral extensive doctoral intensive masters r2 3.1** (0.2) 3.1** (0.4) 0.6 (0.5) 0.11 3.7** (0.7) 2.5** (0.9) -0.6 (0.5) 0.11 6.2** (0.4) -2.5** (0.6) -3.1** (0.4) 0.11 panel f. percent of budget allocated to instruction private institutions constant doctoral extensive doctoral intensive masters r2 1.2** (0.3) 0.1 (0.6) 1.4* (0.6) 0.01 2.6** (0.6) -1.2 (0.8)) -1.4* (0.6) 0.01 1.3* (0.5) 1.2 (0.8) -0.1 (0.6) 0.01 standard errors of the coefficients in parentheses. ** statistically significant at the 1% level. * statistically significant at the 1% level. spend more (twice as large a percentage as do master’s institutions), while among private institutions, doctoral intensive institutions spend twice as large a percentage of their budgets on public service as do doctoral extensive or master’s institutions. the regression results bear this out. the difference between spending in doctoral intensive institutions and master’s institutions is statistically significant in models 1 and 2, and the difference between doctoral intensive institutions fife, b. l. and losco, j. journal of scholarship of teaching and learning, vol. 4., n. 1, may, 2004. 11 and doctoral extensive institutions is nearly (but not quite) statistically significant in models 2 and 3. however, as was the case with instructional spending, none of the models explains a significant percentage of the variation between institutions (the r2 is only 0.01). vii. conclusion. our data reinforce the carnegie foundation assertion that the number and array of doctoral degree conferrals correlates well with institutional commitment to research. the analysis adds the perspective that this correlation holds whether research is measured by absolute dollars or by spending priority, i.e., as measured by proportion of overall spending committed to research. this investigation adds an additional nuance, however. whereas officials representing doctoral institutions spend proportionately more on research than master’s degree institutions in both the public and private categories, support for instruction and public service yields interesting similarities. among private institutions, there is little variance in commitment to instruction between doctoral extensive, doctoral intensive, and master’s i institutions. among public sector institutions, only doctoral extensive institutional representatives spend significantly less on instruction than their counterparts in the other two categories. yet even at doctoral extensive public institutions, however, instructional spending is still the largest budgetary component among the three major functional groups examined. in light of the fact that more undergraduates receive baccalaureate degrees at research institutions than at either masters or baccalaureate colleges (carnegie foundation for the advancement of teaching, 2001, p.20) and considering the recent debate about the relative neglect of undergraduate education at research institutions (boyer commission on educating undergraduates in the research university, 1998), it is important to note the magnitude of expenditures all of these institutions devote to instruction relative to other spending categories. whatever problems may or may not exist with the deliverance of undergraduate education at research institutions, there is clearly no lack of financial effort by doctoral institutions toward meeting the needs of instruction. this research effort also suggests that doctoral intensive universities may be the chief beneficiaries of any new carnegie classification that includes institutional mission. this is because, with the exception of additional resources devoted to research and a small number of doctoral programs, there is little in the objective spending measures that differentiates these institutions from master’s i colleges and universities. spending on instruction and public service is nearly identical for doctoral intensive and master’s i institutions in the public arena and there is little difference between the two in the private sphere as well. doctoral intensive schools may be able to attain a greater degree of differentiation once self-reported institutional mission is sufficiently accounted for in the carnegie classification scheme. references bernstein, robert a. and james a. dyer. (1992). an introduction to political science methods 3rd ed. englewood cliffs, nj: prentice hall. boyer commission on educating undergraduates in the research university. (1998). reinventing undergraduate education: a blueprint for america’s research universities. menlo park, ca: carnegie foundation for the advancement of teaching. carnegie foundation for the advancement of teaching. (2001). the carnegie classification of institutions of higher education–2000 edition. menlo park, ca: carnegie foundation for the fife, b. l. and losco, j. journal of scholarship of teaching and learning, vol. 4., n. 1, may, 2004. 12 advancement of teaching. fife, brian l. and geralyn m. miller. (2002). political culture and voting systems in the united states: an examination of the 2000 presidential election. westport, ct: praeger publishers. holmes, jr. oliver wendell. (1897). the path of the law. harvard law review 10, 457-478. lindbloom, charles. (1959). the science of muddling through. public administration review 19, 79-88. losco, joseph and brian l. fife. (2000). higher education spending: assessing policy priorities. in j. losco and b. fife (eds.), higher education in transition: the challenges of the new millennium. westport, ct: greenwood publishing group–bergin & garvey, 51-81. schroeder, larry d., david l. sjoquist, and paula e. stephan. (1991). understanding regression analysis: an introductory guide. newbury park, ca: sage publications. tufte, edward r. (1974). data analysis for politics and policy. englewood cliffs, nj: prentice hall. u.s. department of education, national center for education statistics. (2003). integrated postsecondary education data system. available at http://nces.ed.gov/ipeds/. 503 service temporarily unavailable 503 service temporarily unavailable nginx/1.14.1 1 teaching physics of everyday life: project-based instruction and collaborative work in undergraduate physics course for nonscience majors authors: marina milner-bolotin e-mail: marinamb@physics.uetxas.edu marilla d. svinicki the university of tx at austin abstract in the current paper, we describe a project-based physical science course for nonscience majors designed and implemented at the university of texas at austin during the fall semester of 1999. we focus on practical implications of project-based instruction1 1 for more information on project-based instruction see: 2 in motivating nonscience majors in science study, as well as the challenges the students and the instructors encountered during the course. cognitive, attitudinal and social outcomes of the project-based physical science course as reported by the students throughout the semester are also discussed. keywords: project-based instruction (pbi), physical science, motivation, collaborative learning http://ouray.cudenver.edu/~nflejeun/lcstrategies.htm and http://forum.swarthmore.edu/~sarah/discussion.sessions/blumenfeld.html. 3 full text: when people turn away from science in a world that is increasingly dependent on technology, it is time to take a look at the way science and technology are taught in our schools robert n. little (little 1971). i. introduction most of us will agree that minimal scientific literacy is required from every educated citizen. not surprisingly, the issue of physics teaching to nonscience majors has become one of the hottest issues in contemporary science education2. thousands of college students, regardless of their majors, are required to enroll in and pass a physical science course. the reason for this requirement is the hope that a physical science course will (a) improve student scientific literacy and critical thinking skills, and (b) serve as a conceptual base for subsequent courses taken (dickinson and flick 1998). however, there is ample evidence that the traditional introductory physical science course too often fails to fulfill any of these goals (mcdermott 1991; hake 1998). two major factors may account for this 2 for more information see physical science resource center (psrc) developed by the 4 failure (tobias 1990; tobias 1992). on the one hand a majority of nonscience majors exhibit very low self-efficacy toward science and mathematics. on the other hand, the students put little personal value in learning physical science (maehr and midgley 1991; pajares 1996; dickinson and flick 1998). these factors substantially contribute to the lack of motivation for science study demonstrated by the students (forsyth and mcmillan 1991). american association of physics teachers (aapt): http://www.psrc-online.org/. our research thus focuses on designing and exploring the learning environment that promotes student motivation and supports meaningful science learning. in this paper, we describe an attempt to address the challenges described above through the incorporation of the project-based instruction (pbi) into a traditional undergraduate physical science course for nonscience majors. the goals of the current study are: 1. to explore effective ways of implementing pbi in the undergraduate physical science hands-on laboratory course for nonscience majors; 5 2. to gather and categorize information about possible student outcomes of the pbi; 3. to analyze student perceptions of pbi as teaching and learning environment; 4. to outline interesting directions for the future study. ii. why project-based instruction? education should be viewed "as a social enterprise in which all individuals have an opportunity to contribute and to which all feel a responsibility". john dewey as we pointed out earlier, the majority of nonscience majors coming to the physical science class have rather negative expectations from the course. the students rarely see how the course contributes to their future beyond being an "additional obstacle in their college career". keeping this in mind, our first goal in redesigning the course was making it relevant and useful to the students. however, convincing them that understanding the basic 6 concepts of physical science will be helpful in the future is not an easy task, unless we can show that this knowledge can make a difference in students' lives. therefore, we decided to focus on the role of science and technology in everyday life. we asked the students to investigate (a) how do the appliances we use in everyday life work? (b) what are the physical principles they are based upon? and (c) what are the most important parameters of these appliances we have to know about in order to be wise consumers on the modern market? by asking these questions, we hoped to raise student awareness and curiosity about science. today it is widely accepted that students do not come to the science classrooms as blank slates. they bring a wide array of common sense knowledge and beliefs that too often "successfully" interfere with the formal science instruction (shipstone, rhoeneck et al. 1988; saxena 1992; smith, disessa et al. 1993; wandersee, mintzes et al. 1994). these preconceptions are very resilient to change and the strategy of a gradual replacement of student "misconceptions" with the "acceptable scientific conceptions" 7 proposed during the 1980's was in general proven unsuccessful (webb 1992; chambers and andre 1997). as it was pointed out by andrea disessa (disessa 1988, p.49) "the transition to scientific understanding involves a major structural change toward systematicity, rather than simply a shift in content". therefore, our second goal in redesigning the course was building an environment that supports student "structural change toward systematicity". we tried to achieve it by: 1. helping the students to become aware of their learning process as well as of the informal science knowledge, they brought to the physical science classroom (milner-bolotin 1999). 2. emphasizing the relations between the theoretical concepts studied during the traditional classroom activities and their applications explored during the pbi. we also put a special emphasize on student learning outcomes, as expressed in the development of student cognitive as well as metacognitive skills. the most important of them, from our point of view, are learning how to: 8 1. ask meaningful questions related to science and to approach the investigation; 2. gather appropriate data and analyze it; 3. evaluate the reliability and the validity of the information, and 4. communicate about science effectively. the pbi offered a promising framework for the goals we chose. as it was outlined by blumenfeld et al.: project-based learning is a comprehensive perspective focused on teaching by engaging students in investigation. within this framework, students pursue solutions to nontrivial problems by asking and refining questions, debating ideas, making predictions, designing plans and/or experiments, collecting and analyzing data, drawing conclusions, communicating their ideas and finding to others, asking new questions and creating artifacts (blumenfeld, soloway et al. 1991, p. 371). however, designing projects that will be relevant and interesting for the students and giving them an opportunity to become independent learners are not the only arguments in favor of pbi incorporation into science teaching to nonscience majors. we will mention here two additional reasons supporting our choice. the first one relates to the population of students participating in the physical science classes. the term "nonscience 9 majors" refers to a very heterogeneous group of students. in addition to majoring in different fields, and being freshmen, sophomores or juniors, these students have various science and mathematics backgrounds. teaching such a heterogeneous class is a challenging task. however, if every one of these students is given an opportunity to make a unique contribution to the project in the field of his or her expertise, the class can benefit from such diversity. this opportunity might have an invaluable impact on students' content knowledge, as well as on their self-confidence and self-efficacy toward sciences (barron 1998). the second reason relates to the learning environment in the project-based classes. pbi is based on the ideas of social constructivism, emphasizing the role of social environment in teaching and learning (boaler 2000). it promotes mutual respect, support and understanding, making an impact on student-student and student-instructor relationships. the role of these relationships in the science class for nonscience majors can not be overemphasized. as it was pointed by walsh and maffei: 10 there are a variety of theoretical grounds for suspecting that a more positive student-professor relationships will lead to increased learning. insofar as motivation plays a critical role in learning by initiating, channeling, and sustaining student efforts to learn, theoretical linkage between the quality of the student-professor relationship and motivation to learn is very important in accounting for the relevance of the student-professor relationship to learning (walsh and maffei 1998, p.1). therefore, by incorporating pbi into a physical science course we expected to affect student affective as well as cognitive outcomes. our next task was to make pbi work in our classroom. in the following sections, we will describe the physical science course and the organization of the project the students were engaged in during the semester. iii. introductory guided discovery laboratory course in physical science in order to understand the organization of the project a brief description of the particular physical science course will be helpful. physical science ps-304 class is a small-section (up to 22 students) introductory 11 guided discovery laboratory course for nonscience majors. the course is usually taught by a graduate student and covers the topics of electricity, magnetism, sound and optics. it is a three-hour course: the second in the six-hour sequence of two physical science courses for non-science majors. the course consists of eight hands-on guided activities aiming to "discover" basic physical principles using very unsophisticated equipment. the students follow the lab manual written about three decades ago and almost unchanged since then. the manual mostly focuses on confirmation of known laws and principles, containing detailed prescriptions of what the students are supposed to do, leaving little room to student open-ended discovery (little 1973). although, the course does not incorporate any technology or computers, it includes a few interesting classical experiments. the textbook of the course is paul hewitt's "conceptual physics" (hewitt 1997). iv. population current study included two sections of the physical science course (n=42). gender 12 and ethnic representation of the students, participating in the study, as well as student college history are given in the table 1. out of 42 students nine (21%) were majoring in elementary education, one student was majoring in computer sciences and all the rest were liberal arts or communication majors (77%). each of the 42 students in the study successfully completed the ps-304 course. total enrollment: n = 42 gender representation ethnic representation year at college male female afric an asi an his pa cauc asian fre sop juni seni number of students 13 29 1 3 3 35 3 26 7 6 percentage 30% 70% 2% 7% 7 % 84% 7 % 62% 17% 14% table 1. gender ethnic and college history representation of the ps-304 students, participated in the study. 13 v. the organization of the project "how things work" in many ways, education is a process of learning to indicate to ourselves things we hadn't seen. julia t. wood the organization of the project "how things work" is schematically described in fig.1. there were five main work-stages in the project: (i) organization of the learning environment; (ii) question generation; (iii) project development; (iv) artifact production; (v) project presentation. research shows that successful project required a lot of peer and instructor feedback and at least a few iterations (barron 1998; krajcik, blumenfeld et al. 1998). therefore, the order of the stages (fig. 1) was important only for the first round of the instruction. as it often happens, the more students proceeded with the project, the more interrelated and interconnected the stages became. for example, while working on artifact production, stage iii, students could decide that the questions they posed during stage ii could be more specified, clarified or some new interesting questions could arise. 14 this is where figure 1 should be figure 1. schematic organization of the five stages of the project "how things work". in the following subsections, we will outline each one of the project stages mentioned above3. 3 for more information about the organization of the ps-304 course and the project "how things work" see the course web site: http://www.ph.utexas.edu/~ps304. stage i: organization of learning environment although, pbi does not necessarily imply collaborative work of a small group of students, contemporary research supports the claim that small group instruction makes pbi more powerful (blumenfeld, soloway et al. 1991; cooper and robinson 1998; cooper and 15 robinson 1998; krajcik, blumenfeld et al. 1998). therefore, during the first month of the semester, we invested much effort into organization of a collaborative classroom environment. during that period, the students were working on standard classroom activities and labs, but most of the time they were encouraged to work in small groups, share responsibilities, help and support each other. we tried to help students experience different possibilities of effective collaborative work (millis and cottell 1998), providing them with various opportunities for small group discussions, problem solving and hands-on activities. we also encouraged them to work together on the homework assignments. another important aspect was establishing after class communication. an electronic discussion board was useful for this purpose. the first few weeks of the semester were found especially critical in creating an effective learning environment and investing effort in doing so at the beginning returns twofold later4. 4 for more information about classroom activities, we found helpful for this purpose look in 16 ps-304 web site: http://www.ph.utexas.edu/~ps304. 17 stage ii: question generation as was pointed out by blumenfeld et al. (blumenfeld, soloway et al. 1991), the main feature distinguishing the pbi from any other inquiry-based activity is the presence of a driving question that guides classroom activities: projects are decidedly different from conventional activities that are designed to help students learn information in the absence of a driving question. such conventional activities might relate to each other and help students learn curricular content, but without the presence of a driving question, they do not hold the same promise that learning will occur as do activities orchestrated in the service of an important intellectual purpose (blumenfeld, soloway et al. 1991, p. 372). therefore, the next step in the organization of the pbi was helping the students to formulate questions that they would be interested to pursue in the future. a general driving question of the project was to investigate and to report in the classroom how one of the appliances we use in everyday life works. the students were given freedom to choose any appliance they wished. the only restriction on their choice was that the appliance had to use physical principles we were dealing with in the classroom. 18 as simple as it might look, it is not trivial to choose an appliance that is relevant for the students' lives; is simple enough to be explained using the basic physical principles; and students are interested to work with. moreover, working on the project in small groups of four or five students, the students had to reach a group agreement on the appliance choice. taking all this into account, we gave them up to four weeks to discuss what appliance they would like to investigate and what interesting questions they would like to ask. we also tried to put students' attention on different applications of the principles they have been learning. the course textbook (hewitt 1997) and on-line resources about "how things work"5 were good starting points for students' research. in order to get continuous feedback on student progress we established very close e-mail communication. after choosing a particular appliance, each group prepared a tentative proposal of 5 for example, see http://www.howstuffworks.com/microwave.htm. 19 the research, including the research questions, possible information sources etc. originally, the students came up with rather broad questions. every group met with the instructor during the office hours to translate these broad questions into small subproblems that (a) could be solved more easily and (b) would lead the students to the solution of the original question. during this meeting, the instructor also emphasized what labs and hands-on activities might be especially helpful for answering the questions the students posed. this meeting gave an impetus for the process of students making connections between the concepts learned in the classroom and the driving question they wanted to answer. for example, one of the groups decided to investigate how the toaster works. after thinking about this project for a few weeks, the students became interested in understanding how it happens that the toast pops up before it gets burned. this group put special attention on the hands-on activities related to electric current and to the thermal properties of metals. the students were very surprised when they realized that the timer device they wanted to 20 find inside the toaster was a simple bimetallic strip they had been working with in the lab. another group decided to build binoculars, using simple laboratory equipment. this group became so engaged in the project that they decided to investigate a few additional devices: a kaleidoscope, a telescope and even night vision binoculars. stage iii: project development the development stage was the central stage of the project. in addition to studentinstructor classroom communication, every group had to e-mail on biweekly basis a progress report as well as to describe the problems they encountered. the students got a prompt constructive feedback from the instructor. sometimes, the electronic communication was not sufficient and then the instructor met with the students during the office hours and discussed possible solution to the existing problems. moreover, in order to provide every student with an opportunity to express his or her opinion without a fear of "being punished for it", we incorporated the classroom assessment technique called a minute paper 21 (angelo and cross 1993, p. 149). it consisted of asking the students to provide anonymous three-sentence response on the following questions: (a) what were the best/worst things about the project they encountered during the last week? (b) what is the most important thing related to the project they learned during the last week? (c) what important questions still remain unanswered?" this simple but very effective technique gave us the opportunity to adjust and improve the group work as well as to help the students to formulate and answer their research questions. stage iv: artifact production although we included artifact production in a separate stage, the students started creating, recording and gathering artifacts from the very beginning of the project. by "artifacts" we mean all the products the students created, found and analyzed during the project. 22 creating interesting and effective artifacts requires critical thinking abilities as well as observational skills. writing component played an important role during the project, since in addition to gathering and analyzing the information, necessary to answer the research question, the students were asked to reflect on their project. during this stage, the students focused on learning (a) how to report their findings to other students; (b) how to incorporate technology in their research (power point, video, etc.); (c) how to communicate information effectively. learning these important skills is necessary for contemporary undergraduate students. however, only a few students in the physical science course felt confident about them. stage v: project presentation toward the end of the semester, each group wrote a report, based on their investigations, including all the artifacts they created during the semester. after discussing 23 their findings with the instructor, the students presented them to the classmates. the goal of the presentation was to communicate science via teaching other students what they had learned during the project. students were encouraged to bring the appliances they were researching to their presentation. the presentations were evaluated by the instructor and by their classmates, using the criteria developed together with the students. after the presentation every group met with the instructor in order to reflect on the presentation and on the group work. seven out of nine student teams' presentations were videotaped and analyzed. three groups out of seven asked to watch the video in order to reflect on their presentations. vi. student cognitive, affective and social outcomes while learning has many ends, teaching has only one: to enable or cause learning. k. patricia cross data collection procedures 24 in order to evaluate the project "how things work" and to find out what were the outcomes of the pbi we continuously gathered and analyzed course information throughout the semester. trying to uncover students' understanding of basic physics concepts, we conducted, videotaped and analyzed multiple interviews focused on student understanding of electricity and waves (milner-bolotin 1999; milner-bolotin, lane et al. 1999). minute papers, student reflections and e-mail communication also served us an invaluable source of information about the project. moreover, every test included at least one open-ended question about the science applications in everyday life. in addition to that, at the end of the semester we asked the students to reflect on their project experience, writing a threepage essay. we asked them to answer four questions: (a/b) what did/didn't you like about the project? (c) what did you learn from participating in the project? (d) how the project can be improved? we did not ask specific questions about different cognitive or affective outcomes, since we did not want to influence student responses. all the outcomes described in this section were reported independently by a large 25 number of students. the outcomes mentioned by the students and caused, in their opinion, by the pbi fall into three interrelated categories: cognitive/metacognitive, affective and social outcomes. in the next paragraphs, we will describe them, illustrating each one with students' quotes. a) cognitive and metacognitive outcomes: the majority of the students mentioned how much they learned while participating in the project, not only learning the content of the course, but also making connections between different course related activities. not surprisingly, the students emphasized that the project helped them to relate scientific concepts learned in the classroom to everyday life: the things i learned in this physical science class came into play when doing research on the toaster. l.c. the students also pointed out that teaching the ideas they had learned to other 26 students helped them to deepen their understanding. science teaching experience was very unusual for the majority of the students. the part i liked most about the project was presenting the material we had learned, and being able to teach it to others. i also thoroughly enjoyed watching everyone else's final product being presented to the class. a.w. another student wrote: this project was much more interesting than the labs. i felt like we had discovered something unique instead of given the labs easy format. i liked that we had to find information. however, because we had all worked together before, it was easy to work together on this project. k.m. more than 50% of the students mentioned that the project helped them to learn how to learn and become more responsible for their learning. taking responsibility for the group learning was a difficult step for the students. however, a lot of them emphasized that it was a very positive learning experience. more than 25% of the students mentioned 27 that working on the project helped them to learn how to read popular science articles and books and be critical about what they have read. a few students mentioned how surprised and sometimes disappointed they were to find out that not all the information published on the web is trustworthy and they should not believe every written word. a few of them pointed out that working in small groups on a systematic basis helped them to become more critical of their own learning habits. some of the students acknowledged that they learned to monitor their learning and to manage their time. b) affective outcomes: affective outcomes were the most frequently mentioned outcomes in students' reflections. the majority of the students emphasized that the project helped them to look at science and at themselves doing science in a different way. more than 70% of the students emphasized gaining an appreciation of science. this outcome relates to the gain in self-confidence and self-efficacy toward science, as mentioned by more than 60% of 28 the students. a lot of students reflected that they were surprised to find they could do science and even teach other people science concepts they understand. i also liked the project because we got to apply physical science to our everyday life. that aspect is not common in most classes. i think it helped me respect physical science more, because i realize how crucial it is to all of our lives. m.r. i think that as far as physics goes, this project made me look around and think more carefully about the way things work instead of just taking them for granted. i don't know why i had never contemplated how a toaster worked, for example, but i hadn't, so this gave me the opportunity to see things in a different light. e.h. c) social outcomes: more than 50% of the students pointed out that during the project they learned some social skills. they acknowledged that learning to compromise, to listen to other people, to work collaboratively, to respect other students and the instructor were very important. i learned much about the telescope and new scientific terms, but this project also taught me how to compromise and to work better with other people. 29 a.w. there were many things that i thoroughly enjoyed about our group project, but one thing stands above all of the others. at the university level, many professors have forgotten the importance of the group work. working well with others is a vital skill necessary for success in today’s society; and this idea is often lost in institutions of higher learning. group members attempt to combine their skills for attaining a common goal. compatibility is uncertain and groups do not always agree. even though dissention often teachers a better lesson when working with others, i was fortunate to be a part of a work-efficient foursome, whose skills combined for a phenomenal project. d.f. a few students mentioned that making friends was one of the very important personal outcomes of the course. what i liked most about the group project was the group efforts. i feel that i became much closer to my group and may even have some lasting friendships after the class is over. we worked well together, and taught each other the things that we were researching. we had good discussions over the subject matter that helped to give me a better understanding of the course content at that time. v.b. vii. addressing the challenges of the pbi 30 in this section, we will outline possible strategies that might be useful in overcoming the common challenges to successful implementation of the pbi, based on our observations. these challenges can be roughly divided into three groups: (a) conceptual challenges, (b) challenges related to project management, (c) communication challenges related to student difficulties in communicating project findings. a) conceptual challenges conceptual or cognitive challenges relate to choosing a topic of the investigation, gathering data or analyzing the information. sometimes students insist on a topic that is much more complex than it seems. for example, one group decided to investigate how the cellular phone works. with the background they had it was a great challenge to pursue this question in depth. too often, the students do not have enough knowledge and experience to comprehend the complexity of the issue. they need good advice from the 31 instructor to help them decide if it is feasible to investigate the topic in the available time. frequent student feedback and formative assessment are necessary components of the successful pbi. in our case, the minute papers mentioned above, personal meetings and e-mail communication were very useful in solving this problem. there is one more reason why continuous student feedback is so important in pbi. too often students find some on-line information and without testing its reliability base all their work on it. sometimes the students find different contradictory sources and they don't know "which one to believe". by establishing close student-instructor communication the instructor can help the students to become critical consumers of on-line information. this very important skill will save the students a lot of time and might be very helpful in the future. very often the interaction with the instructor and getting immediate feedback are crucial (walsh and maffei 1998). obviously, not all problems can be solved immediately using an electronic mail or any other type of communication software. however, the 32 students have to know that the instructor is available for them and they can rely on his/her help and support. the worst situation occurs when students have problems and do not ask for help. in order to avoid this a variety of various formative assessment techniques might be useful6. in another case, the instructor might not be an expert in the topic the students want to pursue. pbi opens doors for learning for the students as well as for the instructors. moreover, in the ideal case, the students will become experts in the chosen field and then they will know the specific topic in more depth than may be the instructor does. therefore, it might be useful to connect students with other experts in the field of their interest. for example, in our case, the students who were interested in building a telescope found it very helpful and exciting to use the solar telescope of the department of astronomy at the 6 the book "classroom assessment techniques" by thomas a. angelo and k. patricia cross can serve as a very helpful reference on the ways of incorporating formative assessment into classroom practice (angelo and cross 1993). 33 university of tx at austin7. b) project management 7 our thanks to prof. bob robbins and the department of astronomy staff member lara eakins for supporting the students and helping them to operate the solar telescope. the second type of difficulty relates to project management. pbi requires students to work outside of the classroom and to take responsibility for their learning. helping students to organize their meetings and to find out other ways of communicating is a part of this challenge. as we mentioned earlier, electronic communication amongst the students and between the students and the instructor might be very helpful in solving this problem. students also have to be given time to discuss these issues during the class time. a clear outline of deadlines of different stages of the project will also help the students and the instructor to manage their time more effectively. dr. joylynn reed's (reed 2000) idea to organize students' groups around their schedules might be another way to solve this problem. 34 another important issue in project work management is helping the students to find the information more effectively. providing students with hints about the library or on-line web search and some useful references might be very important and can save them a lot of time and unnecessary frustrations. for example, one group of students wanted to investigate how the microwave oven works. however, the results of their on-line search, using the key-words "microwave oven" were not helpful, since they were getting a lot of sales information and obviously not what they wanted. one of the group members emailed the instructor and got a prompt response to use a search using a different combination of key words: "how stuff works" (brain 2000). the organization of effective collaborative work can also be very challenging. this problem relates to the time management, but it is even more complex. barbara millis and philip cottell in their book "cooperative learning for higher education faculty" (millis and cottell 1998) addressed a few important issues in cooperative group management. the key-issue is being flexible and responsive to students' feedback helping them to organize 35 their work more effectively. c) communication of project findings the third type of challenge relates to teaching students to communicate their findings. we found that the majority of the freshmen and many sophomores had never given a twenty-minute presentation, especially communicating scientific findings. therefore modeling a good presentation, emphasizing important points and supporting the students was necessary to help them in preparation of a successful presentation. in order to reduce student presentation anxiety we modeled the final presentation during the semester, by assigning students to make a five-minute report about one of the topics or experiments discussed in the classroom. we also spent some times discussing the criteria of a good presentation and preparing an outline of the criteria students had to keep in mind while preparing to the presentation. a few groups also consulted the instructor regarding the visual aides they could use 36 in order to make a presentation more attractive to the audience. it is important to mention that during the presentation stage students majoring in communication, theater, photography and other related fields were especially helpful to their classmates and made invaluable contributions to the group project. viii. project evaluation and conclusions the project-based physical science course described above was our first, and we hope a successful, attempt to incorporate the pbi into a traditional physical science course for nonscience majors. we have no doubts that the design can be and should be improved, and we are currently working on incorporating student reflections and suggestions into a new version of the project (milner-bolotin 2000). in order to be able to make some statistical inferences and generalizations about the population of nonscience majors we are also planning to incorporate rigorous experimental design in the future study. however, this pilot study made it clear for us that nonscience majors are capable of 37 science learning and can be very motivated, productive and creative. the key-issue is letting the students ask questions they want to answer supporting their endeavor through meaningful science learning. the current study also posed a few important questions for further investigation: what are the other key-issues in the pbi that are responsible for cognitive and attitudinal outcomes of the nonscience majors? what effect does pbi have on student conceptual understanding of physical science? does pbi have an impact on student "misconceptions" about physical science? what are the long-term outcomes of the pbi? what is the impact of pbi on student metacognitive skills? how does pbi impact student problem-solving abilities? what is the impact of pbi on instructor's attitudes toward teaching nonscience majors? in conclusion, we believe that the field of teaching sciences to non-science majors will experience an extensive growth during the next decade, and pbi will have a special place in this process. 38 references: angelo, t. k. and k. p. cross (1993). classroom assessment techniques: a handbook for college teachers. san francisco, jossey-bass publishers. barron, b. j. s. (1998). “doing with understanding: lessons from research on problem and project-based learning.” the journal of the learning sciences 7(3&4): 271-311. blumenfeld, p. c., e. soloway, et al. (1991). “motivating project-based learning: sustaining the doing, supporting the learning.” educational psychologist 26(3,4): 369398. boaler, j. (2000). “exploring situated insights into research and learning.” journal for research in mathematics education 31(1): 113-119. brain, m. (2000). how stuff works, marshall brain, http://www.howstuffworks.com/microwave.htm, 2000. chambers, s. k. and t. andre (1997). “gender, prior knowledge, interest, and experience in electricity and conceptual change text manipulations in learning about 39 direct current.” journal of research in science teaching 34(2). cooper, j. and p. robinson (1998). small group instruction. an annotated bibliography of science, mathematics, engineering, and technology resources in higher education. (occasional paper no6). madison: university of wisconsin-madison, national institute for science education. cooper, j. and p. robinson (1998). “small-group instruction in science, mathematics, engineering and technology (smet) disciplines: a status report and an agenda for the future.” journal of college science teaching. dickinson, v. l. and l. b. flick (1998). “beating the system: course structure and student strategies in traditional introductory undergraduate physics course.” school science & mathematics 5: 238-246. disessa, a. a. (1988). knowledge in pieces. constructivism in the computer age. p. b. p. george forman. hillsdale, new jersey, lawrence erlbaum associates, publishers. 1: 49-70. 40 forsyth, d. r. and j. h. mcmillan (1991). “practical proposals for motivating students.” new directions for teaching and learning 45: 53-65. hake, r. r. (1998). “interactive-engagement versus traditional methods: a six-thousandstudent survey of mechanics test data for introductory physics courses.” american journal of physics 66(1): 64-74. hewitt, p. g. (1997). conceptual physics. reading massachusetts, etc, addison-wesley longman. krajcik, j., p. c. blumenfeld, et al. (1998). “inquiry in project-based science classrooms: initial attempts by middle school students.” the journal of the learning sciences 7(3,4): 313-350. little, r. n. (1971). “on physics in general education.” the physics teacher march 1971(3): 119-120. little, r. n. (1973). motion and matter: an introductory guided discovery laboratory course in physical science ps-304. austin, the university of texas at austin. 41 maehr, m. l. and c. midgley (1991). “enhancing student motivation: a schoolwide approach.” educational psychologist 26(3&4): 399-427. mcdermott, l. c. (1991). “millikan lecture 1990: what we teach and what is learned closing the gap.” american journal of physics 59(4): 301-315. millis, b. j. and p. cottell, j. jr. (1998). cooperative learning for higher education faculty. phoenix, arizona, american council on education, oryx press. milner-bolotin, m. (1999). simple electric circuits: an analysis of students' conceptions and teaching strategies. austin, tx, the university of tx at austin: 19, (unpublished). milner-bolotin, m., j. m. lane, et al. (1999). knowing electrical circuits: linking student conceptions to the microscopic model of electricity. austin, tx, the university of tx at austin: 25, (unpublished). pajares, f. (1996). current directions in self-research: self-efficacy. american educational research association. new york. 1999. reed, j. h. (2000). private communication. 42 saxena, a. b. (1992). “an attempt to remove misconceptions related to electricity.” international journal of science education 14(2): 157-162. shipstone, d. m., c. v. rhoeneck, et al. (1988). “a study of students' understanding of electricity in five european countries.” international journal of science education 3: 303316. smith, j. p. i., a. a. disessa, et al. (1993). “misconceptions reconceived: a constructivist analysis of knowledge in transition.” the journal of the learning sciences 3(2): 115-163. tobias, s. (1990). they are not dumb, they are different. stalking the second tier, research corporation a foundation for the advancement of science. tobias, s. (1992). revitalizing undergraduate science: why some things work and most don't, research corporation a foundation for the advancement of science. walsh, d. j. and m. j. maffei (1998). “never in a class by themselves: an examination of behaviors affecting the student-professor relationship.” the teaching network 20(1): 1-4. 43 wandersee, j. h., j. j. mintzes, et al. (1994). research on alternative conceptions in science. handbook of research on science teaching and learning: a project of the national science teachers association. d. l. gabel. new york, macmillan publishing company. 1: 177-210. webb, p. (1992). “primary science teachers' understandings of electric current.” international journal of science education 14(4): 423-429. v4n2finkelstein journal of scholarship of teaching and learning, vol. 4., no. 2, september, 2004, pp. 1 – 17. teaching and learning physics: a model for coordinating physics instruction, outreach, and research noah d. finkelstein1 abstract. this paper describes the development of a new university physics course designed to integrate physics, education, research, and community partnerships. the coordinated system of activities links the new course to local community efforts in pre-college education, university education, university outreach, and research on teaching and learning. as documented both by gains on conceptual surveys and by qualitative analyses of field-notes and audiotapes of class, the course facilitates student learning of physics, as well as student mastery of theories and practices of teaching and learning physics. simultaneously, the course supports university efforts in community outreach and creates a rich environment for education research. the following narrative describes the motivation, structure, implementation, effectiveness, and potential for extending and sustaining this alternative model for university level science education. keywords: physics, education, research, outreach, teaching, service learning i. introduction. the explicit mission of many large-scale research universities includes three core elements: the pursuit of excellence in research, teaching, and community service. however since the midtwentieth century, many universities have heavily emphasized research without equal commitment to teaching or community service. efforts directed at supporting high quality teaching (at the university or pre-college level) and partnerships with the communities that house the universities are largely treated as separate, and often non-essential, programs at these institutions of higher education. this paper addresses the question of how such institutions might begin to coordinate these three seemingly disparate elements of the university mission into a single activity system that enhances all three. the focal point of the coordinated system is a class entitled teaching and learning physics offered within the physics department (finkelstein, 2003). the emphasis of the present work is to describe the structure of the class and the impact of the class on students. through the use of preand posttests of students’ conceptual grasp of the physics content, audio tapes of classes, ethnographic field-notes, course evaluations and student interviews, the class is presented as a case study to demonstrate that such an approach is useful for improving students’ grasp of physics and of teaching. at the same time, in addition to documenting the effects that the course has on students, this study examines how well this environment is suited for physics education research, how supporting and surrounding institutions respond, and the potential for sustaining such a pursuit. the discussions of lines of educational research, and the likelihood of sustaining this course, follow the course description and student evaluation. 1 department of physics university of colorado, boulder. the author may be reached at noah.finkelstein@colorado.edu © noah finkelstein, 2004 all rights reserved, do not quote without permission of the author. this research was conducted with the support of the national science foundation’s post-doctoral fellowships in mathematics, science, engineering, and technology education (nsf’s pfsmete grant number: dge-9809496) under the mentorship of michael cole (university of california, san diego) and andrea disessa (university of california, berkeley). i wish to thank these mentors and my colleagues at the laboratory of comparative human cognition and those in the department of physics (barbara jones, edward price and omar clay) for intriguing discussions, insights, and their support. finally, i am grateful for the support and critical feedback of the members of the physics education research group at the university of colorado finkelstein, n. d. journal of scholarship of teaching and learning, vol. 4., no. 2, september 2004. 2 the effort to create a coordinated system of teaching, research, and community partnership builds on recent efforts to support student learning in physics, to incorporate education research within departments of physics, and to address a critical shortage of teachers and lack of diversity at the university level. more and more widely, university faculty acknowledge that the traditional lecture-style physics course fails to impart a deep-seated conceptual understanding of course content (hake, 1998; mcdermott and redish, 1999; redish, 2003). as a result, in some institutions, a new breed of physics class is evolving -one that encourages student engagement. coupled with this recognition, the physics community is beginning to re-assess both the goals of undergraduate courses and what constitutes the discipline more broadly. one outcome of this reassessment is the idea that education research is an integral part of the discipline of physics (aps, 1999). another outcome is that more departments of physics and schools of education acknowledge the need to better prepare teachers of physics (schmidt et al. 1999; timss, 1999). furthermore, in california and elsewhere, a host of political initiatives and educational reforms have challenged the university’s ability to meet its charter commitment to serve all of the state’s population.2 at the same time, studies of service learning programs, those which send university students to engage in community-based activities as part of their education, demonstrate significant and improved outcomes for students engaging in these activities (astin et al. 2000; slch 2003). as a result, a significant response from both the legislature and the university system is to support community outreach in an effort to better prepare current and potential students, especially those from traditionally underrepresented populations. this research program addresses these related problems: 1) the improvement of student interest, understanding, and expertise in physics, teaching, and learning; 2) the creation of community-based activities which address the outreach and service interests of the university; 3) the provision of a research site for the study of the teaching and learning processes. the coordinated ensemble, represented by this course, is an opportunity to merge these many agenda. such an effort follows the work of cole and others who create rich, theoretically motivated environments that foster student learning and support fundamental research on development and culture (cole, 1996; cole, 1998). ii. the activity. the course on teaching and learning physics is composed of three elements: a study of physics content, readings about the teaching and learning of physics, and practical experience teaching physics to less educated students. the course is designed for upper-division undergraduate physics majors who have expressed an interest in education. it is described in the course catalog as: a course on how people learn and understand key concepts in introductory physics. readings in physics and cognitive science plus fieldwork teaching and evaluating pre-college students. useful for students interested in teaching science at any level. pre-requisites: [introductory level courses in electricity and magnetism] each of the three curricular components of the course (physics content, theories of teaching and learning, and practical teaching experience) represents roughly one third of the course. one of the two weekly class sessions focuses predominantly on the study of traditional 2 the anti-affirmative action debates have received widespread publicity and response both at the state level and at the university of california level. the passing of california proposition 209 in 1996 was the culmination of many of these debates. finkelstein, n. d. journal of scholarship of teaching and learning, vol. 4., no. 2, september 2004. 3 physics content. the other emphasizes readings in physics education and cognitive theories of learning. at least once per week, students engage in the laboratory portion of the course, teaching college and pre-college students. each of the course components is designed to complement the others by explicitly providing varied perspectives from which to view physics. because the course draws upon and addresses questions from different domains (physics, education research, and community outreach), it sits at the interfaces between each of these domains and borrows material and methods from each of these bordering worlds (star, 1989). not only do the course participants benefit from the variety of resources, but also by acting at the interfaces of disciplines, the class provides a mechanism for communication between, and coordination of, these differing domains. for the department of physics and the teacher education program, the course serves both as a catalyst for improving the university students’ conceptual understanding and as a common object of discussion and coordination for departments. for physics students, the class acts as an amplifier and reorganizing mechanism for their physics knowledge and as portal from physics into education and teaching. for the outreach program, the course strongly links university efforts in science to community-based education of children. figure 1 illustrates some of these relations. the figure depicts the three interacting components of the course as the vertices of a triangle. each of these components necessarily interacts with and in fact co-constructs the others, as will be described in detail below. as a discipline, physics addresses content and the teaching of content. education concerns itself with the theories and practice of both teaching and learning. lastly, efforts in community outreach blend the practice of teaching (fieldwork) with content (physics) in community-based settings. of course, the boundaries of these domains and activities are not fixed, nor are they mutually exclusive. a program that brings together a study of science content, study of educational and teaching theories, and practical experience teaching science content is remarkably rare, if not unique. usually, these components are separated. for example, in education schools there are science teaching methods classes, where there is some blending of content and pedagogy. in various science departments, there are an increasing number of classes on cognition and student learning. also, in various portions of university, there are an increasing number of servicelearning or practicum classes where students are guided in teaching experiences. however, each of these approaches differs from design and mission of teaching and learning physics, which strives to blend all three elements. in this model for a physics course, the students engage in activities that engender both broad-based skills which span these domains (e.g. problem solving, analysis, and metacognition) as well as specialized domain-specific knowledge and skills (e.g. physics content, and knowledge of and practice in theories of teaching). in addition, the course is designed to be flexible enough to capitalize on the emergent nature of the activity. that is, because the participants, locales, and even the content are dynamic in nature, the precise form of the activity changes over time. the assertion is not that physics itself is changing (though many may argue about the social construction of the discipline), but rather since the course structure is flexible, it allows the coordinated activities to adapt to local context. the arrangement of the components of this activity system (the vertices in figure 1) may be thought to be skeletal in nature, and the actual content, interaction, and environment form the “flesh” that is placed upon the structure. finkelstein, n. d. journal of scholarship of teaching and learning, vol. 4., no. 2, september 2004. 4 figure 1. course structure and disciplinary boundaries. iii. the organizational details. the course is designed for upper division students who have covered at least a minimum level of lower-division coursework in physics. the class meets three hours per week on campus (covering traditional physics content and theories of student learning), and engages students in two to four hours per week of practical experience, teaching in local community centers and schools. each component of the course focuses on the domain of electricity and magnetism (e&m). as much as possible, each component is integrated with the others; the lines between the activities are purposely blurred. a student reading about theoretical difficulties in understanding the concept of electric field is encouraged to wrestle with his own understanding of the topic. furthermore, as much as possible, there is a temporal alignment of the activities. the same week that students study electric fields, they read about student difficulties in understanding the concept of fields, and also attempt to teach the concept to others. the first component of the course covers traditional physics content, approximately twothirds of an introductory course in e&m (using texts such as halliday, resnick and walker, 1997). while calculus is used in the analysis of problems, mathematics and symbol manipulation are not emphasized. rather, each topic is introduced from a conceptual viewpoint, and placed within a broader context of other topics in physics. similarly, the physics segment shifts in focus from symbolic representation and the coverage of text to an active engagement of the students in project-oriented lessons, which foster active construction of models of physics. the class follows a constructivist approach. that is, the course emphasizes learning as a personal and social act where the learner actively builds up understanding using local resources, rather than passively accepting knowledge that is transmitted from the teacher to the student. most often these lessons focus on the physical construction of material and public presentation. finkelstein, n. d. journal of scholarship of teaching and learning, vol. 4., no. 2, september 2004. 5 the added level of emphasis physical construction and public display places the present approach within what papert calls the constructionist camp (papert, 1991). these course activities vary from tutorials (mcdermott et al., 2002), to discussion, to group problem solving (brown and campione, 1990; brown, 1992), to teaching and materials development. the lessons are designed to force students to confront traditional difficulties within electricity and magnetism (posner 1982). the class encourages student learning during class hours, rather than solely after hours. homework is assigned, but emphasizes the conceptual understanding of content. for example, for traditional textbook-based problems, students reflect on the solution process and critique the problem, in addition to deriving an answer. other homework assignments include interviewing or teaching novices about advanced concepts in e&m and subsequently writing-up the process and results. each of these practices is designed to foster development in two domains: mastery of content and improvement of meta-cognitive skills (reflection, regulation, and epistemological development) (schoenfeld, 1986). the second course component, readings in physics education research, occurs in a seminar format. each session begins with brief student presentations followed by discussion. students support or refute ideas presented in the readings using evidence from the other components of the course. readings in physics education research fall into several categories: empirical research on learning (mcdermott and shaffer, 1992), theoretical underpinnings of learning physics (disessa, 1988), and cognitive science approaches to teaching and learning processes more generally (brown et al., 1989). students hand in weekly notes with summaries or questions relating to the readings. the notes are commented upon and returned to the students. these informal notes insure that students read the assigned papers, and force some level of reflective analysis. student teaching occurs at one of four sites, in and after school hours at the junior and senior high school level. students are encouraged to develop and teach their own curriculum (within e&m); in each instance, supervisors, both at the university level and at a local level in the partnering junior or senior high school programs, oversee student work. in this fashion, student fieldwork differs from many traditional service-learning models, as the students are guided and are studying both the content and the practice of teaching while engaged in the process itself. each week students and supervisors write detailed field-notes describing their experiences, curriculum, interactions, and reflections. in addition to using their experiences at the field-sites as a proving ground to test and refine theories of education, students use these sites as resources for research for final projects and papers. again, the final papers are a mechanism for students to reflect back upon the quarter’s activities. though not a goal made explicit to the students, this teaching experience is designed to help students master physics content as well. iv. evaluation and discussion. course evaluation occurs at several levels: at the level of student learning, as a research venue, and as an organizing tool for institutional coordination enabling outreach. the data are presented as a proof-of-concept to demonstrate that this class has the potential to improve student understanding of physics and teaching and learning principles, to serve as a rich venue for research, to provide an avenue for community partnership, and finally, to coordinate these activities into a cohesive whole where the individual components complement one another. the present work predominantly focuses on evaluation at the student level. however, no less significant is the analysis of this system as a research site, or the role that this activity serves in the coordination of various institutions. the data presented are primarily from the first offering of the 10-week (one quarter) course, which enrolled 14 students at a large research university in california. finkelstein, n. d. journal of scholarship of teaching and learning, vol. 4., no. 2, september 2004. 6 a. teaching / learning -student expertise in physics students’ improved capabilities in the domain of physics were of primary interest. it is worthy of note, however, that students generally did not enroll in this course to remediate their understanding of physics. all students in the course had passed one, two or in some cases, three classes in electricity and magnetism. nonetheless, all students demonstrated improved understanding of the domain. evaluation of student performance included: preand posttest of basic concepts in electricity and magnetism (described in more detail below), audio-recordings of class sessions, student evaluations of the course, and in-class observations. all students who completed the course (n=13) participated in all forms of evaluation with the exception of days when students were absent from class. the diagnostic test was a mix of thirty-five free-response and multiple choice questions drawn from the conceptual survey of electricity and magnetism (hieggelke et al., 2001), the electrical circuit concept evaluation (sokoloff et al., 1998), and two original questions.3 in addition to selecting answers for each question, students provided confidence levels for their answers on a 3-point likert-like scale (guessing, somewhat sure, certain). results of the preand posttest are shown in figure 2. the independent axis of the plot lists individual students. the left most student, a, had never formally studied the material and withdrew from the course. the right most student, n, was a fifth year graduate student in physics. a dashed line indicates a division between physics majors and non-majors. the dependent axis plots student performance. the mean preand post-test scores are respectively 54% (s= 25%) and 74% (s= 24%). the average of individual student gains is 51% (s=30%; n=13; p<0.001).4 aside from demonstrating improved conceptual understanding of physics, a few points are worthy of note. no student had a complete mastery of the most basic concepts at the beginning of the course, despite each student having had some background in physics (and having covered this same material previously). while it is not argued that this class is the most effective mechanism for students to learn concepts of physics, it is clear that it is an effective mechanism for increasing student mastery of basic concepts. furthermore, upon entering the class, some of the students, even a physics major (student e), performed at levels roughly equivalent to the unschooled student, student a. generally, those students who had a better grasp of the material upon entering this class made greater improvements than those who were weaker at entry. (the half of the class that performed best on the pre-test made average gains of 66%; whereas, the bottom half of the class made gains of 31%.) perhaps this is due to the challenge of offering a class to such a diverse range of students.5 the students backgrounds spanned a range of eight years of exposure to physics. the * next to the student letters on the bar graph indicates female students. on average, there was no difference in performance by gender. however, the two 3 the following questions were used from the csem: 1-2,6-8,10-11, 15,17-18,20-22,25 the following questions were used from ecce: 1-5,9-10,12-14,27-32 additional questions were developed to augment these instruments with free response questions of the same content, and to cover concept of flux. these added questions are similar to the conceptual questions presented in halliday, resnick, and walker (1997). in all cases, these questions address conceptual understanding of introductory material in electricity and magnetism. 4 the measured gain is similar to the gain hake (1998) reports : (post-pre)/(100 pre). however, here, the average of individual gains is reported rather than the gain of the class average. inverting the order of operations (averaging and measuring gain), shifts the statistical weighting of individual students. the measure of significance is evaluated by means of a single sample t-test of the gain defined above. 5. the class was designed for students who had some familiarity with the material at the outset. as a result, the course was better suited for those students who performed better on the pre-test. however, this is not to say the class model could not be used for an introductory level, or for the lower performing students, but rather the class could not equally well address all of the students who spanned a range of 8 years exposure to formal physics. finkelstein, n. d. journal of scholarship of teaching and learning, vol. 4., no. 2, september 2004. 7 figure 2: student pre and post-test scores on conceptual survey of the basic ideas in electricity and magnetism. mean preand post-test scores are respectively 54% (s= 25%) and 74% (s= 24%). the average of individual student gains is 51% (s=30%; n=13; p<0.001) greatest improvements in absolute score (post-test less pre-test score) were both women (students e & k). in this case, there is some suggestion that while there may be some correlation between gender and class performance (students e & k show 63.5% gains), it is masked by familiarity with the material (by including student b, the average gain drops to the class mean). the audiotapes of classes and observational notes written immediately following each class serve as complementary tools for evaluating student understanding in a qualitative fashion. these ethnographic observations are full of examples that corroborate the pre-test data - students do not begin the course with the expected grasp of material. for example: from this discussion it became very clear that [student c], whom i had asked to step to the board, didn’t really understand electric fields all that well (the topic had recently been covered in [this course], and the class pre-req.) ... it was clear that the discussion helped 2 people in the room [students b and c], was probably useful for [student i] (whom i often caught guessing).day 12 similarly, class discussions reveal when students may not have a thorough grasp of the material. in a reporting on a research study of students’ difficulties with elementary circuits 0 10 20 30 40 50 60 70 80 90 100 a b c d e f g h i j k l m n students s co re ( % ) pre-test post-test non-major major / grad less background more background * * * finkelstein, n. d. journal of scholarship of teaching and learning, vol. 4., no. 2, september 2004. 8 (mcdermott and shaffer 1992), student f reveals some of his own difficulties on audiotape, stating: student f: the point is: students tend to reason sequentially and locally rather than holistically. the students don’t really see the big picture. ... if you take one light bulb out of the circuit, what would happen? and if it is in parallel or in series is there going to be a change? and the students are not understanding that. and that’s actually where i had ... an interesting thing ... like one of those things we could go over is ummm if it’s ...they’re doing one of those in series umm ... they’re talking about the switch, and figure 5 [student reads:] “since the total resistance of the circuit would increase, the current through bulb a would decrease, and it would be dimmer.” and to me, my gut feeling say that it would become brighter, which is kinda interesting. day 9 the use of audiotape and notes helps detail when and why students make conceptual shifts. for example, in a discussion about one of the course readings on the use of analogies for teaching electric circuits (gentner and gentner, 1983), a student reflects on the utility of a water reservoir analogy: student f: can we just talk about…like if you have the uhhh two batteries in series. you get twice the current instructor: right f: which actually [pause] taught me something. i always thought the batteries in parallel gave you [inaudible] [ discussion of the water analogy and two batteries in series produce twice the current of a single battery for a fixed load ] f: okay but see, i thought it was the opposite of that. because i think i was using the wrong model ... [audio tape transcription of class day 7] from my notes written immediately following the class: student f made a very interesting revelation, with which student j also identified: total lack of conceptual understanding of series and parallel batteries and bulbs. student f made the comment that he had thought batteries worked differently until he had read the article. still, throughout the class he and student j would make little mistakes about relative brightness, voltage etc. when asked to think about it in terms of the article and the analogies presented, [however,] they would get the right answer. it required conscious effort and thought. day 7 a comparison of the preand post-test responses for students f and j confirms both that the students better understood how elements behave in series and parallel and that the students had greater confidence in their answers. on the series and parallel circuit questions in the conceptual survey, student f improves 33% from pre-test to post-test (changing from 70% to finkelstein, n. d. journal of scholarship of teaching and learning, vol. 4., no. 2, september 2004. 9 80% correct) and his confidence in his answers rises from 1.8 to 1.1 (where 1 is certain and 3 a guess). student j improves 75% (60% to 90%) with confidence rising from 1.7 to 1.1. of course, not all student comprehension increases linearly. these notes and class recordings capture instances when students’ understanding regresses and detail the conditions that lead to the retreat in understanding. one particularly interesting case details the regression of student d who changes models for understanding current flow in a circuit. on the pre-test, the student consistently demonstrates a more expert view. on the post-test, the student consistently uses a more naive model of current consumption. analysis suggests that this student learned of the more naive model from class discussion, and in particular from another student. such findings are consistent with others who argue that a more confident, but less expert student may convince a more advanced but less confident student to adopt the more naive view (hogan and tudge, 1999). these notes allow for in depth case-studies of students which provide insights into the mechanism behind student achievement, or lack thereof. understanding and learning physics is intertwined with students’ attitudes. by quarter’s end, in an open ended ‘comments’ section of the course evaluation, students report on their own understanding of the material, and their greater comfort and interest in the subject area: “i’m finally enjoying this material [e/m ...] overall, i’ve learned (understand finally) so much about e & m and i’m learning about techniques to teach it” week 5 “i learned a lot about teaching, and even found a new interest in the subject of physics through this course” week 9 “[the best part of the class was] discovering that i didn’t know what i thought i knew about physics” week 10 “i’m not good at [discussion]. this is really the first class where i have really had to talk about what i think” week 10 the goal for students in this course was not simply to improve their conceptual understanding of and attitude towards physics, but also their epistemological development (what it means to know physics) and their awareness of their own understanding. following hammer’s metric of epistemological development in physics (hammer, 1994), there is some suggestion in these data that students are moving from a belief that physics is a mastery of disjointed formula handed down by authority to a belief that physics is a coherently organized and related set of principles useful for an independently developed understanding of the world. furthermore, it is suggested in the above quotes (and those reported later in the paper) that students become more aware of their own knowledge. for example, in a discussion about current conservation, student f reveals: i don't know some of these things. i have the same misconceptions that kids and undergraduates that we're reading about. i'm a physics major, and i don't know these things. i can do the advanced stuff (calculations etc...) but not the conceptual side. day 9 and field-notes document a similar response by another student: student j detailed his experience of not believing in current conservation. he also identified where this belief arises from. finkelstein, n. d. journal of scholarship of teaching and learning, vol. 4., no. 2, september 2004. 10 ironically, such thought hasn't been countered by any formal training. he was a little [upset] about this. day 9 in terms of schoenfeld’s definition of metacognition, students are self-assessing, which is a necessary precursor to regulating their knowledge of physics (schoenfeld, 1986). to summarize briefly, students develop greater expertise in physics broadly conceived. students demonstrate gains in conceptual mastery, attitudes, beliefs of what constitutes physics, as well as their ability to monitor and potentially modify their own level of understanding of physics. b. teaching/learning -student expertise in teaching. the structure of the course was motivated by the belief that such expertise is strongly influenced by students’ experiences teaching. in line with this hypothesis, students report improved ability and interest in teaching. in the ‘comments’ section of course evaluations students report: “i got so excited [about teaching]” week 10 “i thought i had a pretty good grasp on how to teach physics, but i’ve learned enough to revamp my whole style” week 9 “i loved fieldwork b/c i actually was able to observe the teaching theories involved in class and even put them into practice” week 10 “this [fieldwork] really drove home some of the points made in our discussions and readings” week 10 students also report that their conceptions of what constitute teaching changed. during the first and last weeks of class, students turned in “statements of teaching,” where they were charged with writing a paragraph or two on their approach to teaching and teaching philosophy. they reflect upon what it means to teach: [6] student l, pre: ... there seems to be two ways of going about [getting people to learn]. one school of thought is that repetition is how one learns, and the teacher should focus on the most important ideas and go over them repeatedly. the other methods is to saturate the students with information... i have no opinion on which method works better... post: i believe that teaching is less telling and more leading through interactive experiences. it is important for a teacher to know the subject material and be able to convey it clearly, but it is equally important for a teacher to be able to prompt students into learning experiences through which students learn on their own, and in the process own the knowledge themselves. ...another important duty of a teacher is to provide an environment for the student that is conducive to learning. this may include ... 6 though only three statements are presented here, these responses are representative samples, rather than extra-ordinary student statements. finkelstein, n. d. journal of scholarship of teaching and learning, vol. 4., no. 2, september 2004. 11 providing groups of students for interaction and making sure the students are learning and not just memorizing by getting involved in the learning process. student e, pre: i think that the most important thing to do when teaching physics is to keep the class’s attention. this can be done by inspiring students ... making physics ... relevant to their lives, by being humorous or animated ... make physics class an inviting atmosphere and hold class discussions. post: my teaching strategy this quarter in class and at site has focused on creating a solid foundation of physics concepts for the students through hands on activities ... i’ve made a conscious effort ... not to make previous assumptions about one’s knowledge ... i think that group work and project based learning is a more successful way to go than just lecturing student h, post: i have gained invaluable experience in (and learned the main underlying principles of) teaching, both in general, and as it relates to physics. i think this experience has helped me to refine my goals, strategies, and implementation for teaching. ... i also was able to see just how important it is to keep students actively involved with the lesson, participating in throughprovoking projects, thinking, answering questions, asking questions, explaining, and discussing ... these activities are where the real learning takes place, not half sleeping through a lecture on the finer points of proving the schrödinger equation it should be clear that the class holds a heavily constructivist bent (papert, 1991), which seems to have seeped its way into the students’ consciousness. while may be argued that students were parroting discussions from class rather than shifting epistemological and pedagogical view-points, evidence from students’ discussions in class, their field-notes, and final papers, suggests that the students constructed a rich framework of inter-related ideas about teaching and learning. a significant effort was made to ensure that students wrestled with the theoretical underpinnings of their convictions and teaching experiences. some of these theories and tools for understanding the teaching / learning process begin to cycle through public communication in the course as demonstrated by an increased use of technical language from the course readings in student field-notes. for example, student h writes of pre-college students’ failure to grasp a lesson, “this might be a consequence of the fact that they were not forced to confront many of their pre-conceptions, come upon a conflict, and resolve it.” these sentiments parallel posner’s comments on developing a theory of accommodation (posner, 1982). the fieldnote continues, “knowledge ... never really became integrated as a system,” which, in this context, appears to refer to disessa’s notion of knowledge in pieces (disessa, 1988) and reif’s discussion of knowledge structures (reif, 1986). students adopt strategies from the readings and reflect on their own success and failure to implement these strategies in the teaching environment. based on observations, students’ field-notes, and student final projects, there is strong indication that students became better teachers. students were found to implement and evaluate practices discussed in class, to research other methods of teaching, and to appropriate these for use in their own teaching environments. students constantly evaluated their own practices (and finkelstein, n. d. journal of scholarship of teaching and learning, vol. 4., no. 2, september 2004. 12 each other’s). for example, in student f’s field-notes, he reflects on the effectiveness of two approaches to teaching: [the high school] students seemed to respond fairly well to the light bulb/ resistor box experiment, but seemed bored when explained to them by theory on the white board. after the explanation, many students were not able to guess [correctly] about the change in brightness of the bulb as the resistance in the series with the bulbs changed. only after they were able to play with this themselves, were the students able to make theories. day 13 approximately half of the final student projects were directed at assessing pre-college student performance and how performance correlated with such variables as teaching style, learning environment, representational form of the material, or gender. these studies served to confirm or refute others’ theories of student learning, and to evaluate which strategies work best for the university students in their working environments. for example, in a study of the effectiveness of representational forms (white-board versus worksheets), student l confirms the benefits of active-engagement and begins to examine why this works in his classrooms: to see why these two environments [high school and college] yielded such opposite results, one must contrast the situations of the students involved. one can expect that when students learn from a lecture format lesson, they will not be able to apply the concepts as abstractly as when they were involved in the learning. not only will the students be merely watching and not participating, but also it is quite likely that hey will not keep interest in the presentation … at the [college] session, the students were actively learning, discussing and sharing. the [high school students] were instructed to draw diagrams, whereas the u[niveristy] students were using diagrams as tools to reach a goal – finding a solution to a problem. final paper, student l student l uses the opportunity of teaching and of conducting a study of his students’ learning to develop his own theories of and strategies for teaching. in summary, because of the coordinated activities of the course, students demonstrate a greater grasp of both physics and of teaching, and student improvement in each of these areas is broad and multi-faceted. students demonstrate an improved grasp of content and application of content. c. potential for research. this course provides a valuable research venue for making insights into the process of learning physics. while a host of such opportunities exist, the present focus emphasizes undergraduate learning. data from student work, interviews, audio taped classes, and field-notes suggest how the course affords insight into the importance of context in the learning process. the sections above suggest several features of the context and content of this program -the interplay between the study of teaching / learning and the re-examination of physics--that enhance student learning. first, it appears that teaching a topic forces an added level of reflection both upon the content and about an individual’s own mastery of the subject. in support of this work, data from the class are being analyzed to observe the effects of teaching. preliminary analyses of student performance indicate that students are more likely to master a subject conceptually if they teach finkelstein, n. d. journal of scholarship of teaching and learning, vol. 4., no. 2, september 2004. 13 the subject than if they cover textbook homework problems for the same amount of time. however, more data and analysis are required to make any definitive judgment. a related line of research explores a critical link between student, content mastery and local context. following the work of cole (1996, 1998) current efforts address the relevance of researching student learning in context. in studying these environments and their implications for student learning, cole argues that assessment of cognitive ability is contextually dependent - that is, the further removed an experiment or study is from the domain of use/ application, the less applicable the result is to that domain. this same notion has been reported in a somewhat different form in physics. studies, such as those using the force concept inventory (fci), report that while students may perform well in a traditional physics course, a course in which they have managed to master formulae and various mathematical procedures, the students miss the broader setting and conceptual basis for the discipline of physics (redish, 2003). the first step in this exploration has been to develop a model of how context may be brought into the present research discussions on student learning of physics content (finkelstein 2001; 2004). lastly, a rich area for investigation is the effect such a course has on students crossing disciplinary boundaries, and in particular, whether education can become a legitimate pursuit for physicists. there is evidence that this course helped students cross disciplinary boundaries. of the six undergraduate physics majors in this study, four enrolled in teacher education programs. three of these four enrolled in the university's teacher education program (tripling the annual enrollment of physics majors). a fifth student took a year abroad to teach and ultimately returned to enroll in an education program. of the four graduate students enrolled, one took a post to in the physics department to direct the undergraduate laboratories. another two were active participants in the american association of physics teachers-sponsored graduate training program, preparing future physics faculty (pfpf), offered by the department. a more in-depth longitudinal study would be required to observe the longer term and broader impacts of the course. however, these preliminary signs indicate that by presenting the opportunity to explore and seriously consider education as a pursuit within the physics department, students begin to do just that. d. institutional response --can this activity survive? the institutional response to this new activity system has not been a simple or monolithic process, and is worthy of a detailed research study in its own right; however, the program has initially met with some success. in seymour sarason's framework this coordinated set of activities constitutes a new setting, or a new and sustained relationship between individuals (sarason 1989; 1997). sarason contends that program success or failure depend critically upon two factors: the initial structure of the program and the adaptation of that structure to local conditions. after extensive setup (and lobbying of the department and other institutional entities), the course on teaching and learning physics was incorporated into the institutional structure. the department of physics has adopted the course as an upper division restricted elective in the sequence of classes required for a bachelor’s degree. the teacher education program offers the class as part of its certification program. additionally, this class is the first course to be cross listed between the physics department and the teacher education program. in establishing and maintaining this course, an inter-disciplinary team has gathered to critique and help shape the course. following the establishment of the course, the director of teacher education preparation and the vice-chair of physics engaged in collaborative discussions surrounding the development of a new undergraduate physics and education major. while the new major was not constructed, the course continues to the present, some several years later. as sarason suggests, finkelstein, n. d. journal of scholarship of teaching and learning, vol. 4., no. 2, september 2004. 14 the creation of a new setting, one that often begins with multidisciplinary work, necessarily affects the local, disciplinary, and interdisciplinary cultures in which the new setting is created (sarason 1989). however, as sarason goes further noting that these systems all-too-often depend upon a single individual and hence may not be able to adapt when that individual leaves (sarason 1997). the course on teaching and learning physics was offered by the author three times before leaving the institution. each of the first three sessions was offered voluntarily (that is, with funding support coming from a national science foundation fellowship rather than the department). with an eye to handing the course over, a second faculty member participated in the course the second time it was offered. currently, course is under the supervision of the new instructor and been offered twice, with the support of the department. there is some indication that given the success of the course, it may continue; however, the difficulty of relying upon a single individual's efforts and advocacy remains-when the current instructor retires or moves on, it is not clear that the course will continue. beyond the local or micro-level, the course is part of a larger activity system of community partnership. as described, fieldwork is an integral component of the course, and as such, has required the development and strengthening of ties with community partners. the community agencies which host student-teachers from the university course, such as local schools and boys and girls clubs, have indicated great interest in the continuation of collaborative efforts. the community partners greatly value the added human resources of student-experts who participate in local activities, and in several cases used these added resources to develop new educational programs. without the involvement of the undergraduates in the outreach process, two of the four community-based programs would not have operated. meanwhile the community-based programs serve as necessary resources for the university students and researchers who use these environments as laboratories for studying pre-college student learning. in this way, it is not simply a matter of the university delivering outreach and programming, but rather a collaborative arrangement whereby both partners develop and benefit from the interaction. community-university partnership programs, using this model, continue to expand both in size and scope (into more schools and at more educational levels). an unexpected outcome of offering the course was newfound collaboration with one of the local two-year colleges. the chair of the local community college physics department participated in the class on teaching and learning physics the first time it was offered. through the course and following years, members from both the community college system and the university system explored mechanisms to increase transfer rates from the two-year college into the university, and in particular, into physics. one project that stemmed from these discussions was the augmentation of the graduate training program, preparing future physics faculty (pfpf), with the opportunity for graduate students to teach students at the community college. using materials from the course on teaching and learning physics, graduate students developed and offered a new course to community college students. the first two offerings of the course were successful at providing graduate students valuable and authentic teaching experiences, exposing students at the two year college to physics and the university culture, and increasing the transfer of students from the community college to the university. once again, however, whether these ties will be sustained with the absence of key individuals is unclear. one of the graduates of the pfpf program continues talks with the community college; however, the joint program has not been offered since the author left the university. v. conclusions. finkelstein, n. d. journal of scholarship of teaching and learning, vol. 4., no. 2, september 2004. 15 the presented model for coordinating physics education, research, and community partnerships may be adopted more broadly within the (science) education community by substituting different content. there is nothing particular to physics, nor undergraduates in this model. the domain of examination could equally well have been newtonian mechanics, or physical chemistry. the outcomes would be similar: increased student interest and ability in the science domain, increased attention to and interest in teaching and education, and development of community partnerships. furthermore, the activity system provides a rich opportunity for science education research that is tightly coupled with and informed by educational reforms. as institutions of higher education begin to develop programs of discipline-based education research within the sciences (science departments, and in particular physics departments are hiring an increasing number of faculty into new lines physics education research / reform), this type of activity system provides an avenue to leverage local interest in reform, research, and community partnership. because such a system addresses the multiple motives of physics, education, and outreach, the hope is that each domain would support the activity and would develop an authentic interest in sustaining a coordinated program. of course, such change is local and depends simultaneously upon fertile ground (local or bottom-up support) and healthy conditions for growth (top-down support). references aps (1999). american physical society. statement 99.2: research in physics education, adopted may 21, 1999. http://www.aps.org/statements/99.2.html astin, a.w., vogelgesang, l. j., ikeda, e.k., and yee, j a. (2000). how service learning affects students. los angeles, ca: higher education research institute, university of california, los angeles. available at: http://www.gseis.ucla.edu/slc/rhowas.html brown, a.l. (1992). design experiments: theoretical and methodological challenges in creating complex interventions in classroom settings, journal of the learning sciences, 2(2), 141-178. brown, a.l. and campione, j.c. (1990). communities of learning and thinking, or a context by any other name, human development, 21, 108-126. brown, j.s,. collins, a. and duguid,p. (1989). situated cognition and the culture of learning, educational researcher, 18(1), 32-42. cole, m. (1996) cultural psychology: a once and future discipline. cambridge, ma,.: harvard university press. cole, m. (1998). can cultural psychology help us think about diversity? mind, culture and activity, 5(4), 291-304. disessa, a.a. (1988). knowledge in pieces, in forman & puffall (ed.) constructivism in the computer age. hillsdale nj: lawrence erlbaum assoc. finkelstein, n. (2001). context in the context of physics and learning, physics education research conference proceedings. rochester, ny: perc publishing. finkelstein, n. d. journal of scholarship of teaching and learning, vol. 4., no. 2, september 2004. 16 finkelstein, n. (2003). coordinating physics and education instruction, journal of college science teaching, 33(1), 37-42. finkelstein, n. (2004, to appear). learning physics in context: a study of student learning about electricity and magnetism, international journal of science education gentner, d., and gentner, d.r. (1983). flowing waters for teeming crowds: mental models of electricity, in gentner and stevens (ed.) mental models. hilsdale, nj : lawrence erlbaum associates. p. 99. hake, r.r. (1998). interactive-engagement versus traditional methods: a six-thousand-student survey of mechanics test data for introductory physics courses, american journal of physics 66, 64-74. halliday, d.. resnick,r., and walker, j. fundamentals of physics, 5th ed. (1997). new york, ny: john wiley and sons, inc. hammer, d. (1994). students’ beliefs about conceptual knowledge in introductory physics. international journal of science education., 16(4) 385-403. hieggelke, c. maloney, d., . van heuvelen, a., o'kuma, t., (2001). the conceptual survey in electricity and magnetism (csem). american journal of physic. 69(s1), s12-s23. hogan, d.m. and tudge, j.r.h. (1999). implications of vygotsky’s theory for peer learning, in o’donnell, a.m., & king, a. (ed.) cognitive perspectives on peer learning. mahwah, nj : erlbaum, ,p. 39-65. mcdermott, l.c. and redish, e.f. (1999). resource letter: per-1: physics education research. american journal of physics, 67(9),755-767. mcdermott, l.c. and schaffer, p.s. (1992). research as a guide for curriculum development: an example from introductory electricity parts i&ii,. american journal of physics, 60(11), 9941013. mcdermott, l.c. shaffer, p.s. and the physics education research group (2002). tutorials in introductory physics. upper saddle river, nj: prentice hall. papert, s. (1991). situating constructionism, in harrel and papert, constructionism new york, ny: ablex publishing corporation. posner, g.j., strike, k.a, hewson, p.w. and gertzog, w.a. (1982). accommodation of a scientific conception: toward a theory of conceptual change, science education, 66(2), 211227. redish e.f. (2003). teaching physics with physics suite. new york, ny: wiley. reif, f. (1986). scientific approaches to science education, physics today. 39(11), 48-54. sarason, s. b. (1997). revisiting the creation of settings, mind culture and activity, 4(3), 175. sarason, s.b. (1989). the creation of settings and the future societies (san francisco, ca: jossey-bass publishers, finkelstein, n. d. journal of scholarship of teaching and learning, vol. 4., no. 2, september 2004. 17 schmidt, w.h. mcknight, w.h. cogan, l.s., jakwerth, p.m. and houang, r.t. (1999). facing the consequences: using timss for a closer look at u.s. mathematics and science education. hingham, ma : kluwer academic publishers, schoenfeld, a.h. (1986). what’s all the fuss about metacognition? in a.h. schoenfeld, cognitive science and mathematics education. hillsdale, nj: lawrence erlbaum. slch (2003). service learning clearing house at the university of california, los angeles. available online: http://www.gseis.ucla.edu/slc/ sokoloff, d.r, thornton, r.k., laws, p.w. (1998) real time physics, new york, ny: wiley. star, s.l. and griesemer, j.r. (1989). institutional ecology, ‘translations’ and boundary objects: amateurs and professionals in berkeley’s museum of vertebrate zoology, 1907-39. social studies of science, 19, 387-420. timss (1999). the third international mathematics and science study (timss) international study center at: http://timss.bc.edu/ microsoft word hoyertfinal.doc journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006, pp. 1 13. a brief intervention to aid struggling students: a case of too much motivation? mark sudlow hoyert and cynthia d. o’dell1 abstract: this study examines two interventions for altering achievement goals in an effort to enhance academic success in struggling introductory psychology students. the procedures involved an in-class lecture and an interactive computerized tutorial. both procedures were successful in altering motivation. this led to changes in study activities and increased examination and course grades. in fact, the rate of failure was cut in half in the students who completed the interventions. however, the procedures were not as successful at attracting students who were at risk for academic failure. only 12% of the target students completed the first intervention while 55% of the target students completed the computerized tutorial. participation in the computerized intervention was enough to markedly reduce the number of failing grades earned by the class as a whole. additions and alterations to the interventions are proposed to extend their efficacy. keywords: achievement goals, goal orientation, academic success, mastery, performance i. introduction many teachers firmly believe that motivation is important to the learning process. most teachers believe that a minimal amount of motivation is necessary and that higher amounts of motivation are better. teachers often attribute problems in the learning process to a lack of motivation. in this paper, we explore some of the links between motivation and academic success. our work is informed by a theory, alternatively known as achievement motivation theory or goal orientation theory, which posits that multiple varieties of motivation exist and that moderate levels of some varieties are preferable to high levels of other varieties. in fact, some of the problems in the learning process may result from high levels of particular kinds of motivation. goal orientation theory suggests that when students engage in a class, they strive to reach one or more goals (ames & archer, 1988; dweck & leggett, 1988; harackiewicz, barron, & elliot, 1998; hidi & harackiewicz, 2000). two goals are of primary importance: mastery goals and performance goals. students who adopt mastery goals are interested in learning the material in the class and strive to master that material (ames, 1992; blumenfeld, 1992; dweck & leggett, 1988). students who pursue performance goals are interested in demonstrating their competence, especially relative to other students (dweck & leggett, 1988; urdan, 1997). mastery and performance goals are multidimensional motivational constructs. that is, these goals provide a framework through which a variety of behavioral, cognitive and affective responses are energized and directed (ames, 1992; dweck & leggett, 1988; blumenfeld, 1992). for instance, research under a variety of laboratory and classroom settings has found that students who pursue mastery goals display a wide variety of largely adaptive behaviors and 1 respectively, professor of psychology, indiana university northwest, mhoyert@iun.edu, and associate professor of women’s studies and psychology, indiana university northwest, codell@iun.edu. address correspondence to mark hoyert, psychology department, indiana university northwest, gary, in 46408, 219-980-6687. hoyert, m.s. and o’dell, c.d. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 2 attitudes not seen in all students. mastery oriented students seek to improve their competence through acquiring new skills and knowledge and by surmounting novel and difficult problems. mastery goals have been found to be associated with increased interest, the enjoyment of challenge and challenging tasks, and the belief that competence is obtained incrementally through effort. students who pursue mastery goals commonly use effective learning strategies such as elaboration and organization, have developed multiple strategies, are interested in developing new skills, become involved in the learning process, display greater persistence, and are likely to respond to challenges through the use of greater effort and the exploration of alternative learning strategies (albaili, 1998; ames, 1992; ames & archer, 1988; blumenfeld, 1992; elliot & harackiewicz, 1994; graham & golan, 1991, harackiewicz, barron, carter, lehto, & elliot, 1997; harackiewicz, barron, tauer, carter, & elliot, 2000; harackiewicz & elliot, 1998; pintrich, zusko, schiefele, & pekrum, 2001). the pursuit of performance goals is more complex, but sometimes has been associated with a less adaptive set of academic outcomes such as self-aggrandizing, task aversion, the pursuit of effort minimizing strategies, a reluctance to seek help, impaired problem solving, greater feelings of self-consciousness, self-handicapping, and helplessness. students who adopt a performance goal orientation seek to elicit favorable judgments of their competence and avoid negative evaluations. these individuals tend to prefer and to seek out easier tasks where success and validation can be obtained and view competence as static and unaffected by effort. the maladaptive behaviors are more likely to appear when validation is not available (albaili, 1998; dweck, 1999; dweck & leggett, 1988; graham & golan, 1991; harackiewicz, barron, & elliot, 1998; harackiewicz, barron, tauer, carter, & elliot, 2000, harackiewicz & elliot, 1993; kong & hau, 1996; midgley, 1993; pintrich & schunk, 1996; pintrich, zusko, schiefele, & pekrum, 2001; ryan, gheen, & midgley, 2000; somuncuoglu & yildirim, 1999; urdan, kneisel, & mason, 1999; urdan & maehr, 1995; urdan, midgely, & anderman, 1998). the adoption of mastery goals should be associated with academic accomplishment. in the college classroom, this would translate into better examination and course grades and a higher grade point average. one would expect that a mastery oriented student who monitors comprehension, connects new information with old, can discriminate more important information from less important information, who sets goals, uses elaboration and other adaptive learning strategies should attain higher levels of academic success than a performance oriented student who procrastinates and self-handicaps. however, most studies employing an objective measure of academic performance in the classroom have not found a consistent or robust advantage. only a third of these studies have reported higher grades for mastery oriented students (bouffard, boisvert, vezeau, & larouche, 1995; eppler & harju, 1997; pintrich, zusko, schiefele, & pekrum, 2001; schraw, horn, thorndike-christ, & bruning, 1995; wolter, yu, & pintrich,1996) and about two thirds have obtained null results (beck, rorer-woody, & pierce, 1991; elliot & church, 1997; harackiewicz & elliot, 1998; harackiewicz, barron, carter, lehto, & elliot, 1997; harackiewicz, barron, tauer, carter, & elliot, 2000; harju & eppler, 1997; greene & miller, 1996; pintrich & garcia, 1991; pintrich & garcia, 1993; roedel & schraw, 1995). no studies have found that pursuing mastery goals is associated with declining academic success. we have identified a set of conditions under which the relative pursuit of mastery and performance goals plays a potent role in academic success. one goal orientation theory (dweck, 1999; dweck & leggett, 1988) suggests that the effects of these goals can be altered or amplified during stress or challenge. we can identify an obvious and frequent source of stress in hoyert, m.s. and o’dell, c.d. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 3 our students, failure on an examination. to see how relevant achievement goals are in this situation, we tracked students to see what happened on the subsequent examination after an objective failure. students who endorsed mastery orientation enjoyed a 15-point increase on the next examination. students who pursued performance goals suffered a 10-point decrease. not only did different patterns of goal orientation lead to an effect on academic performance, but it was very robust and consistent. a 25-point difference in examination scores can lead to a difference of two or three letter grades. further, ninety-five percent of the mastery-oriented students attained a higher examination grade on the subsequent test. about half of the performance-oriented students had a decrease on their test grade on the subsequent examination (hoyert & o’dell, 1999; 2000; 2001a; 2001b; 2004a, 2004b; o’dell & hoyert, 2000, 2002). as a result, we developed an intervention to aid struggling introductory psychology students. we measured goal orientation at the onset of the semester and invited students who primarily pursued performance goals to attend brief tutorial sessions after they had failed an examination. student peers led the sessions. the goal of the intervention was to alter goal orientation in order to prevent the decrement sometimes seen in highly performance-oriented students. the tutors’ coached students on a variety of techniques including orientation modeling from several different perspectives, discussion of multiple study techniques, goal setting, and value referencing. the intervention provided students with knowledge about and practice expressing mastery goals. it must be pointed out that the tutorials only address motivational issues. we did not cover classroom material. the intervention produced clear benefits: students in the intervention began to endorse mastery goals to a greater extent and earned higher examination and course grades. the average student achieved a 15-point increase on their next examination. further, the effects persisted over the duration of the semester even though the intervention ended. the d/f failure rate was nearly cut in half (to 47%). these improvements are even more impressive when their performance is compared to that of the control participants. one could view the control group as a predictor of what could have happened to the intervention group participants without the intervention. these two groups of participants had the same goal orientation profile and the same grades on the first examination. both sets of students endorsed performance goals more than mastery goals. after the first examination, the control students= grades declined precipitously. the failure rate (d, f) was 78%. the successes of this intervention are heartening. however, the procedure reached a relatively small number of students and was labor intensive. over the course of three semesters we were able to involve a little over 60 students. we believe that there are many more students who could benefit from the intervention. in any particular class, we estimate that about half of the students are predominantly performance-oriented and that about half of the students who are predominately performance-oriented will fail at least one examination. this amounts to several hundred students each year at our school. in this research, we explored two techniques for extending the intervention to reach all struggling performance-oriented students enrolled in sections of introductory psychology in an effort to alter the success profile in the class as a whole. ii. method a. instruments goal orientation was measured using a locally developed inventory modeled after roedel, schraw, and plake’s (1994) goals inventory. this instrument consists of 24 statements hoyert, m.s. and o’dell, c.d. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 4 rated on a 5-point likert-type scale for strength of agreement. students were asked to consider how much each of the statements applies to themselves within the introductory psychology class. the statements assess attitudes and behaviors towards learning and performance goals as described by dweck and leggett (1988). b. procedure intervention 1: guest lecture. during the first day of the semester and after the final examination, the goal inventory was administered to all students. one hundred eighteen students were enrolled in the class. many of these students experience an academic challenge over the course of the semester. eleven students did not take the first examination. twenty-eight students received a failing grade on the first examination. sixty students failed at least one examination over the course of the semester. eighty-three students received a grade that was less than a c on at least one examination over the course of the semester. the intervention occurred during the class immediately after the first examination and took the form of a lecture and discussion. in the intervention, we provided information about goals and the meaning of failure, and practiced setting mastery goals. all of the exercises were aimed at increasing the adoption of mastery orientation. the techniques included orientation modeling from several different perspectives, discussion of multiple study techniques, goal setting, and value referencing. following the class, students were instructed to write a paper defining the two goals, describing individuals who exemplify the traits of each goal, and considering their own experience with goal orientation. seventy-two students attended the class and 19 wrote and submitted the paper. at the end of the academic term, the students’ introductory psychology examination and course grades were obtained from the instructor. intervention 2: interactive computerized tutorial. this intervention occurred with a different class during a different semester. following the first examination all students received a cd rom containing the goals inventory and the intervention. the intervention provided students with information about goals and the meaning of failure and practiced setting mastery goals. it took the students 35 to 55 minutes to complete the tutorial. two hundred-twelve students were enrolled in the class. many of these students experienced an academic challenge during the semester. eighty-two failed the first examination. one hundred fifty-six failed at least one examination over the course of the semester. one hundred eighty-three received a grade that was less than a c on at least one examination over the course of the semester. one hundred thirty-seven (65% of the class) students completed the intervention; 75 did not. there were 147 students who were performance-oriented and failed at least one examination. eighty-two out of the 147 (56% of the group) target students completed the intervention. iii. results a. intervention 1 nineteen students completed the motivational intervention. nine of these were not struggling or were not high-performance oriented students. ten were students that we predict could struggle in the class. in this study, we are primarily interested in struggling students. we predict that a student will be at risk for academic failure when they pursue performance goals more than learning goals and when they fail an examination or are otherwise challenged. we will refer to these students as the “target” students. because the 10 struggling students in the hoyert, m.s. and o’dell, c.d. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 5 intervention group and the 9 other students responded to the intervention approximately the same we will display the group data because it is based on more observations. ninety-nine students did not participate in the intervention. of these 72 were at risk for failure because of grades and goal orientation. thus 12% of the target students completed the intervention. the students who completed the intervention did not differ from the students who did not complete the intervention in goal orientation or grades on examination 1. the students who wrote the paper had a mean mastery score of 3.69 (sd=0.79) (both goal orientation scores can range from 1 to 5). the students who did not write the paper had a mean mastery goals score of 3.86 (sd=0.88). these did not differ (t(94)=.612, p=.542). the same pattern holds for performance goals (intervention: m=3.59, sd=0.61; comparison: m=3.75, sd=1.06; t(94)=.425, p=.672). the scores on the first examination also did not differ (intervention group: m=74.63, sd=14.00; comparison: m=71.18, sd=14.19, t(105)=0.963, p=.338). because the two groups were similar, the students who did not complete the intervention can be treated as a convenience control group or a comparison group. the intervention produced an increase in the pursuit of mastery goals. before the intervention the mean was 3.69 (sd=0.79) and after it had increased to 3.78 (sd=0.54) however, the increase was not statistically significant (t(18)=0.527, p=.604). the intervention also produced a decrease in performance goals (first day: m=3.86, sd=0.89; last day m=3.77, sd=0.92; t(18)=.749, p=.463). in contrast, students who had not completed the intervention displayed dramatic decreases in both mastery (first day: m=3.58, sd=0.55; last day m=3.18, sd=1.00; t(37)=2.40, p=.021) and performance (first day: m=3.76, sd=1.07; last day m=3.37, sd=0.70; t(37)=2.31, p=.026) goals over the course of the semester. students who completed the intervention displayed steadily but slightly increasing examination grades (exam 1 m=74.63, sd=14.00, exam 2 m=75.89, d=14.60, exam 3 m= 76.44, sd=12.53, exam 4 m=80.94, sd=12.79, exam 5 m=78.09, sd=10.11). the gradual increase is not statistically significant (f(4,64)=1.11, p=.358). the scores on examination 1 for students who did not complete the intervention were very similar to the students who completed the intervention. however, after the first examination, the grades for the students who did not complete the intervention declined precipitously (exam 1 m=71.18, sd=14.25, exam 2 m=68.33, sd=15.98, exam 3 m= 65.42, sd=13.77, exam 4 m=64.75, sd=19.96, exam 5 m=57.96, sd=13.57). this decline is significant (f(4,144)=6.01, p<.001). the overall pattern of increasing grades for the students who completed the intervention is different from the pattern of decreasing grades for the comparison students (f(1,204)=3.42, p=.010). figure 1 presents the difference in mean examination grades between students who completed the intervention and those who did not. on examination 1 the difference was less than 3 points. with each subsequent exam, the disparity widened. by examination 5, there was a 20-point difference between the test scores of the two groups. the intervention also led to improvements in their final course grades. table 1 displays the percentage of grades in the two groups. we compared the semester average for the two groups. students who completed the intervention earned higher average grades than the other students (intervention: m=78.9, sd=19.47, comparison: m=46.89, sd=27.99; t(116)=4.75, p<.001). because we are primarily interested in the effects of the intervention on the target students who are highly performance-oriented and who fail an examination, we report their scores separately. the pattern of results was similar for these target students as for the overall sample. the intervention led to slight increases in mastery goals (first class: m=3.62, sd=0.83; hoyert, m.s. and o’dell, c.d. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 6 figure 1: difference in examination scores between intervention and comparison groups 0 5 10 15 20 25 1 2 3 4 5 examination m ea n di ff er en ce be tw ee n sc or es table 1: grade distributions for students completing intervention 1 and for the comparison group grade a b c d f/w completed intervention 26.3 31.6 31.6 5.3 5.5 comparison group 5.2 15.6 17.7 15.6 45.8 last class: m=3.74, sd=0.53; t(17)=0.594, p=0.560) and slight decreases in performance goals (first class: m=3.89, sd=0.90; last class: m=3.78, sd=0.84; t(17)=0.868, p=0.398). target students who failed an examination and did not complete the intervention demonstrated considerable decreases in mastery goals (first class: m=3.58, sd=0.55; last class: m=3.18, sd=1.00; t(17)=2.40, p=0.021), and performance goals (first class: m=3.76, sd=1.07; last class: m=3.37, sd=0.70; t(17)=2.31, p=0.026). as can be seen in table 2, the target students who completed the intervention also maintained higher grades. table 2: grade distributions for performance-oriented students completing intervention 1 and for the comparison group grade a b c d f/w completed intervention 0.0% 30.0% 60.0% 0.0% 10.0% comparison group 0.0% 6.8% 23.3% 19.2% 50.7% hoyert, m.s. and o’dell, c.d. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 7 b. intervention 2 one hundred thirty-seven students completed the motivational intervention. fifty-five of these were not struggling or were not high-performance oriented students. we predict that 82 of these students would struggle in the class because of goal orientation. seventy-five students did not participate in the intervention. of these, 66 were “at risk” because of grades and goal orientation. thus about 55% of the target students completed the intervention. the scores of students who completed the intervention did not differ from the students who did not complete the intervention on examination 1 (intervention, m=54.5, sd=14.12; comparison, m=55.0, sd=14.56; t(197)=0.23, p=0.749). because the two groups were similar, the students who did not complete the intervention can be treated as a comparison group. the intervention produced an increase in the pursuit of mastery goals in both the target students as well as the general body of students who completed the intervention. since our primary interest is in the target students we will present these data first. before the intervention the mean was 4.10 (sd=0.09) and after it had increased to 4.29 (sd=0.10) (t(80)=3.34, p=0.002). the intervention also produced a decrease in performance goals. before the intervention, the mean was 4.07 (sd=0.42) and after it had decreased to 3.06 (sd=0.91) (t(80)=8.69, p<0.001). target students who completed the intervention displayed steadily increasing examination grades (exam 1 m=54.4, exam 2 m=55.9, exam 3, m= 56.0, exam 4 m=57.9, exam 5 m=55.6). in contrast, the students who did not complete the intervention earned increasingly lower grades (exam 1 m=55.0, exam 2 m=47.3, exam 3, m= 43.8, exam 4 m=37.7, exam 5 m=31.0). the grade on examination 1 for the two groups is not statistically significant. however, the overall pattern of increasing grades for the students who completed the intervention and decreasing grades for the comparison group is significant (f(1,109)=43.06, p<0.001). figure 2 presents the difference in mean examination grades between students who completed the intervention and those who did not. on examination 1, the students who did not complete the intervention maintained a higher mean exam grade than the students who would later complete the intervention. after completing the intervention, the intervention students outscored the comparison group by nine points. with each subsequent exam, the disparity widened. by examination 5, there was a 25-point difference between the two groups. figure 2: difference in examination scores between intervention and comparison groups -5 0 5 10 15 20 25 1 2 3 4 5 examination m ea n di ff er en ce be tw ee n sc or es hoyert, m.s. and o’dell, c.d. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 8 the intervention also led to improvements in their final course grades. the dfw rate in introductory psychology is typically similar to the percentage of students who received an f on any one test. in this class the percentage of students who received an f was 74%. thus, we could expect a very high dfw rate. the comparison group resembled this historical pattern. table 3 reports that 76% of these students failed the class. in contrast, in the intervention group, only 37% of the students received an f for the course. the pattern of grades in the intervention group is statistically different from the comparison group (x2(4)=12.38, p<0.001). table 3: grade distributions for students completing intervention 2 and for the comparison group grade a b c d f/w completed intervention 1.2% 9.9% 21.0% 30.9% 37.0% comparison group 0.0% 1.5% 9.1% 13.6% 75.8% we are also interested in the overall effect on grades in the class. table 4 displays the effects on course grades. given the historical pattern and the pattern of goals and examination scores we would expect that 156 students would have earned less than a c. after the intervention, we found that 123 students received a d, an f, or withdrew. table 4: overall effect on the grade distribution students completing intervention 2 and the predicted grade distribution a, b, or c d, f, or w intervention 42.0% 58.0% predicted 26.4% 73.6% iv. discussion teachers are frequently puzzled by the disparate reactions of students to challenge. after failing an examination some students will react with despair and will give up. other students will react to the same challenge by buckling-down, increasing their effort, and developing better study habits. the study of goal orientation seems to provide some insight into patterns of reactions such as these (dweck, 1999; dweck & leggett, 1988; hoyert & o’dell, 1999; 2000; 2001a; 2001b; 2004a, 2004b; o’dell & hoyert, 2000, 2002). the interventions assessed here attempt to encourage students who may have been prone to despair to adopt a more positive approach to challenge. we have now attempted three variants of the intervention. the first involved peer-tutors. it was not successful in reaching large numbers of students. however, when students participated, they enjoyed changed goals and increased academic success (hoyert & o’dell, 2004b). to an extent, goal orientation theory suggests a reason for the lack of participation. highly performance-oriented students tend to avoid evidence of a lack of competence. to a hoyert, m.s. and o’dell, c.d. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 9 performance-oriented student, seeking help from a tutor is an admission that they are not capable of completing the task without assistance, something that a truly competent student could have done. as a result, we developed two variants of the intervention that can be embedded in the course structure so that participation will not carry any perceived stigma. providing the intervention through a guest lecture was similar in effect to the peer-tutors. not very many students were willing to participate. however, when they completed the intervention, they enjoyed a changed pattern of goals, greater persistence, and higher grades. one unique finding occurred during this particular class. mastery and performance goals tend to remain rather stable over time. typically, when we measure these goals at the beginning and the end of a class without an intervention, they are approximately the same for each student and for the class as a whole. in this class, goals changed dramatically. both mastery and performance goals declined significantly over the semester. we have not observed this before and suspect that it may have been related to messages provided by the instructor or to the pattern of academic success experienced by the students (hoyert & o’dell, 1999; 2000; 2001a; 2001b; 2004a, 2004b; o’dell & hoyert, 2000, 2002). given the backdrop of declining goals, the increase in mastery goals observed in the students who completed the intervention provides strong evidence for the efficacy of the intervention. the intervention occurred once, early in the semester, and lasted about 45 minutes. however, the effects of the intervention persisted for the duration of the semester. it is noteworthy that the average examination grades of these students increased over the course of the semester in much the same way as the mastery goals. the students who completed the intervention passed the class and earned high grades at far higher rates than the students who did not. the major drawback of this intervention is that it did not attract many students. only 12% of the target students completed the intervention. one of our goals in this research is to determine whether we can alter the pattern of academic success in a whole class. because we did not attract many students to complete the intervention, we were not able to decrease the proportion of students who did not pass the class (dfw rate). the computerized tutorial has been the best procedure for involving students in the intervention to date. over half of the students who we predicted could be at risk for academic failure participated in the intervention. after completing the brief intervention, they pursued mastery goals to a greater extent and were less interested in pursuing performance goals. the change in goals led to changes in studying and increased academic success over the duration of the semester. one of our interests is in trying to decrease the rate of failure in individual students. the students who completed the intervention earned failing course grades at less than half the rate of the historical pattern and at less than half the rate of the comparison group. another interest is in altering the rate of failing grades in an entire class. it appears that the intervention was successful in decreasing the rate of the f grade by about 25%. the present interventions produce positive results. however, they still can be improved. both procedures could be more fully integrated into the class structure to encourage more students to participate. for instance, making the programs mandatory components of the classes could be explored. in addition, the intervention has only been attempted within introductory psychology classes. an interesting extension could be to try to introduce the intervention in other introductory level courses in other disciplines. the success profiles in many mathematics and science classes are lower than introductory psychology. this would suggest that they are prime candidates for this type of intervention. hoyert, m.s. and o’dell, c.d. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 10 finally, this study contained a troubling methodological flaw. both interventions may have a selection problem. the students determined who would participate in the intervention. we have identified goal orientation and challenge as important variables in course success. we have observed that these variables are similar in the two groups. thus, we believe that it is reasonable to use these students as a comparison group. to a certain extent, this was unexpected in the first intervention. however, we would predict that in the event of a low rate of participation, more learning-oriented students would participate in the intervention than performance-oriented students. regardless, because students could determine participation, it could be that the differences in grades, persistence, and changes in grades are related to the selfselection. another important future study could involve a procedure in which students are randomly assigned to the intervention or to one of several control conditions. we are optimistic that these changes will further strengthen our findings that the intervention positively alters goal orientation, examination scores and course grades in a college setting. references albaili, m. (1998). “goal orientations, cognitive strategies and academic achievement among united arab emirates college students.” educational psychology, 18, 195-203. ames, c. (1992). “classrooms: goals, structures, and student motivation.” journal of educational psychology, 84, 261-271. ames, c., & archer, j. (1988). “achievement goals in the classroom: students’ learning strategies and motivation processes.” journal of educational psychology, 80, 260-267. aronson, j., & fried, c. (1998). “reducing stereotype threat and boosting academic achievement of african americans: the role of conceptions of intelligence.” unpublished manuscript. beck, h. p., rorer-woody, s., & pierce, l. g. (1991). “the relations of learning and grade orientations to academic performance.” teaching of psychology, 15, 35-37. blumenfeld, p.c. (1992). “classroom learning and motivation: clarifying and expanding goal theory,” journal of educational psychology, 84, 272-281. bouffard, t., boisvert, j., vezeau, c., & larouche, c. (1995). “the impact of goal orientation on self-regulation and performance among college students.” british journal of educational psychology, 65, 317-329. dweck, c. s. (1999). self-theories: their role in motivation, personality, and development. philadelphia: psychology press. dweck, c. s. & leggett, e. l. (1988). “a social-cognitive approach to motivation and personality.” psychological review, 95, 256-273. elliot, a. j., & church, m. a. (1997). “a hierarchical model of approach and avoidance achievement motivation.” journal of personality and social psychology, 72, 218-232. hoyert, m.s. and o’dell, c.d. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 11 eppler, m. a. & harju, b. l. (1997). “achievement motivation goals in relation to academic performance in traditional and non-traditional college students.” research in higher education, 38, 557-573. graham, s., & golan, s. (1991). “motivational influences on cognition: task involvement, ego involvement, and depth of information processing.” journal of educational psychology, 83, 187-194. greene, b. a., & miller, r. b. (1996) “influence on achievement: goals, perceived ability, and cognitive engagement.” contemporary educational psychology, 21, 181-192. harackiewicz, j. m., barron, k. e., carter, s. m., lehto, a. t., & elliot, a. j. (1997). “predictors and consequences of achievement goals in the college classroom: maintaining interest and making the grade.” journal of personality and social psychology, 73, 1284-1295. harackiewicz, j. m., barron, k. e., & elliot, a. j. (1998). “rethinking achievement goals: when are they adaptive for college students and why?” educational psychologist, 33, 1-21. harackiewicz, j.m., barron, k.e., tauer, j. b., carter, r. a., & elliot, a. j. (2000). “short-term and long-term consequences of achievement goals: predicting interest and performance over time.” journal of educational psychology, 75, 183-205. harackiewicz, j.m., & elliot, a.j. (1993). “achievement goals and intrinsic motivation.” journal of personality and social psychology, 65, 904-915. harackiewicz, j.m., & elliot, a.j. (1998). “the joint effects of target and purpose goals on intrinsic motivation: a mediational analysis.” personality and social psychology bulletin, 24, 675-689. harju, b. l. & eppler, m. a. (1997). “achievement motivation, flow and irrational beliefs in traditional and nontraditional college students.” journal of instructional psychology, 24, 147157. hidi, s. & harachiewicz, j.m. (2000). “motivating the academically unmotivated: a critical issue for the 21st century.” review of educational research, 70, 151-179. hoyert, m.s. & o’dell, c.d. (1999). “achievement motivation in traditional and non-traditional students.” poster presented at the annual meeting of the midwestern psychological association, april 29 may 1, 1999, chicago, il. hoyert, m.s. & o’dell, c.d. (2000). “achievement motivation and the impostor phenomenon.” poster presented at the annual meeting of the midwestern psychological association, may 5, 2000, chicago, il. hoyert, m.s. & o’dell, c.d. (2001a). “goal orientation and challenge in college students.” paper presented at a symposium entitled “toward a scholarship of teaching and learning in developmental psychology” at the biennial meeting of the society for research in child development, april 10-22, 2001, minneapolis, mn. hoyert, m.s. and o’dell, c.d. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 12 hoyert, m.s. & o’dell, c.d. (2001b). “achievement motivation and excuse making.” poster presented at the annual meeting of the midwestern psychological association, may 3-5, 2001, chicago, il. hoyert, m. s., & o’dell, c.d. (2004). “the impostor phenomenon and goal orientation.” journal of the indiana academy of social sciences, 7, 1-10. hoyert, m.s., & o’dell, c.d. (2004). “altering achievement motivation to enhance classroom learning.” selected papers from the 15th international conference on college teaching and learning, 15, 119-135. kong, c., & hau, k. (1996). “student achievement goals and approaches to learning: the relationship between emphasis on self-improvement and thorough understanding.” research in education, 55, 74-85. midgley, c. (1993). “motivation and middle level schools”, in advances in motivation and achievement (vol. 8, pp. 217-274). greenwich, ct: jai press. o’dell, c.d., & hoyert, m.s. (2000). “achievement motivation and academic success in traditional and non-traditional aged college students.” poster presented at the 16th biennial meeting of the conference on human development, april 14-16, 2000, memphis, tn. o’dell, c. d., & hoyert, m.s.(2002). “altering achievement motivation and the response to failure.” poster presented at the ctup creative classroom poster session held during the 73rd annual meeting of the midwestern psychological association. pintrich, p. r., & garcia, m. (1991). “student goal orientation and self-regulation in the college classroom.” in advances in motivation and achievement (vol. 11, pp. 123-158). greenwich, ct: jai press. pintrich, p. r., & garcia, m. (1993). “intraindividual differences in students= motivation and self-regulated learning.” german journal of educational psychology, 7, 99-107. pintrich, p. r., & schunk, d. h. (1996). motivation in education: theory, research, and applications. englewood cliffs, nj: prentice hall. pintrich, p. r., zusho, a., schiefele, u. & pekrun, r. (2001). “goal orientation and selfregulated learning in the college classroom: a cross-cultural comparison.” in f. salili, c. y. chiu & y. y. hong (eds.), student motivation: the culture and context of learning. new york: plenum. roedel, t. d. & schraw, g. (1995). “beliefs about intelligence and academic goals.” contemporary educational psychology, 20, 464-468. roedel, t. d., schraw, g., & plake, b. s. (1994). “validation of a measure of learning and performance goal orientations.” educational and psychological measurement, 54, 1013-1021. hoyert, m.s. and o’dell, c.d. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 13 ryan, a.m., gheen, h. h., & midgley, c. (2000). “why do some students avoid asking for help? an examination of the interplay among students’ academic efficacy, teachers= socialemotional role, and the classroom goal structure.” journal of educational psychology, 90, 528535. schraw, g., horn, c., thorndike-christ, t., & bruning, r. (1995). “academic goal orientations and student classroom achievement.” contemporary educational psychology, 20, 359-368. somuncuoglu, y. & yildirim, a. (1999). “relationship between achievement goal orientations and use of learning strategies.” journal of educational research, 92, 267-277. wolters, c. a., yu, s. l., & pintrich, p. r. (1996). “the relation between goal orientation and students= motivational beliefs and self-regulated learning.” learning and individual differences, 8, 211-238. urdan, t. c. (1997). “examining the relations among early adolescent students’ goals and friends’ orientation toward effort and achievement in school.” contemporary educational psychology, 22, 165-191. urdan, t. c., kneisel, l., & mason, v. (1999). “interpreting messages about motiation in the classroom: examining the effects of achievement goal structures.” in advances in motivation and achievement (vol. 11, pp. 123-158). greenwich, ct: jai press. urdan, t.c., & maehr, m. l. (1995). “beyond a two goal theory of motivation: a case for social goals.” review of educational research, 65, 213-243. urdan, t., midgley, c., & anderman, e. (1998). “the role of classroom goal structure in students= use of self-handicapping strategies.” american educational research journal, 35, 101-122. journal of the scholarship of teaching and learning, vol. 8, no. 1, february 2008, pp. 111 – 135. a preliminary investigation of pedagogical content knowledge and techniques in the teaching of spanish to native speakers vickie r. ellison1 the author has withdrawn this article. 1 department of modern and classical languages, kent state university, 109 satterfield hall, kent, ohio 44278, vellison@kent.edu. responding to student feedback 1 running header: responding to student feedback improving the instruction of engineering calculus: responding to student feedback barbara m. moskal colorado school of mines bmoskal@mines.edu 303-273-3867 an earlier version of this paper titled, "using student feedback to improve instruction in engineering calculus", appears in the proceedings of the frontiers in education conference, kansas city, mo, 2000. acknowledgements teri woodington supported many of the electronic resources discussed here and barbara b. bath designed the engineering calculus sequence at csm. responding to student feedback 2 abstract the purpose of this article is to illustrate how student feedback was used for instructional improvement in a sequence of engineering calculus courses. the methods that are employed here are appropriate for other classrooms and disciplines. this article describes the instruction that the students received and the feedback that the students provided. this feedback was used to design the next mathematics course that these students completed. after completing the next course, the students were asked to provide feedback on the changes that had been made. index terms calculus, classroom assessment, classroom research, college mathematics, course evaluations responding to student feedback 3 improving the instruction of engineering calculus: responding to student feedback in an article that was written for pre-college teachers, i proposed a model (moskal, 2000a) of the classroom assessment process that consisted of four phases: planning, gathering, interpreting and using. this model is equally appropriate for college level instruction and is shown in figure 1. the vertical columns divide the model into the phases of assessment and the rectangles represent the outcome of each phase. the primary mediators of each phase are distinguished in the model by circles. each phase of the assessment process implies an action on the part of the instructor (i.e., the instructor plans, gathers, interprets, and uses) and each concludes with an outcome(s). figure 1. model of the classroom assessment process assessment instru ment student response student •feedba ck •cla ssroom decision making instru ctor instru ctorinformationinstru ctor planning gathering interpreting using instru ctor the planning phase includes the processes of selecting or developing assessment items and assembling these items into an instrument. the outcome of this phase is the attainment of an assessment instrument. currently there is a large body of information that is available to assist college instructors in selecting appropriate assessment instruments. these instruments may be designed to assess individual performances, group performances or the effectiveness of a course (angelo & cross, 1993; brookhart, 1999; lewis, aldrige & swamidass, 1998; mehta & schlecht, responding to student feedback 4 1998; moskal, 2000b; moskal, knecht & pavelich, 2001; shaeiwitz, 1998). on-line databases are also available that can assist college instructors in finding assessment instruments that meet their classroom needs (brisseden & slater, 2001; southern illinois university, edwardsville, 2001). the gathering phase begins when the instructor administers the selected assessment instrument to students. the students then interpret the requests of their instructor and the tasks as they construct their responses. unlike the previous phase, which is mediated by the instructor, the gathering phase is primarily mediated by the student. although the professor administers the task to the students, it is the student who controls what appears in the response. the interpretation phase consists of the instructor's efforts to make sense of students' responses and results in the acquisition of information. the interpretation phase is supported by the use of measurement tools (e.g., scoring rubrics or checklists) and often the application of statistical techniques (angelo & cross, 1993; deek et. al., 1999; mcneil, bellamy & burrows,1999; moskal, 2000b; moskal & leydens, 2000). the assessment event, which is a single pass through the assessment process, concludes with the application of the acquired information to serve particular purposes. the use phase may have several outcomes. two commonly identified uses of classroom assessment information are to assist instructors in making appropriate instructional decisions (angelo & cross, 1993; brissenden & slater, 2001; brookhart, 1999) and to enable them to provide accurate feedback to their students (brissenden & slater, 2001; brookhart, 1999; shaeiwitz, 1998). how the information will be used should be considered in the planning phase in order to guide the selection of appropriate assessment instruments. current research (angelo & cross, 1993; brookhart, 1999; moskal, knecht & pavelich, 2001) has emphasized the importance of completing the entire assessment process, which includes responding to student feedback 5 the use phase. it is during this final phase that instructional improvements take place. using assessment information for instructional improvement is one of the most important and the most frequently neglected components of classroom assessment (angelo & cross, 1993; brookhart, 1999). in many colleges, a common assessment practice that is designed to evaluate the effectiveness of a course is the administration of a student survey at the end of the semester. one purpose of this survey is to allow students to provide feedback that may be used to improve instruction in future courses. since faculty often teach a different course the next semester, the information that is acquired at the end of a course may not be useful in the refinement of the next course. additionally, if the selection of the assessment instrument is completed by the institution, the given questions may not be relevant to the instructional needs of the course instructor. poor evaluations are also often explained by the respective instructor as being a result of unmotivated students, heavy teaching loads or an invalid rating systems (lucus, 1999). in addition, faculty have (coburn, 1984) argued that students lack the technical expertise to evaluate course content or instructional style. this, they explain, may result in an over emphasis on the evaluation of the teachers' popularity rather than their teaching ability. concerns have also been raised that an over emphasis on course evaluations results in grade inflation and a reduction in amount of material that is covered in a given course (wilson, 1998). a great deal of research has been completed that examines the validity and reliability of student course evaluations. cashin (1995) has reported that more than 1500 articles and books have been written that address the development, design and appropriateness of student evaluations. based on these resources, he determined that well-designed course evaluations can provide valid and reliable results. other researchers have provided support for this claim (e.g., coburn, 1984; peterson responding to student feedback 6 & kauchak, 1982). researchers (brookhart, 1999; howard & maxwell, 1980; scriven, 1995) have also found that higher grades do not necessarily result in higher course evaluations. in other words, many of the concerns that have been raised with respect to course evaluations appear to be unfounded. a well-designed student evaluation system can produce valid and reliable results. the purpose of this paper is to describe a two-course sequence of engineering calculus at the colorado school of mines (csm) and to illustrate how student feedback was used for instructional improvement. the first course is honors engineering calculus ii, which covers vectors, vector functions, partial derivatives and multiple integrals. the students admitted to this course are first semester freshmen who scored a 4 or 5 on the ab advanced placement test (bath, 1999). the next semester, these same students completed honors engineering calculus iii, which covers vector calculus, sequence and series, and an introduction to differential equations. a variety of different instructional techniques were used in the first course and the students were asked to evaluate these techniques through a course evaluation. this information was used to guide the development of the next course, resulting in the completion of the assessment event. the students in the next course were asked to evaluate the impact of these changes —thus, beginning a new assessment event. honors engineering calculus ii this section describes the structure of honors engineering calculus ii, the evaluation techniques used in that course and the results of the evaluation. students of the 35 students who completed honors engineering calculus ii, 7 students were female, 1 student was international and 1 student was of asian decent. the remaining students were caucasian. responding to student feedback 7 textbook the textbook was calculus concepts and contexts by james stewart (1998). according to the preface of the text, it is designed to focus upon the development of students' conceptual understanding. this is achieved through a combination of geometric, numerical and algebraic approaches and the application of technology to problem solving situations. course design honors engineering calculus ii is a four-credit course. during the semester of interest, the class met for one hour on monday and wednesday and two hours on friday. on monday and wednesday, a modified lecture format was used in which the students were encouraged to actively participate by asking questions and offering suggestions. physical objects were brought to class to illustrate many of the concepts (e.g., a wire was used to illustrate a space curve and a ball was used to illustrate the concepts underlying the calculation of the surface area of a sphere). on friday, the students met for two hours in the computer lab to solve problems in teams of three or four students. sometimes the problems required the use of the computer program, mathematica, and other times they did not. the students' ability to manipulate physical objects and their ability to use mathematica were not evaluated on exams. these activities were designed to deepen the students' conceptual understanding as concepts were introduced. the students were required to submit the completed mathematica assignments. due to the rigorous structure of the course, there was very little time to answer questions on the assigned homework or to give in-class quizzes on the material. for this reason, the solutions to the homework were made available in the library and the quizzes were completed as take-home assignments. responding to student feedback 8 web-based support throughout the course, electronic media were used to support the learning process. lecture notes and solutions to quizzes were posted on the web. an electronic discussion group was maintained. tests and solutions from prior years were made available electronically. students had access to their instructor via e-mail. the option was also available for students to provide anonymous feedback to their instructor via e-mail at any point during the semester. many of these resources could have been made available in a paper format; however, by using the web, the overall expense of distributing this information was reduced. closed response survey at the end of honors engineering calculus ii, the students were asked to complete a survey in which they rated the extent to which each of the instructional techniques impacted upon their learning process. the four-point scale ranged from "no impact" to "significant impact." students were asked not to include personal identification on the survey and to indicate the grade that they expected to receive in the course. short response survey the short response survey is administered at the end of each semester in all departmental courses. in the current course, the short response survey was completed before the closed response survey. the questions that comprise this instrument are as follows: 1. what aspects of instruction did you find effective for promoting your learning? 2. what recommendations would you make that would improve the instruction that you received in this course? 3. if you have any additional comments, please write them in the space below. responding to student feedback 9 closed responses survey: across students table 1 displays the activities that the students evaluated. a higher average rating suggests stronger student agreement that the given activity positively impacted their learning. responses that indicated that a given activity was "not applicable" were not included in this analysis. the highest rated course component was the electronic availability of solutions to prior tests via the web. the other components of the course that were rated as having a "strong impact" were classroom instruction, the three unit tests, the textbook, access to information concerning the course via the instructor's web page and the take-home quizzes. these were closely followed by group work, availability of course notes on the web and the use of manipulatives in class. the activities in the course that were rated as having "no impact" or a "slight impact" on student learning were: the use of the computer program, mathematica; the availability of the electronic discussion group; the availability of providing electronic anonymous feedback to the instructor via the web and the availability of solutions to take-home quizzes on the web. closed response survey: within grade categories at the start of the survey, the students were asked to indicate the grade that they expected to receive in the course. thirteen, sixteen and six of the students expected to receive an "a", "b", and "c", respectively. the actual assignment of grades resulted in 11, 17 and 7 students receiving an "a", "b", and "c", respectively. since the student predicted distribution closely approximated the actual distribution of grades, it is likely that the student predicted grades were accurate indicators of the actual grade that they attained. the final grades in this course were high, which is not surprising given the demanding screening process to enter the course. responding to student feedback 10 table 1 ratings of instructional techniques in calculus ii questions mean "a" "b" "c" availability of previous tests and solutions on the web. 3.26 (n=34) 3.46 (n=13) 3.20 (n=15) 3.00 (n=6) classroom instruction. 3.23 (n=35) 3.15 (n=13) 3.31 (n=16) 3.17 (n=6) the three chapter tests. 3.20 (n=35) 3.23 (n=13) 3.25 (n=16) 3.00 (n=6) textbook. 3.09 (n=34) 2.69 (n=13) 3.27 (n=15) 3.50 (n=6) access to course information via instructors' web page. 3.03 (n=34) 2.92 (n=13) 3.07 (n=15) 3.17 (n=6) take-home quizzes. 3.00 (n=35) 3.15 (n=13) 2.81 (n=16) 3.17 (n=6) group work. 2.97 (n=35) 3.23 (n=13) 2.69 (n=16) 3.17 (n=6) availability of course notes on the web. 2.94 (n=35) 2.92 (n=13) 2.94 (n=16) 3.00 (n=6) concrete manipulatives (physical objects). 2.89 (n=35) 2.77 (n=13) 3.06 (n=16) 2.67 (n=6) homework assignments. 2.79 (n=33) 2.67 (n=12) 2.93 (n=15) 2.67 (n=6) availability of solutions to homework problems. 2.72 (n=33) 2.17 (n=12) 3.13 (n=15) 2.83 (n=6) access to your instructor via electronic mail. 2.26 (n=31) 2.08 (n=12) 2.31 (n=13) 2.50 (n=6) availability of solutions to take-home quizzes. 2.08 (n=31) 2.08 (n=12) 1.93 (n=14) 2.00 (n=5) ability to provide electronic anonymous feedback. 1.85 (n=29) 1.82 (n=11) 1.75 (n=12) 2.17 (n=6) electronic discussion group. 1.79 (n=33) 1.85 (n=13) 1.73 (n=15) 1.80 (n=5) the use of the computer program, mathematica. 1.29 (n=35) 1.23 (n=13) 1.31 (n=16) 1.33 (n=6) responding to student feedback 11 out of the students who expected to receive an "a" in the course, the highest rated component of the course was the availability of solutions to prior tests on the web. this was followed by group work and chapter tests. instruction and take-home quizzes were also highly rated. for the students who expected to receive a "b" in the course, instruction was rated highest and was closely followed by the textbook, the chapter tests and the availability of previous test solutions on the web. for the students who expected to receive a "c" in the course, the highest rated component of the course was the textbook. this was followed by classroom instruction, take-home quizzes, group work, and access to information concerning the course via the instructors web page. the only component of the course that was consistently rated in the top five across groups was instruction. across all three groups, the lowest rated component of the course was the use of the computer program, mathematica. across all three groups, the availability of the solutions to takehome quizzes on the web, the electronic discussion group, the availability of providing electronic anonymous feedback to the instructor via the web, and access to the instructor via e-mail were rated in the bottom five course components. short response survey: students' written comments the students' written comments provided further insight into why a given component of the course was or was not effective for promoting learning. thirty-four out of thirty-five students completed the short response survey. twelve students explained that classroom instruction was greatly enhanced by the visual aids. one student wrote, "props (straws, balls, wire) are very effective in visualizing in 3d" and another student wrote, "the visual aids were always helpful as well as 'entertaining'." although the students had not rated the manipulatives as highly as they had responding to student feedback 12 rated instruction on the closed response survey, their comments indicated that these activities had contributed to the high rating of instruction. another component of the course about which the students frequently commented was the availability of the notes on the web. fifteen students commented on the effectiveness of this approach. one student stated, "the notes on the web is the biggest help." although the students did not explain on the short response survey why this was useful, several students had stated during the semester that by printing the notes out before class they could spend class time listening rather than "frantically writing." the students not only provided comments on what was effective, they also made suggestions as to how to improve the course. overall, the students had highly rated the group work on fridays. four students had provided positive comments on the short-response survey on the effectiveness of the group work for promoting their learning. however, 13 students complained either that there was a need for more in class instruction or that the time spent in groups was too long. their reactions indicated that although group work was helpful, it may have been overdone. for example, one student explained, "more time allotted for difficult concepts. we are sometimes pressured for time as we only have 2 hours in the classroom a week". the same student suggested, "3 hours in the classroom/1 hour in lab [group work]." another student complained, "it is thrown at us for 2 days, then we get tested. do we need to spend every friday just on w/sheets? [group activities]." three of the students complained about the take-home quizzes, "friday quizzes are too long [group work]. take home quizzes are even longer." however, 8 students commented on the effectiveness of this technique for promoting their learning. one student stated, "the quizzes, i hated doing, but they really helped me learn." in other words, the majority of the student responses supported the effectiveness of this technique. responding to student feedback 13 anonymous feedback via e-mail over the semester, i received three anonymous messages; one contained a sequence of nonsense letters and the statement, "wanna learn gibberish?" and another student wrote, "if i did my homework, i would be much better off. so i think i will do some homework this weekend." the remaining message complained extensively about the amount of work that was required in the course. although the given student was obviously unhappy, the feedback that he or she provided was not helpful for improving the course. honors engineering calculus iii this section describes the changes that were made in the next course, calculus iii, and the results of the student evaluations to these changes. students i had 32 students in calculus iii. twelve (38%) of these students had been in my class the previous semester. nine students were female and one student was african american. the remaining students were caucasian. changes calculus iii, which was also a four-credit course, met four times a week. in response to the students' recommendations, i reduced group work to one hour a week and allowed more in-class time for questions. i continued to place my course notes on the web and use concrete materials to illustrate the concepts that were being addressed. my course web page provided the students with links to my notes, other instructors' notes, solutions to prior tests and solutions to quizzes. although the students had indicated that the availability of the solutions to quizzes had only a minimal impact on their learning, maintaining this resource took very little time and it provided one form of responding to student feedback 14 feedback to my students on how to solve the problems. i stopped supporting the electronic discussion group and the option of providing electronic anonymous feedback. mathematica was also eliminated. follow-up surveys an altered version of the closed response survey from the previous semester was administered at the end of calculus iii. the questions that referenced mathematica, the electronic discussion group, and the option of providing anonymous feedback were eliminated from the survey. additionally, the students were asked to indicate whether they had been in my class the previous semester. four questions were added to the survey in which the students rated the changes that had been made on a four-point scale that ranged from "very bad change" to "very good change." the students also had the option of indicating that they had no opinion. closed response survey: across students twenty-nine students completed the closed response survey and their responses are summarized in table 2. when a response indicated that a given activity was "not applicable", it was eliminated from the analysis. three of the activities that had been rated in the top five during the previous semester were rated in the top five during the current semester, i.e., chapter tests, availability of previous tests and solutions on the web and instruction. the students rated the take-home quizzes as having the strongest impact on their learning experience. group work, the three chapter tests, the availability of previous tests and solutions on the web, classroom instruction and access to information concerning the course via the instructors' web page were rated as having had a "strong impact" on learning. the remaining activities were rated as having had at least a slight impact on the student learning; none of the activities were rated as having "no impact" on learning. responding to student feedback 15 table 2 ratings of instructional techniques in calculus iii questions mean "a" "b" "c" take-home quizzes. 3.66 (n=29) 3.64 (n=11) 3.67 (n=15) 3.67 (n=3) group work. 3.41 (n=29) 3.45 (n=11) 3.40 (n=15) 3.33 (n=3) the three chapter tests. 3.10 (n=29) 3.27 (n=11) 3.00 (n=15) 3.00 (n=3) availability of previous tests and solutions on the web. 3.07 (n=28) 3.18 (n=11) 3.14 (n=14) 2.33 (n=3) classroom instruction. 3.07 (n=29) 3.45 (n=11) 2.80 (n=15) 3.00 (n=3) access to course information via instructors' web page. 3.00 (n=26) 3.10 (n=10) 3.00 (n=14) 2.50 (n=2) homework assignments. 2.97 (n=29) 3.09 (n=11) 2.93 (n=15) 2.67 (n=3) textbook. 2.86 (n=29) 3.00 (n=11) 2.73 (n=15) 3.00 (n=3) concrete manipulatives (physical objects). 2.65 (n=29) 2.73 (n=11) 2.60 (n=15) 2.67 (n=3) availability of course notes on the web. 2.52 (n=27) 2.30 (n=10) 2.79 (n=14) 2.00 (n=3) availability of solutions to take-home quizzes. 2.43 (n=28) 1.90 (n=11) 2.67 (n=15) 3.50 (n=2) availability of solutions to homework problems 2.18 (n=22) 2.00 (n=9) 2.17 (n=12) 4.00 (n=1) access to your instructor via electronic mail. 2.00 (n=21) 2.25 (n=8) 1.92 (n=12) 1.00 (n=1) responding to student feedback 16 as discussed earlier, four questions had been added to this survey in which the students were asked to evaluate the changes that had been made since the previous semester. only students who had been in my class the previous semester were included in this analysis. nine students responded that the elimination of the computer program mathematica was either a good change (n=2) or a very good (n=7) change. four students indicated that the elimination of the discussion group was a good change and 4 students indicated that this was a bad change. only 3 students responded to the question concerning the electronic anonymous feedback to the instructor and all three indicated that this was a good change. in response to the question concerning the reduction of group work, 1 student indicated that this was a "very bad change", 9 students indicated that this was a "bad change" and 1 student indicated that this was a "good change". closed response survey: within grade categories as was done the previous semester, the students were asked to indicate the grade that they expected to receive in the course. eleven, fifteen and three of the students expected to receive an "a", "b", and "c", respectively. the actual assignment of grades resulted in 8, 13, 10 and 1 students receiving an "a", "b", "c" and "d", respectively. based on this distribution, many students over predicted the actual grades that they would receive. take-home quizzes were rated as the activity that had the greatest impact on learning by both students who expected to receive an "a" and students who expected to receive a "b" in the course. in all three groups, take-home quizzes, group work and the three chapter tests were rated in the top five activities. the lowest rated activity by the students who expected to receive an "a" was the availability of solutions to take-home quizzes on the web. the lowest rated activity for students who expected to receive a "b" or "c" in the course was access to their instructor via electronic mail. responding to student feedback 17 short response survey: students' written comments in general, the comments that the students provided with respect to the course were favorable. this is illustrated in the following examples, "enjoy your teacher and its easier to learn," and "i liked going to calculus this semester. the class wasn't just a regular old boring lecture." as had been the case in the previous semester, the written comments also indicated the aspects of instruction that had supported their learning process. nine students commented on the effectiveness of the use of manipulatives. two of these students suggested that even more visual demonstrations be made, i.e., "more visual aids!" and "more toys!" although many students had indicated that the reduction of group work was a bad change on the closed response survey, only 3 students commented on this component of the course on the short response survey. one student indicated, "i think that the course should be held 3 days a week with one two hour lab section, like calc 2 honors." this was the only comment that strongly supported returning to the previous course design. the remaining comments indicated that the group work that had been completed during the current course had been useful. by reducing the group work, i had more time in class to devote to student questions. six students commented that this was an important component of their learning experience, e.g. "i think it's amazing that you can spend as much time answering the homework questions as you do and still get through all the material!" and "she is always willing to answer questions." eleven students also indicated that the notes on the web continued to be useful. concluding remarks an important component of the assessment process is using the information that is acquired for instructional improvement purposes. in this study, i had the opportunity to collect information from my students and use the information to design the next course in the sequence. the changes responding to student feedback 18 that i made were: 1) the reduction of group work, 2) the elimination of mathematica, 3) the elimination of the discussion group and 4) the elimination of the option of providing electronic anonymous feedback to the instructor. both the elimination of mathematica and the option of providing electronic anonymous feedback to the instructor were well received by the students. their reactions to the other two changes were met with mixed results. i continue to believe that mathematica or some other 3 dimensional graphing software could have a positive impact on my students' understanding of calculus concepts. in interpreting my students' negative responses to this program, i have concluded that it was my method of implementation that was ineffective. i spent very little time introducing the software and assumed that my students would be able to use this tool effectively. based on student feedback, this is not what happened. in the future, i intend on reintroducing mathematica into my classroom. this time, i will do so slowly and with more careful attention to my students' learning needs. another change that i had made in my classroom was a reduction in the amount of group work. this provided me the opportunity to increase the time that was devoted to students' questions. although having more time for questions was well received, the reduction in group work was not. the students' responses suggest that they wanted more time for questions and more opportunities to work in groups. increasing both of these activities is not feasible without increasing the time in class. during the semester that the course discussion group was available, i logged-in on a regular basis and responded to the students' questions. when this activity was eliminated, i had more time for planning class and organizing the course web site. the students had highly rated both classroom instruction and the course web site. in other words, for each of the changes that were made, there were tradeoffs of which the students were unaware. responding to student feedback 19 in order to determine when the benefits outweighed the drawbacks, i needed to move beyond the student responses and consider how the evaluation was completed. the short response survey was administered before the closed response survey. this ordering was purposeful. the closed response survey directs the students to the specific changes that had been made and asks the students to evaluate the impact of each change. the short response survey allows the students to select what they will discuss. if i had administered the closed response survey first, the students' responses to the short response survey may have mirrored the concerns that had been raised through the closed response survey. in other words, the closed response survey could have directed the students to consider specific issues. by administering the short response survey first, i hoped to capture the concerns that were foremost in the students' minds. in calculus iii, only one student indicated that there was an inadequate amount of group work on the short response survey. none of the students' recommended the reintroduction of the discussion group. the students needed to be directly asked about these changes before commenting on their impact. this suggests that these issues were not pressing concerns for the majority of students. coupling this observation with the overall positive comments that were made to the short response survey, supports the assertion that overall the changes had improved the course. another observation that can be made through this study is the value of combining information that is collected through different forms of assessment. the importance of using multiple sources of data has been given a great deal of attention in the assessment literature (angelo & cross, 1993; brissende & slater, 2001; brookhart, 1999). it was through the combination of the information that was provided through the closed response survey and the short response survey that i was able to make-sense of what was and was not working within the given courses. after changes were implemented, it was through the combination of information acquired through the two surveys responding to student feedback 20 and through the examination of the process used to administer the surveys that i was able to determine the extent to which the given changes had been effective. responding to student feedback 21 references angelo, t.a., & cross, k. p. (1993). classroom assessment techniques: a handbook for college teachers (2nd edition). san francisco, ca: jossey-bass. bath, b. (1999). using student feedback to improve instruction in engineering calculus. unpublished manuscript. brissenden, g. & slater, t. (2001). field-tested learning assessment guide: for science, math, engineering and technology instructors [on-line]. available: http://www.wcer.wisc.edu/nise/cl1/flag/default.asp?startpage=flag1.asp. brookhart, s. (1999). the art and science of classroom assessment: the missing part of pedagogy. ashe-eric higher education report (vol. 27, no.1), washington, dc: the george washington university, graduate school of education and human development. cashin, w. e. (1995). "student rating of teaching: the research revisited". idea paper, (32) [eric document reproduction service no. ed 402 338]. coburn, l. (1984). "student evaluation of teacher performances." eric/tme update series [online]. available: http://ericae.net/edo/ed289887.htm. deek, f. hiltz, s. r., kimmel, h. & rotter, n. (1999). "cognitive assessment of students' problem solving and program development skills." journal of engineering education, 88 (3), 317-326. howard, g. & maxwell, s. (1980). "correlation between student satisfaction and grades: a case of mistaken causation." journal of education psychology, 72 (6), 810-820. lewis, p., aldridge, d. & swamidass, p.m. (1998). "assessing teaming skills acquisition on undergraduate project teams." journal of engineering education, 87 (2), 149-155. lucas, a. (1999). "reaching the unreachable: improving the teaching of poor teachers." the department chair. bolton, ma: anker. responding to student feedback 22 mcneill, b., bellamy, l. & burrows, v. (1999). "a quality based assessment process for student work products." journal of engineering education, 88 (4), 485500. mehta, s. & schlecht, n. w. (1998). "computerized assessment techniques for large classes." journal of engineering education, 87 (2), 167-172. moskal, b. (2000a) "an assessment model for the mathematics classroom." mathematics teaching in the middle school, 6 (3), 192-194. moskal, b. (2000b). "scoring rubrics: what, when and how?" practical assessment, research & evaluation, 7 (3) [on-line]. available : http://ericae.net/pare/getvn.asp?v=7&n=3. moskal, b., knecht, r. & pavelich, m. (2001). "the design report rubric: assessing the impact of program design on the learning process". journal for the art of teaching: assessment of learning, 8 (1), 18-33. moskal, b. & leydens, j. (2000). "scoring rubric development: validity and reliability". practical assessment, research & evaluation, 7 (10)[on-line]. available: http://ericae.net/pare/getvn.asp?v=7&n=10. peterson, k & kauchak, d. (1982). teacher evaluation: perspectives, practices, and promises. salt lake city, ut: utah university, center for educational practices. [eric document reproductive services no. ed 233 996]. scriven, m. (1995). "student ratings offer useful input to teacher evaluations." practical assessment, research & evaluation, 4 (7) [on-line]. available: http://wricae.net/pare/getvn.asp?v=4&n=7. shaeiwitz, j. a. (1998). "classroom assessment." journal of engineering education, 87 (2), 179181. responding to student feedback 23 stewart, j. (1998). calculus concepts and contexts. washington, dc: brooks/cole publishing company. southern illinois, edwardsville (2001). classroom assessment techniques [on-line]. available: http://www.siue.edu/~deder/assess/catmain.html. wilson, r. (1998). "new research casts doubt on value of student evaluations of professors". the chronicle of higher education, 44 (9), a12-a14. microsoft word 1206 revised.doc journal of the scholarship of teaching and learning, vol. 12, no. 3, september 2012, pp. 62 – 72. students’ misconceptions in psychology: how you ask matters…sometimes annette kujawski taylor1 and patricia kowalski1 abstract: misconceptions about psychology are prevalent among introductory students. just how prevalent and what can be done to change these misconceptions depends on valid methods of assessment. the most common method of assessment, the true/false questionnaire, is problematic. the present study compared true/false with forced choice formats to determine whether the formats give different estimates of student misconceptions. introductory psychology students (n = 165) answered 39 misconceptions in both the true/false and forced choice formats. students differed in accuracy when assessed with the different formats, with 33.05% accuracy for true/false and 41.29% accuracy for forced choice. in the analyses of individual items we observed that some items did not differ in level of accuracy across formats and other items did differ. we conclude that the true/false method of assessing misconceptions may overestimate students’ level of misconception and recommend continued attention to how researchers assess misconceptions. keywords: assessment, psychology, student misconceptions i. introduction research in cognitive psychology has much to say about how people learn (bransford, brown, & cocking, 2000). we know that meaningful learning is an active process. it involves the use of prior knowledge to help make sense out of new information. when prior knowledge supports or fits with new information, then the new information can be learned more easily. however, when prior knowledge contradicts new information—as when students have misconceptions—it actually makes learning the new information more difficult than if there were no prior knowledge at all (lipson, 1982). for over a decade we have been investigating the misconceptions that students bring with them to the introductory psychology classroom (see taylor, kowalski, & laggren 2000 and taylor & kowalski, 2004). consistently, we find low levels of accuracy in students’ beliefs, ranging from 30% to 39% accuracy. in the time since our first studies, a number of other studies have reported similar findings (amsel, et al., 2009; kuhle, barber & bristol, 2009; lilienfeld, 2010). students come into the introductory psychology course with strongly held beliefs that are in contrast to the preponderance of the scientific evidence, and, unless there is some strong intervention to challenge these false beliefs, they blissfully leave the introductory psychology course with those beliefs still intact. furthermore, examples of misconceptions that do not change based on simple instruction have been documented across a wide variety of academic disciplines such as earth science (harackiewicz, 1999), physics (baser, 2006; dykstra, boyle, & monarch, 1992), biology (boyes & stanisstreet, 1991), political science (lorenzo, 1999) and philosophy (williamson, 2007).                                                                                                                           1 department of psychological sciences, university of san diego, san diego, ca 92110, taylor@sandiego.edu taylor, a.k. & kowalski, p. journal of the scholarship of teaching and learning, vol. 12, no. 3, september 2012. josotl.indiana.edu 63 as scholar/teachers of psychological science we believe this low level of psychological science literacy is not an acceptable outcome. reflecting on the tenacity of some students’ misconceptions, our primary focus has shifted from enumerating beliefs and their sources to whether we can change students’ false beliefs, how we can effect such change, why we should do so, and for how long or short a time such changes endure. in investigating how students might be encouraged to change their misconceptions, we have followed the lead of the science education and reading research literatures (guzzetti, snyder, glass, & gamas, 1993; strike & posner, 1992). much of this literature suggests that to alter a misconception, students need to understand that what they think they know does not match the current scientific view and need to be presented with a plausible and useful alternative in a way they can understand. the strategy of presenting the misconception followed by a clear discussion of the scientifically correct view is referred to in the reading research as refutational text or refutational teaching (guzzetti et al.). by using refutational texts and lectures in our introductory psychology course, we found that we significantly reduced student misconceptions and that students maintained much of this change over time (kowalski & taylor, 2009; taylor & kowalski, 2010). despite the desire to move forward in understanding how to change student misconceptions, we realize we may first need to step back. if we want to study factors that promote change in misconceptions we need a meaningful way to measure those beliefs. at a classroom level, angelo & cross (1993) recognized the importance of assessing student misconceptions as a way of informing instruction and suggested creating simple questionnaires, focusing on a number of troubling beliefs students are likely to hold. within various disciplines, researchers have developed diagnostic instruments to assess such deficient understanding. the force misconceptions test (eryilmaz, 2002) and the chemistry concepts inventory (mulford & robinson, 2002) are examples of such tools. both instruments rely on multiple choice questions to sample the extent of alternative concepts in the discipline, and to study changes that occur following course instruction. within the field of psychology, the current methods of measuring misconceptions in psychology appear to be problematic. psychological misconception assessments have been criticized for including outdated items, items that are not important in the science, and items that reflect claims that introductory texts do not cover (griggs & ransdell, 1987; lamal, 1979). in addition, most studies have used a true/false approach in which students read a series of statements and designate each statement as being true or false. although many criticisms of true/false testing may be misconceptions themselves (see burton, 2005), it is generally agreed that true/false questions are difficult to write and that because of the role of guessing, true/false assessments make it difficult to know whether student responses reflect incorrect knowledge or simply a lack of knowledge. this distinction between incorrect knowledge and no knowledge is particularly important for misconception assessment because the intent is to identify student misinformation that may be tenacious and in need of being forcefully refuted. there are additional aspects of the true/false assessments that make them particularly problematic in the study of misconceptions. because of the nature of most misconceptions, these true/false assessments tend to phrase all items as “false” in order to be answered correctly—in other words, the misconception is a false statement students tend to believe. with all of the items needing to be correctly responded to as “false,” it is difficult to know the degree to which responses reflect actual beliefs or reflect a demand characteristic, a response bias, or some other artifact of the test’s format rather than the test’s content. sax (1989) noted that when students guess on true/false questions they tend to mark the statement “true”. a “true” response set, taylor, a.k. & kowalski, p. journal of the scholarship of teaching and learning, vol. 12, no. 3, september 2012. josotl.indiana.edu 64 therefore, would overestimate student misconceptions. as an alternative to the true/false only questionnaire, gardner & dalsing (1986) found that by just adding an option for “i don’t know,” they reduced the level of reported misconceptions, presumably more accurately reflecting the students’ level of misconceptions. perhaps the strongest criticism of misconception assessments is that the questions include claims that are not clearly countered by the evidence, are poorly worded, or may hold a “grain of truth.” as mckeachie (1999) noted, it is difficult to write true/false questions that are clearly true or false. brown (1984) and ruble (1986) levied such a criticism of the true/false format for psychological misconception questionnaires. both researchers provided evidence that the constraint of developing a single, brief statement about some complex behavior leaves most true/false statements of misconceptions ambiguous. frequently such brief statements are only partly true and partly false. for example, many students accept as true what is known as the 10% myth, the statement that “most people only use 10% of their brains.” researchers in the psychological sciences clearly consider this statement false. however, outside of the research community people may have various interpretations of the word “use” in the sentence. it could be interpreted as meaning that at some point in time some people only use 10% of their available processing capacity, preferring to be lazy or in a deep stage of sleep, even if most of the brain itself is in constant use to maintain bodily functions and homeostasis. the ambiguity in interpreting true/false items becomes more obvious when one tries to write true versions of false statements. a straightforward “true” version of the 10% item could be, “few people only use 10% of their brains.” or “most people use 90% of their brains.” neither of these options is a satisfactory “true” contrast to the “false” statement in that each could be read as suggesting there are some people who do use only 10% of their brains. the problem with true/false assessments must be addressed before researchers can begin to evaluate the effectiveness of different pedagogies to effect change in misconceptions. we need an instrument that is reliable, valid, and sensitive enough to detect change without the ambiguity inherent in the true/false statements. in the present study we compared two formats which addressed the same content. a format which bensley and colleagues have recently developed is a 2-item forced choice format (bensley & lilienfeld, 2010; bensley, lilienfeld, ferree, powell, & southerly, 2011; southerly, bensley, lilienfeld, ferree, & powell, 2011). the 2-item (a/b) forced choice format generally has a true/false statement as one option (usually the misconception) along with a complementary statement of the contrasting viewpoint based on the preponderance of the evidence. for the example above of the 10% myth, we adopted a version of the item taken from bensley et al. (2010), “which is most true about how much of the brain people typically use? a. people use all of their brains, but not all at once. b. most people only use about 10% of their brains.” in this case, the two options do not exactly negate each other; however, the alternative provides a more accurate representation of what the evidence shows, in contrast to the incorrect option, the one which most students incorrectly mark as “true” in the true/false format (see higbee & clay, 1998; kowalski & taylor, 2009). the forced choice format, therefore, assesses whether the student both accepts the correct claim and rejects the incorrect claim when presented with the two alternatives simultaneously. we therefore compared a true/false version with a force choice version of a test of misconceptions in psychology. we wanted to know whether there would be different estimates of introductory psychology students’ misconceptions across the two formats. we expected that items that show similar patterns of response across the two formats more closely represent what we might consider to be a “misconception.” in other words, if students pick the incorrect forced taylor, a.k. & kowalski, p. journal of the scholarship of teaching and learning, vol. 12, no. 3, september 2012. josotl.indiana.edu 65 choice response at the same rate as they designate the false statement as being “true,” then we believe that the item more clearly reflects a genuine level of misconception among the students. however, if students when presented with two options show a greater tendency to favor the more correct option, even if they incorrectly designate a false statement as being “true” in the true/false format, then that pattern of response suggests that the estimated level of misconception is tied to how we ask rather than to students’ underlying beliefs for the present study we asked these research questions: is there an overall tendency for students to respond differently to questions regarding psychological misconceptions when they evaluate true/false items compared with forced choice items that simultaneously include the incorrect and correct alternatives? when students agree with an item in the true/false format, is the agreement a result of the item format, or because they have underlying false beliefs? evidence that students’ responses are related to item format would be suggested by a difference in the level of false belief between the true/false and force choice formats. ii. method. a. participants. the participants for this study included 164 introductory psychology students who completed the two questionnaires. only 155 students completed both questions all the way to the end, so that the final analysis included only the 155. we did not collect demographic information; however, the 164 students comprised approximately 80% of the individuals in the introductory psychology subject pool. the individuals in the subject pool for that semester were primarily female, caucasian, and traditional college-age freshmen. all participants received course credit for completing the questionnaire. b. materials. we selected 39 items from previous questionnaires that we have used (kowalski & taylor, 2009; taylor & kowalski, 2004) as well as from the forced choice format developed by bensley & lilienfeld (2010). we worded all true/false (t/f) items so the correct response was “false,” except for the statement “the suicide rate is higher among the elderly than among adolescents,” which is true. for each item there was a (t/f) version and a forced choice (a/b) version. for example, the t/f version of the item regarding sugar and hyperactivity reads, “too much sugar, such as from eating candy and sugary snacks, causes hyperactivity in children.” the forced choice version read, “which statement about the effect of sugar on behavior is most true? a. too much sugar, as from eating candy and sugary snacks, causes hyperactivity in children. b. sugar has a limited effect on behavior, similar to any carbohydrate, such as potatoes or pretzels.” the preponderance of the scientific evidence favors response “b.” furthermore, most legitimate websites that disseminate parenting information and which examine the origins of hyperactivity in school-aged children emphasize this finding rather than the “sugar = hyperactivity” explanation (see comisarow, 1996; huynh, 2010.). taylor, a.k. & kowalski, p. journal of the scholarship of teaching and learning, vol. 12, no. 3, september 2012. josotl.indiana.edu 66 c. procedure. the participants, all of whom were students in the introductory psychology subject pool for the fall 2011 semester, received notification of the study via an email announcement. the announcement provided instructions on how to access the survey online, where participants began with informed consent and then completed the questionnaire. the entire questionnaire took less than one hour to complete. iii. results for almost all of the items accuracy was greater when we tested participants with the a/b rather than the t/f version. overall, a t-test comparing accuracy showed a significant difference with t(38) = -4.539, p < .001. on average, accuracy was 33.05% correct with the t/f format and 41.29% with the a/b format. to determine whether students’ responses differed depending on the format of the item, we calculated t-test differences between mean accuracies for each item based on either the t/f response or the a/b forced choice response. we then corrected the significance level for all comparisons by using simes’ (1986). table 1 lists the items which met this standard of comparison. fourteen of the 39 items met this criterion. for each of the 14 items, a greater percentage of students reported holding the misconception when answering the t/f item compared with the a/b item. thus, for 36% of the items students’ responses depended on how we asked the question. we then subdivided the remaining items into two categories. there were those which did not meet this stringent simes’ correction procedure for significance but which independently met the .05 level of significance. ten items met this criterion. on each of these 10 items, students also tended to favor misconceptions more frequently when responding to t/f items. finally, the remaining 15 items met neither criterion for statistical significance. these were the items to which the participants responded at the same level of accuracy no matter how we asked. we included items in this questionnaire based on the items’ inclusion in previous questionnaires used by us and by others. there is, in fact, no absolute way to define what constitutes a misconception. previous research has tended to differentiate items which truly represent wide-spread misconceptions from those which do not by using a criterion of 50% accuracy (lamal, 1979; vaughn, 1977). that is, if at least 50% of students hold a particular belief that contradicts the preponderance of scientific evidence, that is a sufficiently large percentage to consider that belief a popular misconception. using this criterion, nine items would be considered misconceptions with the t/f format but not the a/b format. these nine items included seven of the 14 items that resulted in truly different responses across formats (t/f vs. a/b) and two additional items from the set showing significant differences without the simes correction. we also noted that five items on the questionnaire failed to meet the 50% criterion of accuracy in both the t/f and the a/b formats and would not have been considered misconceptions with either format. thus, of the 39 items we identified from the previous literature as reflecting frequent student misconceptions, 25 items appeared to be common misconceptions regardless of how we asked, nine depended on how we asked, and five did not appear to be misconceptions no matter how we asked the students. taylor, a.k. & kowalski, p. journal of the scholarship of teaching and learning, vol. 12, no. 3, september 2012. josotl.indiana.edu 67 table 1. a comparison of accuracy rates for items stated in the true/false and forced choice formats and the difference between the two question formats. brief item: accuracy t/f percent accuracy a/b percent accuracy a/b accuracy t/f percent items which provide different response accuracies based on question format. there’s safety in numbers. 38.71 49.03 10.32 opposites attract. 41.29 54.19 12.90 women talk more than men. 45.16 58.06 12.90 psychological profilers can describe a criminal. 17.42 30.97 13.55 attachment is based on filling the need for food. 36.13 50.32 14.19 if unsure of a test answer, stick with the first hunch. 7.74 21.94 14.20 right-brained and left-brained people are different. 18.06 33.55 15.49 lie detectors are highly accurate. 57.42 73.55 16.13 it is better to vent your anger than to hold it in. 9.03 30.97 21.94 most people experience a midlife crisis. 27.74 51.61 23.87 esp is a well-established phenomenon. 49.68 75.48 25.80 dyslexics see letters as their reverse. 43.23 16.77 -26.46 autism has become an epidemic. 44.52 72.90 28.38 dreams reflect symbolic wishes. 36.13 76.13 40.00 items which provide different response accuracies based on looser statistical criterion. too much sugar causes hyperactivity in children. 32.28 30.33 -1.95 raising self-esteem improves achievement. 4.52 10.32 5.80 playing mozart increases infant intelligence. 38.71 32.36 -6.35 subliminal messages affect buying behavior. 12.26 18.71 6.45 elderly have higher suicide rate than adolescents. 15.48 22.58 7.10 people can learn while asleep. 49.68 57.42 7.74 people repress traumatic memories. 10.97 18.71 7.74 crack babies have serious neurological deficits. 15.48 23.23 7.75 immediate contact at birth is critical for bonding. 18.71 28.39 9.68 people with schizophrenia have a “split” personality. 44.52 54.84 10.32 items which provide similar response accuracies regardless of how one asks. most therapies are based on freud. 46.45 46.45 0.00 criminal acquitted based on insanity defense. 51.61 50.97 -0.64 most people use only about 10% of their brains. 47.74 48.39 0.65 electroconvulsive (“shock”) therapy is dangerous. 26.45 27.10 0.65 drug education programs are effective. 43.87 45.16 1.29 full moon affects behavior. 77.42 78.71 1.29 learning styles should match teaching styles. 10.97 9.68 -1.29 the first three years are critical to development. 3.87 7.74 3.87 adolescence is a time of psychological turmoil. 19.35 15.48 -3.87 older people are crankier and depressed. 54.19 49.78 -4.51 inkblots reveal personalities. 39.35 43.87 4.52 taste areas are defined on the tongue. 27.10 33.55 6.45 therapy must examine childhood root causes. 29.68 36.77 7.09 amnesiacs cannot remember their previous life. 30.97 39.35 8.38 vision depends on light rays emitted from eyes. 64.52 85.16 20.64 taylor, a.k. & kowalski, p. journal of the scholarship of teaching and learning, vol. 12, no. 3, september 2012. josotl.indiana.edu 68 iv. discussion the purpose of this study was to examine the different ways in which students can be asked about their misconceptions related to psychological science and whether these different assessment formats make a difference. one format which researchers previously used is a true/false format wherein students read a statement and designated whether they believe it reflects a true or a false statement. the second is a forced choice format in which researchers ask students to pick one of two options. we asked whether there is an overall difference in accuracy when we assess student misconceptions with true/false or forced choice formats. we also wanted to know whether student agreement with items in the true/false format reflects the question format or students’ underlying false beliefs. we found that for some items, how you ask matters, and for other items it does not matter very much. overall, students reported fewer false beliefs when asked with the forced choice format. this suggests that our assessment of students’ misconceptions can sometimes depend on how we ask. although we do not know the exact thought processes, it is possible that for some claims, when students see only the more popular misconception (t/f version), they fail to think through the alternatives. however, when students see the correct response immediately next to the incorrect response (a/b version) they may stop to consider the correctness of the alternate choice. further studies, perhaps with think-aloud procedures, may help determine these processes. knowing what a student is thinking at the time of answering the item would help to determine if the student has an alternative understanding of the item rather than a misconception of the item. we also noted that some claims on our instrument were not endorsed by the majority of students. we recommend that researchers pretest their instrument, which is likely to vary somewhat over time and over samples. for example, although other researchers have found support for the “extramission” misconception (winer & cottrell, 1996)—the idea that visual perception depends on light emitted from the eyes, rather than on the correct conception that visual perception depends on light entering the eyes—we failed to find support for this belief. less than 35% of our participants thought this is an accurate statement when asked in the true/false format. (in visual perception, as a person sees something, tiny particles or light rays are emitted from the eyes.) the misconception rate fell to less than 15% when we asked about it with the forced choice format. (which is most true about what happens in vision as a person sees something? a. tiny particles or light rays emit from the eyes. b. light is converted to nerve impulses in the back of the eyes.) therefore, we conclude that how you ask matters but that cohort effects may affect beliefs; past cohorts endorsed extramission, current students do not. in addition, television programming may influence some of the misconceptions related to psychology. for several years we found that students strongly favored the misconception that the polygraph (“lie detector”) test is a highly accurate means of detecting dishonesty (see taylor & kowalski, 2004); however, in recent years the accuracy rate of recognizing this as being false has risen perhaps due to the influence of television shows based on fictionalized court cases which these days more correctly portray that information. based on our informal conversations with students, the programs that do represent the polygraph as being highly accurate (“maury”) tend to be perceived as having less credibility. therefore, different cohorts of college students may express beliefs based on information they learn from various media outlets, and the content or quality of that information can change over time, leading to changing beliefs among the taylor, a.k. & kowalski, p. journal of the scholarship of teaching and learning, vol. 12, no. 3, september 2012. josotl.indiana.edu 69 general public, including students. further investigations of the source of students’ misconceptions will help clarify these types of cohort changes over time. a. conclusions in conclusion, we found evidence that the use of the true/false format may over-estimate the prevalence of misconceptions that students endorse when they enter the introductory psychology classroom. our overall data analyses showed that accuracy rates on average across all items, and across almost all individual items, were lower when we asked the true/false version of the item when compared to the forced choice version of the item. although the misconceptions in this study reflect students’ knowledge of psychological science, the findings fit well within the accumulating body of knowledge on student misconceptions across a wide range of disciplines (bransford et al., 2000). misconceptions exist in all disciplines. how you ask students about these preconceptions is likely to matter. having the most accurate measure possible becomes important when instructors and researchers use the results of such tests in additional ways, such as examining the effectiveness of interventions to reduce these false beliefs. we recommend continued attention to methods of assessing misconceptions as researchers cautiously pursue factors which may reduce misconceptions in the classroom. appendix 1. selected items, first in the true/false format and next as a/b forced choice format. drug education programs (i.e., dare) are effective in deterring drug use among teenagers. which of the following statements about drug education programs is most true? a. drug education programs (i.e., dare) are effective in deterring drug use among teenagers. b. drug education programs (i.e., dare) are ineffective in deterring drug use among teenagers. too much sugar, such as from eating candy and sugary snacks, causes hyperactivity in children. which statement about the effect of sugar on behavior is most true? a. too much sugar, as from eating candy and sugary snacks, causes hyperactivity in children. b. sugar has a limited effect on behavior, similar to any carbohydrate, such as potatoes or pretzels. taste areas for sweet, sour, salty and bitter are well defined on the tongue. which statement is most true about the sense of taste? a. taste areas for sweet, sour, salty and bitter are well defined on the tongue. b. people can perceive all taste qualities all over their tongue. the suicide rate is higher among the elderly than among adolescents. which statement about the suicide rate is most true? a. the suicide rate is higher among the elderly than among adolescents. b. the suicide rate is higher among adolescents than among the elderly. most "crack babies" end up with serious neurological deficits. which is most true about “crack babies”? a. most “crack babies” end up with serious neurological deficits. taylor, a.k. & kowalski, p. journal of the scholarship of teaching and learning, vol. 12, no. 3, september 2012. josotl.indiana.edu 70 b. most “crack babies” develop normally in the long run. immediate contact between a mother and infant after birth is critical for bonding. which statement about mother-infant bonding is most true? a. immediate contact between a mother and infant after birth is critical for bonding. b. mothers and infants do not need immediate contact after birth to develop a bond. a baby’s attachment for its mother is based on mom’s filling the physiological need for food. which statement is most true about the source of the attachment? a. baby’s attachment for its mother is based on filling the physiological need for food. b. a baby’s attachment for its mother is based primarily on physical contact and not nourishment.   references amsel, e., johnston, a., alvarado, e., kettering, j., rankin, l., et al. (2009). the effect of perspective on misconceptions in psychology: a test of conceptual change theory. journal of instructional psychology, 36(4), 289-295. baser, m. (2006). effect of conceptual change oriented instruction on students' understanding of heat and temperature concepts. journal of maltese education research, 4, 64-79. bransford, j. d., brown, a. l., cocking, r. r. (2000). how people learn: brain, mind, experience, and school. washington, d. c.: national academy press. bensley, d. a., & lilienfeld, s. o. (2010). the test of psychological knowledge and misconceptions. unpublished manuscript. bensley, d. a., lilienfeld, s. o., ferree, s.k., powell, l. a., southerly, t. d. (2011, march). reliability and validity of the knowledge of psychology test. poster presented at the 82nd annual meeting of the eastern psychological association, cambridge, ma. boyes, e., & stanisstreet, m. (1991) misconceptions in first-year undergraduate science students about energy sources for living organisms. journal of biological education, 25, 209-13. doi:10.1080/00219266.1991.9655208 brown, l. t. (1984). misconceptions about psychology aren’t always what they seem. teaching of psychology, 11(2), 75-78. doi:10.1207/s15328023top1102_3 burton, r. f. (2005). multiple-choice and true/false tests: myths and misapprehensions. assessment & evaluation in higher education, 30(1), 65-72. doi:10.1080/0260293042003243904 comisarow, j. (1996). can sweet treats drive kids crazy? sugar and hyperactivity in children. nutrition bytes, 2(1), taylor, a.k. & kowalski, p. journal of the scholarship of teaching and learning, vol. 12, no. 3, september 2012. josotl.indiana.edu 71 dykstra, d. i., boyle, c. f. & monarch, i. a. (1992). studying conceptual change in learning physics. science education, 76(6), 615-652. doi:10.1002/sce.3730760605 eryilmaz, a. (2002). effects of conceptual assignments and conceptual change discussions on students’ misconceptions and achievement regarding force and motion. journal of research in science teaching, 39(10), 1001-1015. gardner, r. m., & dalsing, s. (1986). misconceptions about psychology among college students. teaching of psychology, 13(1), 32-34. doi:10.1207/s15328023top1301_9 griggs, r. a., & ransdell, s. e (1987). misconceptions tests of misconceived tests? teaching of psychology, 14(4), 210-214. doi:10.1207/s15328023top1404_4 guzzetti, b. j., snyder, t. e., glass, g. v., & gamas, w. s. (1993). promoting conceptual change in science: a comparative meta-analysis of instructional intervention from reading education and science education, reading research quarterly, 28(2), 116-159. doi:10.2307/747886 hapkiewicz, a. (1999). naïve ideas in earth science. msta journal, 44(2), 26-30. higbee, k. l., & clay, s. l. (1998). college students' beliefs in the ten-percent myth. journal of psychology: interdisciplinary and applied, 132(5), 469-476. doi:10.1080/00223989809599280 huynh, n. (2010). does sugar really make children hyper? yale scientific. retrieved on 10-9-14 from http://www.yalescientific.org/2010/09/mythbusters-does-sugar-really-make-children-hyper/ kowalski, p., & taylor, a. (2009). the effect of refuting misconceptions in the psychology classroom. teaching of psychology, 36(3), 153-159. doi:10.1080/00986280902959986 kuhle, b. x., barber, j. m., & bristol, a. s. (2009). predicting students' performance in introductory psychology from their psychology misconceptions. journal of instructional psychology, 36(2), 119-124. lamal, p. a. (1979). college students’ common beliefs about psychology. teaching of psychology, 6(4), 155-158. doi:10.1207/s15328023top0603_8 lilienfeld, s. o. (2010). confronting psychological misconceptions in the classroom: challenges and rewards. observer, 23(7), retrieved on 11/11/11 from http://www.psychologicalscience.org /index.php/publications/observer/2010/september-10/confronting-psychological-misconceptions-in-theclassroom.html. lipson, m. y. (1982). learning new information from text: the role of prior knowledge and reading ability. journal of literacy research, 14(3), 243-261. doi: 10.1080/10862968209547453 lorenzo, d. (1999). countering popular misconceptions of federal bureaucracies in american government classes. ps: political science and politics, 32(4), 743-747. doi:10.2307/420166 taylor, a.k. & kowalski, p. journal of the scholarship of teaching and learning, vol. 12, no. 3, september 2012. josotl.indiana.edu 72 mckeachie, w. j. (1999). teaching tips: strategies, research, and theory for the college and university teacher, boston: houghton mifflin co. mulford, d. r., & robinson, w. r. (2002). an inventory for alternative conceptions among first-semester general chemistry students. journal of chemical education, 79(6), 739-744. doi:10.1021/ed079p739 ruble, r. (1986). ambiguous psychological misconceptions. teaching of psychology, 13(1), 3436. doi:10.1207/s15328023top1301_10 sax, g. (1989). principles of educational and psychological measurement and evaluation (3rd ed.), belmont, ca: wadsworth. simes, r. j. (1986). an improved bonferroni procedure for multiple tests of significance. biometrika, 73(3), 751-754. doi:10.1093/biomet/73.3.751 southerly, t. d., bensley, d. a., lilienfeld, s. o., ferree, s.k., powell, l. a. (2011, march). psychology students’ knowledge of psychology and psychological misconceptions. poster presented at the 82nd annual meeting of the eastern psychological association, cambridge, ma. strike, k. a., & posner, g. j. (1992). a revisionist theory of conceptual change. in r. duschl & r. hamilton (eds.), philosophy of science, cognitive psychology, and educational theory and practice (pp. 147-176). albany, ny: suny. taylor, a., & kowalski, p. (2004). naïve psychological science: the prevalence, strength and sources of misconceptions. psychological record, 54, 15-25. taylor, a. k., & kowalski, p. (april, 2010). goal orientation and learning strategies predict conceptual change over time. poster presented at the 90th annual western psychological association conference, cancun, mexico. taylor, a., kowalski, p., & laggren, l. (april, 2000) myths & misconceptions about psychology: strength of belief is not related to accuracy. poster presented at the 80th annual convention, western psychological association, portland, or. vaughan, e. d. (1977). misconceptions about psychology among introductory psychology students. teaching of psychology, 4, 138-141. doi:10.1207/s15328023top0403_9 williamson, t. (2007). the philosophy of philosophy. oxford, uk: john wiley & sons. winer, g. a., & cottrell, j. e. (1996). does anything leave the eye when we see? extramission beliefs of children and adults. current directions in psychological science, 5, 137-142. doi:10.1111/1467-8721.ep11512346 503 service temporarily unavailable 503 service temporarily unavailable nginx/1.14.1 microsoft word stickelsfinal the journal of scholarship of teaching and learning, vol. 7, no. 1, may 2007, pp. 1-15. helping alleviate statistical anxiety with computer aided statistical classes john w. stickels and rhonda r. dobbs1 abstract: this study, helping alleviate statistical anxiety with computer aided statistics classes, investigated whether undergraduate students’ anxiety about statistics changed when statistics is taught using computers compared to the traditional method. two groups of students were questioned concerning their anxiety about statistics. one group was taught statistics using the traditional “calculator and paper/pencil” method while the other group was taught statistics using computers to perform the required calculations. frequency distributions and the chi-square test of independence indicated there were significant differences between the two groups. students in the computer class had significantly less statistical anxiety than students in the traditional class. keywords: statistics, statistical anxiety, computer, spss, teaching methods. i. introduction. most undergraduate students experience moderate to considerable anxiety when faced with their first undergraduate statistics class. this anxiety stems from many sources including the dread of taking another college level math class, discussions with other students about how they have ‘suffered’, or just plain ‘fear of the unknown.’ this apprehension about taking statistics is referred to as “statisticophobia” (dillon, 1982). in addition, students may fail to see the relevance of statistics for their criminal justice careers. accordingly, they may delay taking the course until their very last semester, thus allowing anxiety to build unnecessarily. one difficulty, then, of teaching statistics is overcoming “statisticophobia” so that students can learn statistics while enjoying the class. one of the primary sources of fear and anxiety about statistics is caused by the very nature of statistics. in a traditional statistics class, students have heard ‘horror’ stories from other students about using a pencil and a calculator to solve complex statistical formulas. then, at the start of the semester, students buy a textbook, go back to their dorm room or apartment, open the book, and see formulas that look like this: t = m1 – m2 √ s 2m1 + s 2m2 consequently, this is going to cause anxiety about taking statistics. during the 2003 – 2004 academic year, a decision was made to change how introductory statistics was taught at texas state university│san marcos. the change was made from teaching statistics the traditional “calculator and paper/pencil” method to teaching statistics through 1 department of criminology and criminal justice, the university of texas at arlington, box 19595, arlington, texas 76019-0595, jstickel@uta.edu and rdobbs@uta.edu. stickels, j.w., and dobbs, r.r. the journal of scholarship of teaching and learning, vol. 7, no. 1, may 2007. 2 computer. the primary goal of this change was to make statistics more ‘user friendly’ and, hopefully, re-focus the students from merely performing the required calculations to understanding statistical theory and the appropriate use of statistical processes. the purpose of this study was to investigate whether using computers changed students’ attitudes toward statistics. ii. literature review. statistics is required for graduation in criminal justice programs at almost all accredited universities with the required statistics course being followed by a course in research methods. often, the thought of facing statistics with the accompanying fear of failure causes severe consternation to most students in a criminal justice program (kennedy and mccallister, 2002). thus, being able to successfully teach statistics so that students can succeed in statistics and research methods has become important in present day criminal justice programs (bushway, shawn, and flower, 2002). dillon (1982) noted that a primary source of statistical anxiety is the fear of taking a complicated college level math class. her attempts in alleviating this anxiety were having her students express their fears by answering questionnaires detailing their feelings about statistics and combining this with a lecture on math anxiety. dillon reported that this procedure eased the students’ fears to the point where they could master statistical concepts (dillon, 1982). journal writing has also been tested as a method of reducing statistical anxiety. for example, sgoutas-emch and johnson (1998) compared a group of undergraduate students who participated in journal writing to a group that did not in order to determine whether there was a difference in statistical anxiety between the two groups. sgoutas-emch and johnson found that the group that participated in journal writing had improved grades, less anxiety before exams, and lower physiological reactions (sgoutas-emch and johnson, 1998). one of the authors has experienced statistical anxiety in his college teaching career and has attempted to alleviate it several ways. in his first semester of teaching introductory statistics, the author tried to lessen the student’s anxiety by having a math review session followed by a simple math test, followed by a demonstration about how data and statistics are useful in everyday life using examples from football and basketball. unfortunately, it did not work and the students appeared to be just as anxious after the lecture as before it. one newer strategy teachers use to reduce statistical anxiety is by incorporating computer use into statistics classes (collins, oberg, and shera, 1989). kennedy, mccallister, and corliss (2001) investigated whether using computers as the focus of instruction in an advanced statistic class, together with an individualized, self-paced, student-centered, activity-based course, positively impacted student attitudes about statistics. the respondents, 15 graduate students in an advanced statistics class and 4 graduate students in a multivariate statistics class were questioned about their attitudes regarding statistics at the beginning and end of the course. the survey results indicated that the students’ attitudes about statistics improved over the course of the semester. kennedy et al. attributed this improvement to the use of computers in the classroom (kennedy, mccallister, and corliss, 2001). kennedy, mccallister, and corliss (2002) used this same methodology when they reexamined whether using computer positively impact attitudes about statistics. in this second study, 43 graduate students in an advanced statistics class were taught using computers as the focus of instruction in an advanced statistic class, together with an individualized, self-paced, stickels, j.w., and dobbs, r.r. the journal of scholarship of teaching and learning, vol. 7, no. 1, may 2007. 3 student-centered, activity-based course. as in the previous study, the students were questioned about their attitudes regarding statistics at the beginning and end of the course. the survey results confirmed the findings of the previous study and indicated that students’ attitudes about statistics improved over the course of the semester. as in the previous study, kennedy et al. attributed this improvement to the use of computers in the classroom (kennedy, mccallister, and corliss, 2002). results indicate that using computers in a statistics class has been generally successful in lowering statistical anxiety. for instance, ware and chastain (1989) investigated whether using computers in an undergraduate statistics class significantly impacted the students’ attitudes about statistics. in this study, the researchers compared attitudes about statistics from students in a computer aided statistics class to attitudes about statistics from students in a non-computer statistics class. the results indicated a significant difference between the students’ attitudes with the students in the computer aided class having significantly more favorable attitude about statistics than students in the class without computers (ware and chastain, 1989). using computers in the classroom has also been shown to increase statistical knowledge. lane and aleski (2002) took advantage of a redesign in undergraduate statistics classes to examine whether using computers and collaborative learning methods influenced student performance. prior to the redesign, statistics classes were taught in a traditional lecture style format to sections of 240 students by full time faculty. after the changes, the courses were more student-centered than instructor-centered with students having one session each week in one large lecture session (240 students) and two sessions in a computer lab (40-60 students). the faculty member taught all three classes and a graduate student teaching assistant was present in the computer lab. a content knowledge test was administered to the students at the beginning and the end of each semester. the results of this study found that students’ overall performance was higher in the redesigned classes than in the traditional course (lane and aleski, 2002). the intent of the current study was to investigate whether there was a difference between students’ attitudes toward statistics depending on whether computers were used as a teaching aid in an introductory statistics class. specifically, it was hypothesized that the student’s attitudes about statistics would be more favorable for students taught statistics using a computer than students taught statistics using the traditional method. iii. methods. a. research design. the ex post facto research design was selected for this project because it was closest available to the specific teaching situation presented (cook, 1979). one of the recognized problems associated with this design is attempting to eliminate any selection bias since there is no pre-test to use as a comparison. this potential problem was addressed in the survey instrument when the respondents were asked how strongly they wanted to take statistics and why they took the class. the responses to these questions indicated there was not a significant difference between the attitudes of the two groups of students (non-computer vs. computer class) prior to taking the class (see results section below). the students registered for the computer courses did not report different levels of enthusiasm or motivations for taking the course compared to those in the non-computer course, indicating that selection bias associated with those factors did not exist or, at the most, was minimal. stickels, j.w., and dobbs, r.r. the journal of scholarship of teaching and learning, vol. 7, no. 1, may 2007. 4 another potential problem associated with this research design is attempting to eliminate any bias caused by the instructor and teaching materials (cook, 1979). the potential problem resulting from instructor bias was addressed by using the same instructor and teaching methods for both computer and non-computer classes. the primary difference between the classes was the addition of how to use spss by brian c. cronk for the computer-based class (cronk, 2004). this book was used only to teach the students how to use spss. while it is not possible to say that instructor bias was completely eliminated, after all, it is reasonable that the same instructor might perform somewhat differently in different classrooms and with different students. it is believed that any instructor bias that might still exist was of minimal impact on the results of this study. b. subjects. the subjects consisted of students enrolled in an undergraduate criminal justice statistics class at texas state university│san marcos for fall 2003, and spring and summer i 2004. a total of 88 students were tested, 40 from the fall 2004 (non-computer) class and 48 from the spring and summer 2004 (computer) classes. of these respondents, there were 24 male and 16 female students for the non-computer class. there were 22 male and 25 female students in the computer classes. c. survey instrument. the questions comprising the survey instrument were borrowed from a combination of four sources. the first source for questions was the scales test developed by fennema-sherman (fennema and sherman, 1976). the second source for questions originated from the students’ attitude toward statistics test developed by sutarso (sutarso, 1992a, 1992b). the third source for questions was the school of education – university of virginia course evaluation form (braskamp, brandenburg, and ory, 1984). finally, questions were asked that specifically applied to the issues investigated by this project. the survey instrument was a likert scale questionnaire that queried the students about general and individual attitudes toward statistics. the general attitudinal questions included questions such as, ‘learning statistics is mostly memorizing,’ ‘statistics is interesting,’ ‘statistics is mostly about symbols and formulas rather than about ideas,’ and ‘in statistics, knowing why an answer is correct is important.’ the individual questions included: ‘my interest in statistics has increased over the semester,’ ‘overall, this class was useful to me,’ and ‘i would rate the value of this course to me as ________,’ to gauge whether there was a bias in the students either for or against enrolling in this class. a copy of the survey instrument is attached as appendix i. d. procedure. the undergraduate statistics for criminal justice class at texas state university│san marcos is an introductory upper-level statistics class taught to criminal justice majors. usually, the classes consist of juniors and seniors. according to the 2002-2004 undergraduate catalog, the expectations for this class are: stickels, j.w., and dobbs, r.r. the journal of scholarship of teaching and learning, vol. 7, no. 1, may 2007. 5 the theory and application of statistical inferential techniques, and correlation and regression for behavioral science data and its application in criminal justice. emphasis is on the collection, analysis, and interpretation of statistical data in criminal justice settings (texas state university│san marcos undergraduate catalog, http://www.txstate.edu/academicaffairs/ugcat02-04/). both the non-computer and computer classes emphasized descriptive and inferential statistics. the descriptive statistics taught were frequency distributions, measures of central tendency, measures of variability, graphs, and correlation as a descriptive statistic. the inferential statistics presented were correlation, z-test, t-test, analysis of variance, regression analysis, and the chi-square test. in addition, the lectures emphasized the appropriate rules for using each type of descriptive and inferential statistic. the non-computer class was a ‘traditional’ statistics class taught in the fall 2003 semester. students were expected to perform all required mathematical calculations using a calculator, pencil and paper. a computer was not used in the non-computer class except for the regression analysis lectures. in the regression analysis lectures, students were provided examples of this statistic using microsoft excel. the survey instrument was administered to 40 students from this class. the computer classes were taught in the spring 2004 and summer i 2004 semesters with the students performing all required calculations using spss version 12.0. the survey instrument was administered to 48 students from these classes. iv. results. as stated, the goal of this study was to determine whether student’s attitudes about statistics would be more favorable for students taught statistics using a computer than students taught statistics using the traditional method. two statistical processes were used to asses this question. the first was a simple frequency distribution and the second was a chi-square test of independence (х2) using the crosstabulation function in spss with “computer/non-computer” as the independent variable and the various attitudes as the dependent variable. the overall results indicated a significant difference between the computer and non-computer classes with the computer classes being routinely judged more favorably than the non-computer classes. a. selection bias. as noted, one of the problems associated with the ex post facto research design is the chance of selection bias. two survey questions were asked in an attempt to determine the presence of selection bias. these two questions were question 18: “when registering for this class, how strongly did you want to take it?” and question 19: “which one of the following was the most important reason for taking this class?” the results of these questions indicate there was no selection bias and that the two groups were basically equivalent. question 18: when registering for this class, how strongly did you want to take it? this question was asked in an attempt to determine whether there was any selection bias attributable to the ex post facto research design. the frequency distribution for question 18 demonstrates that the students were fairly uniform in their lack of enthusiasm for taking statistics. stickels, j.w., and dobbs, r.r. the journal of scholarship of teaching and learning, vol. 7, no. 1, may 2007. 6 table 1. crosstabulation of class by q18 (enthusiasm for taking statistics). q18 total very enthusiastic enthusiastic indifferent unenthusiastic not at all interested class no comp 0 8 18 7 7 40 computer 4 10 21 10 3 48 total 4 18 39 17 10 88 the chi-square statistic for question 1 was х2 = 5.904 (p = 0.206). accordingly, there was not a significant difference between the students’ enthusiasm for taking statistics depending on whether the student was taught in the computer or the non-computer classes. question 19: which one of the following was the most important reason for taking this class? this question was also asked to try to determine whether there was any selection bias attributable to the ex post facto research design. the frequency distribution for question 19 indicates that the students were fairly uniform in their reasons for taking this statistics class with a large majority taking it because it was required. table 2. crosstabulation of class by q19 (reasons for taking statistics). q19 required optional interested instructor good grade no response total class no comp 30 3 0 4 1 2 40 computer 33 1 3 10 0 1 48 total 63 4 3 14 1 3 88 the chi-square statistic for question 1 was х2 = 7.392 (p = 0.194). accordingly, there was not a significant difference between the students’ reason for taking statistics depending on whether the student was taught in the computer or the non-computer classes. the results of questions 18 and 19 support a finding that there was no selection bias present in the respondents. according to the results of question 18, there was a fairly uniform lack of enthusiasm for taking statistics with no significant difference between the two groups. in addition, the reasons for taking the class were, by a large majority, because the class was required. again, there was no significant difference between the groups. students registered for the computer course were no more or less enthusiastic about the course nor were they taking the course for different reasons. as a result, the responses to these questions support an inference that there was little or no selection bias associated with this study, at least in relation to the reasons and motivations for taking the course, whether it be computer focused or not. b. general question results. question 1: learning statistics is mostly memorizing one of the goals of changing from a traditional to a computer based class was to refocus the students’ concentration from memorizing statistical formulas to learning statistical theory and understanding the practical uses of statistics. question number 1, “learning statistics is mostly memorizing,” was intended to determine whether this goal was met. the frequency distribution for question 1 indicates that stickels, j.w., and dobbs, r.r. the journal of scholarship of teaching and learning, vol. 7, no. 1, may 2007. 7 more students in the computer classes either ‘disagreed’ or ‘strongly disagreed’ with this question as compared to the non-computer classes: table 3. crosstabulation of class by q1 (statistics is mostly memorizing). q1 strongly agree agree disagree strongly disagree not applicable total class no comp 3 28 8 0 1 40 computer 7 18 18 5 0 48 total 10 46 26 5 1 88 the chi-square statistic for question number 1 was х2 = 13.00 (p = 0.0111). accordingly, there was a significant difference between the students’ attitudes about whether statistics is mostly memorizing, depending on whether the student was taught in a computer or the non-computer class. as a result, it can be concluded that the goal of refocusing the students’ concentration from memorizing statistical formulas to learning statistical theory and understanding the practical uses of statistics was met with the changed way of teaching method. question 2: statistics is interesting. a desired by-product of changing from a traditional to a computer based class was to make statistics class more interesting to the average criminal justice student. question number 2, “statistics is interesting,” was asked to determine whether this occurred. the frequency distribution for question 2 indicated that more students in the computer classes either ‘strongly agreed’ or ‘agreed’ with this question when compared to the non-computer classes: table 4. crosstabulation of class by q2 (statistics is interesting). q2 total strongly agree agree disagree strongly disagree na class no comp 1 15 16 6 2 40 computer 8 26 10 3 1 48 total 9 41 26 9 3 88 the chi-square statistic for this question was х2 = 10.473 (p = 0.033). therefore, there was a significant difference between the students’ attitudes about whether statistics is interesting, depending on whether the student was taught in a computer or the non-computer class. as a result, it can be concluded that the desired byproduct making statistics more interesting to the students was achieved as a result of the change in the teaching method. question 5: statistics is mostly about symbols and formulas rather than about ideas. one of the goals of both the non-computer and computer classes was to teach the students when, and under what circumstances, each statistical process should be used. question number 5, “statistics is mostly about symbols and formulas rather than about ideas,” was asked to determine whether there was a difference in opinions about this question based on whether the student was in a computer or a non-computer class. the frequency distribution for question 5 shows that students had obvious differences of opinions about the use of statistics depending on whether the student was taught with or without a computer. stickels, j.w., and dobbs, r.r. the journal of scholarship of teaching and learning, vol. 7, no. 1, may 2007. 8 table 5. crosstabulation of class by q5 (statistics is symbols and formulas vs. ideas). q5 total strongly agree agree disagree strongly disagree class no comp 9 24 4 3 40 computer 4 12 23 9 48 total 13 36 27 12 88 the chi-square statistic for question 5 was х2 = 21.746 (p = 0.000). therefore, there was a significant difference between the students’ opinions about whether statistics is mostly about symbols and formulas rather than about ideas, depending on whether the student was taught in the computer or the non-computer class. based on this difference, it can be concluded that the goal of teaching the students when and under what circumstances, to use each statistical process was achieved through changing the teaching method. question 6: in statistics, knowing why an answer is correct is important. a related goal of changing from a traditional to a computer based class was to teach the students to understand the difference between merely solving problems and understanding why the problem was answered in the way in was. question number 6, “in statistics, knowing why an answer is correct is important,” was asked to determine whether the students had a difference of opinion about this question based on whether the student was in a computer or a non-computer class. the frequency distribution for question 6 indicates that more students in the computer class either strongly agreed or agreed with this question than the students in the non-computer class. table 6. crosstabulation of class by q6 (important to know why an answer is correct). q6 total strongly agree agree disagree strongly disagree class no comp 13 19 7 1 40 computer 22 24 0 2 48 total 35 43 7 3 88 the chi-square statistic for question 5 was х2 = 9.581 (p = 0.022), indicating a significant difference between the responses for this question based on whether the student was taught in the computer or the non-computer classes. thus, the goal of teaching students to understand the difference between merely solving problems and understanding why an answer is correct was better achieved in the computer class than the non-computer class. c. individual questions results. question 14: my interest in statistics has increased over the semester. an ever present objective of teaching statistics is keeping the students interested throughout the semester. one hoped-for consequence of changing from a traditional to a computer based class was that the students would stay more interested in the computer based statistics class than in the traditional class. question number 14, “my interest in statistics has increased over the semester,” was asked to determine whether the students’ interest in the class increased more throughout the semester in stickels, j.w., and dobbs, r.r. the journal of scholarship of teaching and learning, vol. 7, no. 1, may 2007. 9 the computer or non-computer class. the frequency distribution for question 14 indicates that the students’ interest in the computer class increased more throughout the semester than the students in the non-computer class. table 7. crosstabulation of class by q14 (interest increased over the semester). q14(interest increased) total strongly agree agree disagree strongly disagree na class no comp 0 18 17 2 3 40 computer 7 25 15 0 1 48 total 7 43 32 2 4 88 the chi-square statistic for question 1 was х2 = 10.625 (p = 0.031), indicating a significant difference between the responses for this question based on whether the student was taught in the computer or the non-computer classes. based on this difference, it can be concluded that the objective of keeping students interested in class was achieved better in the computer than the non-computer class. question 16: overall, this class was useful to me. a related goal of keeping the students interested throughout the semester is to teach students that statistics will be useful in the future. with the change to a computer based method of teaching, it was theorized that teaching students how to perform the required calculations on the computer would illustrate the usefulness of statistics since the student would not have to visualize performing calculations with a computer every time a statistic was needed. question number 16, “overall, this class was useful to me,” was asked to determine whether the computer based statistics class made statistics more useful than the non-computer class. the frequency distribution for question 16 indicates that the majority of students in the computer class thought statistics were useful when compared to the non-computer class. table 8. crosstabulation of class by q16 (class was useful). q16 total strongly agree agree disagree strongly disagree na class no comp 2 27 10 1 0 40 computer 12 32 2 1 1 48 total 14 59 12 2 1 88 the chi-square statistic for question 16 was х2 = 13.282 (p = 0.010), indicating a significant difference between the responses for this question based on whether the student was taught in the computer or the non-computer classes. therefore, it can be concluded that the objective of teaching students that statistics will be useful in the future was achieved better in the computer than the non-computer class. question 25: i would rate the value of this course to me as ___. every college class is expensive and students expect to receive something of value from taking a class. thus, a related goal of teaching students that statistics something useful is to providing the student some value with each class. question number 25, “i would rate the value of this course to me as,” was asked stickels, j.w., and dobbs, r.r. the journal of scholarship of teaching and learning, vol. 7, no. 1, may 2007. 10 to determine whether, in the student’s opinion, about the value of the class was different for the computer and the non computer classes. the computer based statistics class made statistics more useful than the non-computer class. the frequency distribution for question 25 indicates that more students judged the computer course as excellent and good than did the non-computer class. table 9. crosstabulation of class by q25 (value of the course). q25 total excellent good satisfactory poor class no comp 8 20 11 1 40 computer 22 21 5 0 48 total 30 41 16 1 88 the chi-square statistic for question 25 was х2 = 9.156 (p =0.027), demonstrating a significant difference between the responses for this question based on whether the student was taught in the computer or the non-computer classes. as a result, it can be concluded that students in the class taught with the computer judged the class significantly more valuable than the students in the non-computer class. v. discussion and conclusion. the primary goal of changing the way statistics was taught at texas state university│san marcos was to make statistics more ‘user friendly’ and re-focus the students from merely performing the required calculations to understanding statistical theory and the appropriate use of statistical processes. a by-product of this change was to alleviate statistical anxiety by removing the complex mathematical calculations from the class. this study indicates that the goal of alleviating statistical anxiety was met through the change to a computer based class. the responses to the general attitudinal questions, ‘learning statistics is mostly memorizing,’ ‘statistics is interesting,’ ‘statistics is mostly about symbols and formulas rather than about ideas,’ and ‘in statistics, knowing why an answer is correct is important’ demonstrate a significant difference between the computer and non-computer classes. the differences in opinion highlighted by these questions are important because these questions inquire into one of the primary causes of statistical anxiety – the trepidation resulting from the necessity of performing complex mathematical calculations. these questions illustrate that when the necessity of performing complex mathematical calculations by hand is removed, statistical anxiety decreases. the responses to the general attitudinal questions are also important because they lend support to the conclusion that using the computer makes statistics more ‘user friendly’ and refocuses the students from merely performing the required calculations to understanding statistical theory and the appropriate use of statistical processes. as seen, more students understood that statistics is about more than crunching numbers. according to the responses, it appears that the students in the computer classes recognized that statistical processes and theory are as important as being able to perform the required calculations. stickels, j.w., and dobbs, r.r. the journal of scholarship of teaching and learning, vol. 7, no. 1, may 2007. 11 the responses to the individual attitudinal questions, ‘my interest in statistics has increased over the semester,’ ‘overall, this class was useful to me,’ and ‘i would rate the value of this course to me as_______,’ demonstrate a significant difference between the computer and non-computer classes with regard to individual attitudes about statistics. the differences in opinion highlighted by these questions are important because these questions inquire into how using computers for statistics impacted the individual student. these questions illustrate that removing the requirement to perform the complicated calculations by hand makes the class ‘worth more’ to the individual student. this increased value to the individual student is important because the more that a student receives from a class, the less the anxiety. this paper reported the results of a survey that inquired whether statistical anxiety was less in classes that used a computer than in classes that did not use a computer. this study found significant differences between the levels of anxiety for computer when compared to noncomputer classes. in addition, the findings in this study support the conclusion that students understand that statistical processes and theory are as important as being able to perform the required calculations. in summary, the results support earlier research that found that using computers in teaching statistics helps alleviate statistical anxiety and can improve students’ attitudes towards statistics. references braskamp, l. a., brandenburg, d. c., and ory, j. c. (1984). evaluating teaching effectiveness: a practical guide, appendix b. sage: beverly hills. bushway, s. d., and flower, s.m. (2002). “helping criminal justice students learn statistics: a quasi-experimental evaluation of learning assistance.” journal of criminal justice education, 13 (1), 35-56. collins, b., oberg, a., and shera, w. (1989). “an evaluation of computer-based instruction in statistical techniques for education and social work students.” journal of educational technology system, 17( 1), 59-71. cook, t. d. (1979). quasi-experimentation: design and analysis for field settings. rand mcnally: chicago. cronk, b. (2004). how to use spss, 3rd. ed. fred pyrczak:glendale, ca. dillon, k. w. (1982). “statisticophobia.” teaching of psychology, 9(2), 117. fennema, e., and sherman, j.a. (1976). “fennema-sherman mathematics attitude scales: instruments designed to measure attitudes towards the learning of mathematics by females and males”. psychological documents, 6(1). (ms. no. 1225). kennedy, r. l., mccallister, c. j., and corliss, j. (2001). attitudes toward advanced and multivariate statistics when using computers. paper presented at the annual meeting of the mid-south educational research association, little rock, ar. (eric document reproduction service no. ed464097). stickels, j.w., and dobbs, r.r. the journal of scholarship of teaching and learning, vol. 7, no. 1, may 2007. 12 kennedy, r. l., mccallister, c. j., and corliss, j. (2002). graduate students’ attitudes in an activity-based statistics course. paper presented at the annual meeting of the mid-south educational research association, chattanooga, tn. (eric document reproduction service no. ed471303). lane, j. l., and aleskic, m. (2002). transforming elementary statistics to enhance student learning. paper presented at the annual meeting of the american educational research association, new orleans, la. (eric document reproduction service no. ed463332). sutarso, t., (1992). some variables in relation to students’ anxiety in learning statistics. paper presented at the annual meeting of the mid-south educational research association, knoxville, tn. (eric document reproduction service no. ed353334). sutarso, t., (1992). students’ attitudes toward statistics. paper presented at the annual meeting of the mid-south educational research association, knoxville, tn. (eric document reproduction service no. ed353316). sgoutas-emch, s. a. and johnson, c. j. (1998). “is journal writing an effective method of reducing anxiety towards statistics?” journal of instructional psychology, 25(1), 49-57. texas state university│san marcos, 2004, undergraduate catalog. retrieved may 31, 2006, from http://www.txstate.edu/academicaffairs/ugcat02-04/. ware, m. e., and chastain, j., d. (1989). “computer-assisted statistical analysis: a teaching innovation?” teaching of psychology,16(4), 222-227. stickels, j.w., and dobbs, r.r. the journal of scholarship of teaching and learning, vol. 7, no. 1, may 2007. 13 appendix 1. statistics course questionnaire. 1. learning statistics is mostly memorizing. strongly agree agree disagree strongly disagree not applicable 2. statistics is interesting. strongly agree agree disagree strongly disagree not applicable 3. guessing is “ok” to use in solving statistical problems. strongly agree agree disagree strongly disagree not applicable 4. there are rules to follow in solving statistical problems. strongly agree agree disagree strongly disagree not applicable 5. statistics is mostly about symbols and formulas rather than about ideas. strongly agree agree disagree strongly disagree not applicable 6. in statistics, knowing why an answer is correct is important. strongly agree agree disagree strongly disagree not applicable 7. statistics are useful in everyday life. strongly agree agree disagree strongly disagree not applicable 8. i would like to have a job that includes statistics. strongly agree agree disagree strongly disagree not applicable 9. the instructor made class presentations clear. strongly agree agree disagree strongly disagree not applicable stickels, j.w., and dobbs, r.r. the journal of scholarship of teaching and learning, vol. 7, no. 1, may 2007. 14 10. statistics class is fun. strongly agree agree disagree strongly disagree not applicable 11. the level of difficulty in this class was suitable for my background and ability. strongly agree agree disagree strongly disagree not applicable 12. the amount of work required for this class was about right. strongly agree agree disagree strongly disagree not applicable 13. i spent a great deal of effort in this class. strongly agree agree disagree strongly disagree not applicable 14. my interest in statistics has increased over the semester. strongly agree agree disagree strongly disagree not applicable 15. i learned a great deal in this course. strongly agree agree disagree strongly disagree not applicable 16. overall, this class was useful to me. strongly agree agree disagree strongly disagree not applicable 17. are you an: a. undergraduate b. graduate student c. other 18. when registering for this class, how strongly did you want to take it? a. very enthusiastic about it b. enthusiastic c. indifferent d. unenthusiastic e. not at all interested in taking this class stickels, j.w., and dobbs, r.r. the journal of scholarship of teaching and learning, vol. 7, no. 1, may 2007. 15 19. which one of the following was the most important reason for taking this class? a. the course is required b. the course was optional, but recommended c. the subject was of interest to me d. the instructor’s excellent reputation e. i thought i could make a good grade 20. what grade did you expect to make? a. an ‘a’ b. a ‘b’ c. a ‘c’ d. a ‘d’ e. an ‘f’ 21. how many credits are you carrying? a. more than 16 b. 13 to 16 c. 9 to 12 d. 5 to 8 e. 0 to 4 22. i would recommend this class to a friend. strongly agree agree disagree strongly disagree not applicable 23. my opinion about statistics has changed over the course of this semester. strongly agree agree disagree strongly disagree not applicable 24. the work required for this course in relation to other courses was: much lighter lighter about the same heavier much heavier 25. i would rate the value of this course to me as: excellent good satisfactory fair poor 503 service temporarily unavailable 503 service temporarily unavailable nginx/1.14.1 503 service temporarily unavailable 503 service temporarily unavailable nginx/1.14.1 503 service temporarily unavailable 503 service temporarily unavailable nginx/1.14.1 503 service temporarily unavailable 503 service temporarily unavailable nginx/1.14.1 503 service temporarily unavailable 503 service temporarily unavailable nginx/1.14.1 503 service temporarily unavailable 503 service temporarily unavailable nginx/1.14.1 1245brackenbury journal of the scholarship of teaching and learning, vol. 12, no. 4, december 2012, pp. 12 – 28.   a qualitative examination of connections between learner-centered teaching and past significant learning experiences tim brackenbury1 abstract: learner-centered teaching is a collection of instructional practices that shift the emphasis of courses from the instructors’ goals and methods of delivery to the knowledge and skills that the students develop. this study examined potential commonalities between features of learner-centered teaching and the past significant learning experiences of current faculty. a phenomenological analysis of written essays revealed eight dominant themes: 1) student responsibility for learning, 2) learning through direct experience or example, 3) responsive instructors, 4) difficult activities that took time, 5) connections to previous knowledge and experiences, 6) direct research experience, 7) challenging initial ideas and assumptions, and 8) rich in content. these themes are discussed in terms of their connections to features of learner-centered teaching and potential implications for educators. keywords: learner-centered teaching, significant learning experiences, faculty reflections, student learning, active learning. learner-centered teaching is a paradigm of challenge. it challenges students to become active agents in their learning. students must face complex problems in order to acquire new knowledge and skills, while also developing new ways of thinking and acting. learner-centered teaching challenges instructors to release some of their control over the class and what happens. they must care about more than just content; treat student errors as learning opportunities; and change their role from distributors of knowledge to facilitators of learning. learner-centered teaching challenges both students and instructors (as well as administrative and discipline stakeholders) to carefully consider the kinds of professionals/people that graduates should be, and the optimal practices for achieving these desired outcomes (e.g., weimer, 2002). these challenges can be daunting to educators and students who are new to learnercentered teaching because they appear to be dramatically different from the methods of education that most individuals have experienced before (i.e., traditional, instructor-centered models). but, are they significantly different from all previous educational experiences or just certain common ones (such as lecture)? as barr and tagg (1993) stated, “… not all elements of the new paradigm are contrary to corresponding elements of the old: the new includes many elements of the old within its larger domain of possibilities” (p. 15). the primary objective of the present study was to examine commonalities and differences between features of learnercentered teaching and past learning experiences that individuals identified as particularly meaningful and valuable to their development (i.e., significant learning experiences). to do so, applied features of learner-centered teaching, defined as the characteristics that are directly experienced by educators and students (as opposed to the theoretical constructs behind them),                                                                                                                 1  department of communication sciences and disorders, bowling green state university, 200 health center building, bowling green, oh 43403, tbracke@bgsu.edu   brackenbury, t. journal  of  the  scholarship  of  teaching  and  learning,  vol.  12,  no.  4,  december  2012.   josotl.indiana.edu   13 were compared with themes derived from faculty reflections on their past significant learning experiences. a. applied features of learner-centered teaching. learner-centered teaching includes a number of different methods (such as problem-based learning, service learning, and team learning) that are based on research in cognitive development and effective teaching practices. this literature review focuses on three applied features of learner-centered teaching: a constructive basis for learning, the acquisition of knowledge and skills that are purposeful and transferable, and instructor changes that occur when adopting learner-centered approaches. learner-centered teaching emphasizes knowledge and skills that are constructed by students, rather than directed by instructors. based on the collective writings of educators and psychologists such as bruner, dewey, piaget, and vygotsky, constructivism posits that learning is created by individuals and groups as the result of their current knowledge/thoughts/beliefs interacting with new experiences (e.g., hinchliffe, 2011; richardson, 1997; savery & duffy, 1995). in other words, new knowledge and skills are created as learners attempt to make sense of incongruences between their current knowledge and new experiences. constructivism does not presuppose that learning cannot occur from direct transmission (e.g., attending a lecture). rather, it suggests that such learning is not as complex, meaningful, or enduring as knowledge that is constructed by the individual. as a result, learner-centered teaching emphasizes learning activities and experiences in which instructors facilitate, rather than dictate, students’ construction of knowledge. a second applied feature of learner-centered teaching is that the knowledge and skills that are acquired/constructed are purposeful, relevant, and transferable. the terms “surface” and “deep” describe different types of knowledge and skills, as well as the approaches that students and faculty take toward learning (e.g., marton & säljö, 1976; rhem, 1995). surface learning describes the acquisition and recall of facts from experiences. the knowledge achieved here is considered surface because there is little beyond identification and recall that students can do with it. facts learned while watching jeopardy or playing trivial pursuit are examples of surface knowledge. deep learning also includes the acquisition of information, but emphasizes students’ abilities to apply these ideas to new and varied contexts. perkins (2008) described two types of deep learning/knowledge, performative and proactive. performative knowledge addresses students’ abilities to use their current knowledge (including surface learning) to solve complex and atypical problems. proactive knowledge goes further by applying current knowledge in wholly novel contexts. the notions of surface and deep learning/knowledge are also reflected in bloom’s taxonomy (1956). developed specifically to assist teachers/instructors with educational objectives, this taxonomy identified six levels of cognitive functioning. they are (in order from simple to complex) knowledge, comprehension, application, analysis, synthesis, and evaluation. the knowledge, comprehension, and application levels appear to be reflections of surface learning, while the remaining levels address deep learning. although bloom’s taxonomy has been criticized and modified over the years (e.g., marzano & kendall, 2007), it endures within learner-centered teaching because it identifies multiple aspects of knowledge and their applications to student development. brackenbury, t. journal  of  the  scholarship  of  teaching  and  learning,  vol.  12,  no.  4,  december  2012.   josotl.indiana.edu   14 weimer (2002) provided a detailed description of a third applied feature of learner centered teaching; the changes that occur when an instructor moves from a traditional to learnercentered approach. these include changes to the balance of power, the function of content, the role of the teacher, the responsibility for learning, and the purpose and process of evaluation (see table 1). these five changes reflect the dominant aspects of learner-centered teaching as experienced by educators and students, and as presented in the literature (although weimer’s terms are not used consistently across authors). table 1. weimer’s (2002) list of changes that occur between traditional, instructor-centered and learner-centered teaching models. change traditional, instructor-centered models learner-centered models the balance of power instructors design courses with no student input and are in charge of everything. instructors and students work together to design and implement the course. the function of content content is the primary force driving the course because it provides the foundation upon which skills can be built. content is a tool to develop a knowledge base, new ways of thinking, and self-awareness of learning. the role of the teacher knowledge and skills are directly passed from the teacher to the students. students develop knowledge and skills via discovery; indirectly from the teacher. the responsibility for learning rules and regulations are developed by the instructor to direct student learning. course design accentuates student learning and students’ roles in the process. the purpose and process of evaluation the measurement of student progress to date. assessment is directed at improving instruction and providing students additional learning opportunities.   the balance of power. in traditional, instructor-centered classrooms, all of the decisions about the course (including the course goals, learning experiences, assignments, assessments, and topics discussed) are developed and implemented by the instructor. in a learner-centered classroom, the design of the course is developed through collaboration between the instructor and students. learner-centered instructors do not abdicate all of the decision making to their students. rather, they include the students as colleagues within the development process. in shifting the balance of power, learner-centered instructors are attempting to increase student engagement (as students now become part of the process), and facilitate their path towards constructing their knowledge and becoming independent learners (see also doyle, 2008). the function of content. content is one of the primary forces in traditional teaching. because disciplines are consistently adding new content to their fields, instructor-centered courses are typically designed to cover as much content as possible. this restricts instructors from a) going into much depth about the content and how it can be used, and b) allowing students to explore and apply the content. as a result, their students focus on surface-level knowledge of the content, often memorizing the information and forgetting it shortly after the experience is over. in learner-centered teaching, content is only one aspect of courses. one example of this is fink’s (2008) taxonomy of significant learning. this taxonomy is built around six aspects of course design that would purportedly result in learning that is significant, lasting, brackenbury, t. journal  of  the  scholarship  of  teaching  and  learning,  vol.  12,  no.  4,  december  2012.   josotl.indiana.edu   15 and valued later in life. in other words, deep learning. the categories within this taxonomy were: foundational knowledge (including content), application, integration, the human dimension, caring about one’s own learning, and learning how to learn. thus, content is not forgotten or ignored in learner-centered teaching. rather, it is placed within a larger context by using content to motivate learning in the other categories and vice versa. for example, students within a learner-centered class might be given an application problem that is somewhat outside their current knowledge. after doing some initial work on the problem, they could then be directed toward a source of content (e.g., a book or article) as an aid to solving the problem. as a result, the students would not only be exposed to the content but learn it within a meaningful context. they also learn how to use informative sources to solve problems. the role of the teacher. in the traditional classroom, the role of the instructor is that of the knowing professional who dispenses her/his knowledge directly to the students. this is shown through methods such as lectures that describe new and complicated topics, stories from their past relevant experiences, and summaries of the material that is most relevant to examinations. in learner-centered teaching, instructors provide the architecture for learning but do not directly state all of content to be learned. instead, they design class activities that help students discover the important information. as a result, students learn more from the experience and each other than directly from the teacher. for example, rather than telling students about the speed of sound and having them solve equations on the topic, an instructor may have them measure and calculate the speed of sound for themselves using an oscilloscope and tape measure (stoll, 2008). to paraphrase alison king’s (1993) often cited article title, adapting a learner-centered approach changes an instructor, “from sage on the stage to guide on the side.” the responsibility for learning. weimer (2002) points out that in response to students who are often ill prepared for college education, many faculty have developed strict rules and guidelines. course syllabi are typically the place where such regulations are listed, and a good illustration of how learner-centered classrooms emphasize the responsibility differently. the majority of syllabi focus on information about the instructor, descriptions of the course and its objectives, the topics to be addressed, and grading policy (doolittle & siudzinski; 2010; eberly, newton, & wiggins, 2001; garavalia, hummel, wiley, & huitt; 1999). instead of focusing on rules and regulations, the syllabi from learning-centered courses are designed as a “spring-board for the course experience” (eberly et al., p.68). these syllabi establish the class environment as stimulating and collaborative, emphasize what will be done to facilitate student learning and who is responsible for the actions that take place, and are developed with student input (e.g., bain, 2004; o’brien, millis, & cohen, 2008). by doing so, learner-centered syllabi demonstrate from the start of a course that the primary responsibility for learning belongs to the students and that the faculty member is there to facilitate the process. the purpose and process of evaluation. the changes discussed thus far have emphasized the instructor and students. weimer’s (2002) final change focuses on the purpose and process of evaluation. evaluation is a critical part of the teaching process because it identifies if learning has taken place and, if so, what types of learning occurred. the primary purpose of evaluation in traditional classrooms is as a measure of achievement to date. it is typically summative; in that once the evaluation task has been completed there are no opportunities to show further learning (except, perhaps, from a cumulative evaluation at the end of the course). evaluations in learnercentered classrooms are both formative and summative (e.g., black & wiliam, 2009; fink, 2003; rushton, 2005; yorke, 2003). formative evaluations differ from summative ones because their purposes are a) to assess current knowledge and skills in order to improving learning, while b) brackenbury, t. journal  of  the  scholarship  of  teaching  and  learning,  vol.  12,  no.  4,  december  2012.   josotl.indiana.edu   16 providing additional opportunities for student learning. feedback is an important part of formative evaluation. it extends beyond the identification of correct and incorrect responses to address the lines of thought that lead to errors and promote learning. as bain (2004) stated, “the primary goal [of this type of assessment] is to help students learn to think about their own thinking so they can use the standards of the discipline or profession to recognize shortcomings and correct their reasoning as they go (p. 160).” b. significant learning experiences and the present study. in his 2003 book on designing college courses, l. dee fink emphasized creating significant learning experiences. he described significant learning experiences as an engaged, high-energy process that results in meaningful and lasting change that has value in students’ lives. although fink included detailed instructions for creating learner-centered courses aimed at providing significant learning experiences, he did not report any data on whether or not such lasting experiences occurred. he is not alone. although the scholarship of teaching and learning literature includes a large number of qualitative, quantitative, mixed-methods, survey, and experiential research in support of learner-centered teaching (e.g., black, 1993; hake, 1998; lambert & mccombs, 1997; prince, 2004; weimer, 2006), very little research exists on the long-term effects of learner-centered teaching (i.e., years after the learning experience). despite this, significant learning experiences do occur, as evidenced by instructors and students who can easily recall past learning experiences that were engaging and lead to meaningful and lasting change. the primary research objective of the present study is to determine if there are commonalities between features of learner-centered teaching and past significant learning experiences. if so, what are they and what differences exist? this is a retrospective, exploratory study. no a priori predictions were made as to which features of learner-centered teaching would connect with individuals’ past significant learning experiences and which would not. i. method. the method of investigation was that of phenomenological research, specifically psychological phenomenology (e.g., creswell, 1998). this form of qualitative research examines the experiences of multiple people in regards to a particular event or phenomenon (creswell, 2011; miles & huberman, 1994; moustakas, 1994). in this investigation, the individuals were twentyfour faculty members and administrators in the discipline of communication sciences and disorders. the events/phenomena were their past significant learning experiences. in general, the procedures of phenomenology involve identifying the phenomenon and individuals who have experienced it, collecting data from the individuals about their experience, identifying individual idea statements and grouping them into codes and themes, and reflecting on the meanings of the experience. a. participants. demographic information was available, via self-report, for 22 of the 24 participants. the participants included 16 faculty members, eight administrators, and two clinical instructors/supervisors (participants were allowed to select more than one category). they were brackenbury, t. journal  of  the  scholarship  of  teaching  and  learning,  vol.  12,  no.  4,  december  2012.   josotl.indiana.edu   17 from 15 different states in the united states and one was from new zealand. their college-level teaching experiences ranged from 2 to 40 years (median = 15) and their typical teaching load within a 9-month school year was from 0 to 10 courses (median = 4). the programs that they worked in graduated an estimated 6 to 400 undergraduate (median = 30) and 12 to 65 master’s students (median = 20) per year. information on the gender or ages of the participants was not collected (feminine pronouns will be used below when referring to specific participants). b. data collection. all of the participants were self-selected attendees at a presentation during the 2010 council on academic programs in communication sciences and disorders annual convention titled everything i need to know about teaching i learned from speech-language therapy (brackenbury & shaughnessy, 2010). the stated goals of this session were to 1) introduce attendees to aspects of learner-centered teaching, 2) model and demonstrate learner-centered teaching, and 3) entice attendees to learn more about and increase their use of learner-centered teaching. the session began by stating these goals and providing background information about the presenters. the attendees were then briefly introduced to the idea that many aspects of learner-centered teaching are rooted in past experiences (although no specific features of learnercentered teaching had yet been addressed). next, the attendees were directed to take ten minutes and write an essay about a significant learning experience from their own college education. they were given sheets of paper and instructed to: identify an experience through which you really learned the heck out of something (either a class, an experience within a class, or a clinic/research event). what was it? how did this experience improve you (knowledge, skills, insight, caring, ability to learn…)? what did the instructor do to facilitate your learning? why does this experience stand out over all of the others? the session then focused on five features of client-centered therapy. after each feature was introduced, the attendees were instructed to write about if/how the feature related to the significant learning experience (see brackenbury, 2011). the session concluded with one of the presenters sharing her results from engaging in this same process, a description of how the method of the presentation demonstrated aspects of learner-centered teaching, and suggested references for further learning. of the approximately 80 people who attended the session, 24 turned in a signed consent form and written essay. c. analysis. the data analysis followed procedures for psychological phenomenology as outlined by creswell (1998, 2011). it began with the author and two graduate research assistants identifying their own thoughts, experiences, and biases on the subject of learner-centered teaching and past significant learning experiences. the purpose of this was for each member of the research team to identify and limit the potential effects of their own bias on the analysis and interpretation (i.e., bracketing). the author was the developer and principal investigator of the study. he was one of the co-authors of the conference presentation where the data were collected. at the time of the data collection, he had been a certified speech-language pathologist for 19 years and taught at the university level for nine years. he had used client-based therapy throughout his clinical work and brackenbury, t. journal  of  the  scholarship  of  teaching  and  learning,  vol.  12,  no.  4,  december  2012.   josotl.indiana.edu   18 had transitioned his teaching from the traditional model to learning-centered approximately four years earlier. he was well read on learner-centered teaching, but did not directly use either the general literature on the scholarship of teaching and learning or the references cited above during the analysis process. at the outset of the study, he had suspected that there were some connections between past significant learning experiences and aspects of learner-centered teaching (based on his own college experiences), and was curious to examine which features did and did not relate. the two graduate students were first-year master’s students in communication sciences and disorders. they had both experienced aspects of learner-centered teaching during their undergraduate educations, but were not familiar with its literature. they were given the handout from the conference session and discussed it with the author once, before the analysis process. they reported that they could envision connections between past significant learning experiences and learner-centered teaching, but were unsure what those connections might be. in the second step of the analysis, the author and research assistants inputted the participants’ responses into a spreadsheet. the responses were typed in their original form, with adjustments made only for spelling and minor grammatical errors. questions about responses that were illegible or difficult to understand were resolved through discussion between the research team members. once all of the responses were entered, the author read through the entire spreadsheet and double-checked it with the written essays to make sure that all of the entries were accurate and comprehensible. next, each participant’s essay was separated into individual ideas, which were then sorted into codes and themes. each member of the research team, working separately, divided the responses into individual ideas and placed each idea in its own spreadsheet cell (i.e. horizontalization of the data). every idea was then considered for the potential code it represented and copied into another spreadsheet, organized by theme. thus, the themes were derived from the data themselves and not any previously articulated decisions. once completed, each member of the research team shared their codes and themes (along with the idea statements within them) with the other members of the team. a dialogue process was then used to reach consensus on the final set of themes. as with other forms of research, phenomenological investigations are concerned with the credibility and transferability of their findings. of particular concern are the potential negative influences of bias from the examiners and the degree to which the information presented represents the actual experiences of the participants (creswell, 2011; maxwell, 2005; miles & huberman, 1994). credibility and transferability were established in a number of ways, including the above outlined procedures of bracketing, horizontalization, and triangulation during the code/theme development. in addition, having the participants write their own descriptions minimized potential interferences between their thoughts and reflections and the data analyzed. the median length of the participants’ descriptions of their significant learning experiences was 169 words (with a range of 59 to 459). thus, the 10 minutes they were given to write appears to have been sufficient for the participants to reflect and describe their experiences. because they held onto their original essays throughout the presentation, the participants were able to make revisions at any time during the session (although they were not instructed to do so). visual inspection of the written essays and the flow of ideas within them suggest that few such changes were made. brackenbury, t. journal  of  the  scholarship  of  teaching  and  learning,  vol.  12,  no.  4,  december  2012.   josotl.indiana.edu   19 ii. results. the participants wrote about a number of different types of significant learning experiences. some presented aspects of design for an entire course. these included courses that were based on direct “hands on” experiences, discussions, factual information, research experiences, research evidence, self-direction, service experiences, and student accountability. other participants focused on specific activities within a class, such as case presentations, clinical experiences, designing their own final examination, and interpersonal interactions. research was a third topic that was addressed. participants also wrote about their master’s thesis, doctoral dissertation, and research projects that were part of other courses. the other experiences described included direct clinical work, a workshop attended after graduating, and studying with other students. the research team identified eight themes within the participants’ descriptions of their past significant learning experiences. table 2 presents a summary list of the themes, along with a sample statement for each one. each theme is described in detail below. in both the table and text, the themes are presented in order from those with the greatest number of supporting idea statements to those with the fewest idea statements. it is important to note that the number of supporting idea statements is not considered a measure of a theme’s importance. themes with many idea statements may, for example, reflect common, cursory thoughts; while themes with few idea statements may reflect concepts that are innovative and deep. a. theme i: student responsibility for learning. the participants included many statements that identified themselves as active agents in their learning. these occurred in a variety of activities, including identifying a researcher and following their “question trail” back in the literature, self advising, creating case studies, designing and conducting research, developing a final assessment activity, and teaching their fellow students. their descriptions of these activities included action words such as “backtrack/reanalyze,” “choose,” “connect,” “control,” “create,” “develop,” “dig,” “discover”, “engage,” “experience,” “identify,” and “learn.” although this theme focuses on students’ responsibility for their own learning, instructors were reported to have had important roles in this process. one participant stated, “my professor did very little, yet her "minimal" teaching was significant in my learning. she allowed me to be in control of my own learning. she was supportive and strived as a guide.” another said, he [the instructor] required each of us to be the expert at one reading/book each; to write a short summary for him, present the summary to the class and prepare discussion topics… he simultaneously honored our experience and assumed that we would be responsible. in addition to describing their role in the learning process, some of the participants also made statements regarding the importance of these experiences. for example, one participant reported, “this advising method helped me to become independent and in time [lead me] to what i desired out of my schooling and how i wished to change.” another wrote, “i felt empowered like i could make a difference in the world. i didn't feel like a student. i felt like a professional and i acted like a professional.” a third wrote, “as a student, i did "more" than what i thought my professor expected. hence, i expect more of my students. when given the opportunity to construct their own knowledge they always go above and beyond my expectations.” brackenbury, t. journal  of  the  scholarship  of  teaching  and  learning,  vol.  12,  no.  4,  december  2012.   josotl.indiana.edu   20 table 2. the 8 themes identified from the participants’ reflections on their past significant learning experiences and sample statements of them. theme sample statement i. student responsibility for learning. my professor did very little, yet her "minimal" teaching was significant in my learning. she allowed me to be in control of my own learning. she was supportive and strived as a guide. ii. learning through direct experience or example. this activity gave me hands-on experience with preschool children. it required me to apply my classroom-based knowledge of child language development to preschoolers. it improved my knowledge of the course content and my skills of directing activities with children. iii. responsive instructors. the instructor provided guidance, engaged in conversation allowing me to openly share what i had learned (or thought i learned), and asked thoughtful questions to further my understanding and learning. iv. difficult activities that took time. it was frustrating in a way because there were no "right" or simple answers and there were numerous ways to get to the conclusion. v. connections to previous knowledge and experiences. i learned to figure out relationships of existing knowledge to what i was doing and figure out how it fit or did not fit into my research. vi. direct research experience. reading the literature and interpreting the data in novel ways piqued my interest and enhanced my confidence for research. vii. challenging initial ideas and assumptions. the learning style by dr. v was confrontational, debating each premise, sometimes caused frustration, but by the end of each class, we had the satisfaction of knowing that we had learned something meaningful, or we needed to dig a little deeper to find the answer. viii. rich in content. i was in this course (which really could've been an intro to speechlanguage-pathology course) where i couldn't just bs my way through. there were facts, for the first time, not just opinions (even learned opinions). there were answers, not just suppositions and points of view, and i was enamored in it. b. theme ii: learning through direct experience or example. the participants described a number of experiences in which they were directly involved in a professional setting. these included clinical rotations, hands-on experiences with preschoolers, a treatment workshop, a brain dissection lab, a conference workshop, research, volunteering at a homeless shelter, and taking a class as an adult. these experiences were described as “focused,” “hands-on,” “intense,” “meaningful,” and offering an “invaluable perspective.” numerous lessons were reported from these experiences. some related to clinical management and improving understanding of course materials, such as “i learned two things that day, appreciation of client motivation and never give a kid something you have to take back without warning him first” and “it improved my knowledge of the course content and my skills of directing activities with children.” other experiences increased the participants’ understanding of communication disorders, “[the instructor] taught us that day, better than anything we read in our text, what a fluency disorder can do to the person who has it.” along with increased clinical understanding, one participant wrote about how the experience changed her self-image. i loved constructing this assignment. i could be creative. i could do "more" than complete an assignment. i could make a difference by volunteering my time. i felt really good about myself as a learner, but more importantly as a human being. i felt connected to something larger than myself. brackenbury, t. journal  of  the  scholarship  of  teaching  and  learning,  vol.  12,  no.  4,  december  2012.   josotl.indiana.edu   21 c. theme iii: responsive instructors. as reported above, some of the comments made regarding student responsibility for learning included remarks about the instructor. along with these were a number of other comments directed specifically at the instructor and his/her actions. specific instructor activities that were mentioned included case studies, lectures, socratic dialogue, stating and challenging assumptions, student created assignments/assessments, and two-minute quizzes. across activities, the instructors were described as “accessible, encouraging, and responsive” and “reflective, even-handed, non-judgmental, but always organized, logical, goal-directed, serious and building momentum toward deep understanding and knowledge.” they “provided guidance, engaged in conversation allowing me to openly share what i had learned (or thought i learned), and asked thoughtful questions to further my understanding and learning.” the abilities to reason with students and make “great connections” were also listed. one participant described her instructor and his impact on the experience as, he was able to assess our learning based on our ability to apply and make sense of the abstract concepts within the framework of our institution. feedback was seldom about right and wrong. grades were irrelevant. it was about untangling concepts and we could keep trying until we had our own aha moment. d. theme iv: difficult activities that took time. high levels of difficulty and significant amounts of time on task were described throughout the participants’ significant learning experiences. course features that were identified along with this theme included “a very large, very intimidating text,” “extensive reading,” and presentations that were scrutinized “until the presenter could justify the reason why that approach had, or had not, been successful.” one participant summarized her experience by saying that the class “was like taking a trip through unknown and mysterious territories.” along with descriptions of difficulty, some of the participants also commented on the amount of time they spent on the research. for example, one participant stated, “i spent hundreds of hours analyzing this data set. when i started i had never looked at a spectrogram. by the end i could almost identify the word just by looking at the screen.” another participant said, “i worked harder on this project than any other assignment in my undergraduate career. yet, it was the most enjoyable project i completed across my studies.” finally, one participant described the benefits of having spent time on the task, while also expressing concerns about not having had as much time as she wanted: “[i] felt i could absorb anything that i read, if i could only dedicate the time to do it.” e. theme v: connections to previous knowledge and experiences. one of the common benefits described was being able to see connections between ideas and information within the experience and prior knowledge. some of the connections were reported within a course, as shown by the statement “… each class period he would ask for us to relate about our previous cases.” other connections were made between clinical and classroom experiences, such as “it required me to apply my classroom-based knowledge of child language development to preschoolers.” still others, connected research and prior experiences, “i learned brackenbury, t. journal  of  the  scholarship  of  teaching  and  learning,  vol.  12,  no.  4,  december  2012.   josotl.indiana.edu   22 to figure out relationships of existing knowledge to what i was doing and figure out how it fit or did not fit into my research.” f. theme vi: direct research experience. as shown by the quotes in the previous themes, research was a common activity within the significant learning experiences. three types of research were discussed. the first was research within a class; which included creating literature reviews of prior research, developing and refuting points of view based on the literature, and designing and implementing small experiments. the other two types of research experiences discussed were master’s theses and doctoral dissertations. in both of these experiences the participants talked about developing research questions, analyzing data, having to “back-track and re-analyze data files when i learned something new.” the stated benefits of these research experiences included increased understanding of the connection between research questions and methodology, increased abilities to absorb new information, and enhancing confidence for doing more research. some of the faculty members discussed how their instructor / research advisors facilitated their learning. for example, one faculty described an instructor who provided guidance, engaged in conversation allowing me to openly share what i had learned (or thought i learned), & asked thoughtful questions to further my understanding & learning. this experience stands out because i was actively engaged in my own learning. another faculty stated, “my mentor would ask probing questions and continually bring new research articles into the mix. after i became accustomed to this process, it was natural to enter into conversation with my mentor about my observations in the data set.” g. theme vii: challenging initial ideas and assumptions. some of the significant learning experiences described contexts in which the participants’ initial ideas and assumptions were directly challenged. one case reported on an instructor who was confrontational, debating each premise, sometimes causing frustration, but by the end of each class, we had the satisfaction of knowing that we had learned something meaningful, or we needed to dig a little deeper to find the answer. this approach probably would not be tolerated by students of today, but it rewarded good thinking and case building/management skills. in another experience, the instructor would spend the first half of the class laying out a body of research that lead to a logical conclusion. after the break, he went through some same and new research that systematically pulled the rug out from under our beliefs…the process pulled us in. we reasoned along with the instructor. [it] remains the prime example of logical reason and learning in my background. h. theme viii: rich in content. most of the comments that were made about content were related to developing connections between content and aspects of learning (see themes v and vii). two of the participants discussed content richness as central features of their significant learning experiences. in one brackenbury, t. journal  of  the  scholarship  of  teaching  and  learning,  vol.  12,  no.  4,  december  2012.   josotl.indiana.edu   23 case, the participant had previously been a high achieving english literature major who perceived that field to be over-reliant on opinions. her description of her first course in communication sciences and disorders included the following. there were facts, for the first time not just opinions (even learned opinions). there were answers, not just suppositions and points of view, and i was enamored in it…ever since that experience, i crave content myself and i try to make sure there is a great amount of content in all the classes i teach. the other participant who directly wrote about content was unhappy with a discussion-based course because the bulk of the knowledge did not come from the instructor, there was not as much content as she was expecting, and she felt that she needed more structure. she summarized her comments by saying, now i think that course was probably problematic because as a service learner i needed more structure to help me grapple with how to handle content as well as more would-be content to handle. content-to-handle was probably there in the class but seemed less clear to me because i expected it to come in a particular format; a detailed set of readings [or] a textbook. in fact on reflection, this sense of having rules for how to handle content and ideas about where to find that content are enduring concern for me as a teacher and for my students. what i learned the heck out of was about my own learning. iii. discussion. this investigation examined potential connections between features of learner-centered teaching and past significant learning experiences. current university faculty members and administrators in the discipline of communication sciences and disorders wrote short essays about a past college-level experience where they “really learned the heck out of something.” the participants wrote about a variety of experiences, from individual classroom assignments, to whole courses, to research and clinical experiences. a phenomenological analysis revealed eight recurring themes within these experiences: student responsibility for learning, learning through direct experience or examples, responsive instructors, difficult activities that took time, connections to previous knowledge and experiences, direct research experiences, challenging initial ideas and assumptions, and rich in content. a. comparisons with applied features of learner-centered teaching. the primary research questions addressed in this study were: are there commonalities between features of learner-centered teaching and past significant learning experiences? and, if so, what are they and what differences exist? these questions can now be answered by comparing the eight themes found in this study with the applied features of learner-centered outlined earlier. there appear to be many direct connections between the applied methods of learnercentered teaching and the eight themes identified in this study. for example, the constructivist nature of learner-centered teaching promotes learning that is created by individuals and groups, as the result of their current knowledge/thoughts/beliefs interacting with new experiences. this intersection was demonstrated by the participants through the themes of connections to previous knowledge and experiences, and challenging initial ideas and assumptions. the themes of brackenbury, t. journal  of  the  scholarship  of  teaching  and  learning,  vol.  12,  no.  4,  december  2012.   josotl.indiana.edu   24 student responsibility for learning and learning through direct experience or examples may also be reflective of constructivism, by allowing the participants to have had some ownership of their development and providing direct professional experiences. learner-centered teaching’s goal of deep, performative, and proactive knowledge also appears to connect with the significant learning experiences. the participants’ comments suggested that the many complexities that come with difficult activities, especially those that provide direct professional experiences and examples (such as research), prompted them to deeply consider the content and how it could be applied to a variety of problems. likewise, the challenges to their initial ideas and assumptions made them “dig a little deeper to find the answer.” weimer’s (2002) list of changes that occur when an instructor moves from a traditional to learner-centered approach provided another framework for comparison. the themes identified in this study appear to directly reflect almost all of these changes. for example, learner-centered teaching changes the balance of power from the teacher only to a shared responsibility among the teacher and students. the theme of student responsibility for learning addressed this through comments about how the participants were allowed to develop and/or select specific assignments and assessments, research topics, and advising. the participants reported that having some say in their classroom experience was empowering. it allowed them to think and do more, and helped them to feel what it is like to be a professional. likewise, statements within the themes of responsive instructors and learning through direct experience described faculty who included student input as part of the design of the educational experiences. the function of content also appeared to have a direct connection with the significant learning experiences, although there were important differences within comments made regarding content. most of these statements supported learner-centered teaching’s use of content as a tool for developing the students’ knowledge, skills, and self-awareness. these centered around the experiences enlightening prior content knowledge, causing reconsideration of prior content knowledge, and/or requiring the acquisition of new content knowledge. similar sentiments were also expressed in the themes of connections to previous knowledge and experiences, and challenging initial ideas and assumptions. in other words, the participants showed an appreciation for deep learning, through the application of content to new and varied situations. two participants wrote specifically about experiences in which they had wanted more content. even in these cases, their comments were directed towards content that was relevant and would have helped them learn more, as opposed to wanting more content in order to learn more facts. the role of the instructor in learner-centered teaching is to facilitate the students’ development of knowledge and skills. comments within the theme of responsive instructors demonstrated this feature. the participants, for example, described teachers who “provided guidance”, allowed open sharing of student learning, and focused on student conceptual development. along with changing the balance of power, learner-centered teaching creates a greater sense of student responsibility for learning. although observed across many of the themes, this feature was demonstrated the most prominently within student responsibility for learning. the participants described themselves as being actively involved in their experiences. they described their learning as intentional and purposeful. they saw this as helping them to go beyond what they would have before and to become independent learners. brackenbury, t. journal  of  the  scholarship  of  teaching  and  learning,  vol.  12,  no.  4,  december  2012.   josotl.indiana.edu   25 weimer’s (2002) fifth change with learner-centered teaching, the purpose and process of evaluation, was the feature of her list that was not evident as a theme. only two participants made direct comments about the purpose and process of evaluation. one reported that she was given the freedom in a course on manual communication (e.g., sign language and communication boards) to design her own final exam experience. the other described her instructor’s contribution to her learning by stating that, “feedback, was seldom about right and wrong. grades were irrelevant. it was about untangling concepts and we could keep trying until we had our own aha moment.” it is unclear why more of the participants did not comment on the evaluation components of their experiences. it may be that the evaluation part of their experience was a) not important to them because they were focused on their own learning, b) typical of other courses, c) unmemorable, or d) simply not a part of the experience that they thought to include in their descriptions (as their instructions did not identify specific aspects of the features to write about). taken together, the themes identified in this study support a connection between applied features of learner-centered teaching and past significant learning experiences. these connections, however, should be considered as preliminary for a number of reasons. first, the data were based on the participants’ recollections of events that occurred multiple years in the past (in some cases decades later). their memories may have been influenced by their other experiences that have occurred since then, including encounters with scholarly teaching. second, the participants had self-selected to attend the conference session on teaching and learning. although features of learner-centered teaching had not yet been addressed in the session, these may have been faculty who were primed to think in this direction. third, because all of the participants were within the same discipline, they may have been prone to think in terms of learner-centered teaching. this is not likely to be the case, however, because learner-centered teaching is not the dominant teaching method being used in the discipline (brackenbury, folkins, & ginsberg, 2011) and the discipline’s research on it is developing, but not substantial. in addition, potential connections between client-centered therapy and learner-centered teaching had not been previously discussed (either within the published literature or previous conference presentations), suggesting that the participants were not highly predisposed to make these associations. finally, the author and research assistants were all predisposed, to varying degrees, to consider connections between past significant learning experiences and learner-centered teaching. given the procedures taken to address issues of credibility and transferability, however, these potential confounders do not appear to have had a significant negative influence on the results of this study. b. implications and conclusions. the results of this study demonstrate that features of learner-centered teaching can be directly connected with past significant learning experiences. the connections identified between learner-centered teaching and the significant learning experiences suggest that these principles of learning have been around for a long time. although these connections are not entirely surprising, as the cognitive underpinnings of learner-centered teaching have been around for numerous decades, they suggest that learner-centered teaching can result in learning that lasts for many years after the experience. however, not all applied features of learner-centered teaching were connected to past significant learning experiences (most notable the purpose and of brackenbury, t. journal  of  the  scholarship  of  teaching  and  learning,  vol.  12,  no.  4,  december  2012.   josotl.indiana.edu   26 evaluation). further research into learner-centered teaching and past significant learning experiences should help to specify these commonalities and differences. the observed connections between learner-centered teaching and past significant learning experiences may prove to be helpful for faculty who are new to this type of teaching. rather than considering learner-centered teaching as a new, challenging way of educating students, their first steps towards being more learner-centered could be to consider their own past significant learning experiences. they could then identify the aspects that made these experiences so powerful and then look within the scholarship of teaching and learning literature to find evidence of connected features. it would also provide a familiar platform from which they can build learner-centered teaching into their own instruction. additionally, the types of activities and instructors that the faculty described in their experiences could be used to inspire teachers to develop similar experiences and teaching personas. a similar case can be made for using the data and results from this study to assist students who are new to learner-centered teaching. many students have difficultly when they first encounter learner-centered teaching, especially if they had not experienced it before the college level (e.g., doyle, 2008). it may be beneficial for learner-centered instructors to have their students do the same reflection that the participants in this study completed. discussing the students’ experiences and connecting features of them with the methods of the course may help facilitate their understanding of why the class was designed differently than their previous courses. the results of this study suggest that applied principles of learner-centered teaching can be connected to significant learning experiences. although there are no guarantees that the learner-centered experiences that faculty provide will be later identified by their students as significant, educators can use the idea of creating significant learning experiences as a guiding principle within and across their courses. the connections demonstrated here suggest that learner-centered teaching can provide both a framework and specific methods for doing so. references bain, k. (2004). what the best college teachers do. cambridge, ma: harvard university press. barr, r.b., & tagg, j. (1993). a new paradigm for undergraduate education. change, 27, 12-25. black, k.a. (1993). what to do when you stop lecturing, journal of chemical education, 70, 140-144. black p. & wiliam, d. (2009). developing the theory of formative assessment. educational assessment, evaluation, and accountability, 21, 5-31. bloom, b. s. (ed.) (1956). taxonomy of educational objectives: book 1 cognitive domain. new york: longman. brackenbury, t. (2011, november). how past learning experiences compare with learnercentered teaching. poster presented at the annual convention of the american speech-languagehearing association, san diego, ca. brackenbury, t. journal  of  the  scholarship  of  teaching  and  learning,  vol.  12,  no.  4,  december  2012.   josotl.indiana.edu   27 brackenbury, t., folkins, j., & ginsberg, s. (2011, november). the signature pedagogy of communication sciences and disorders. presentation at the annual convention of the american speech-language-hearing association, san diego, ca. brackenbury, t., & shaughnessy, p. (2010, april). everything i need to know about teaching, i learned from speech and language therapy. paper presented at the capcsd convention, san antonio, tx. creswell, j.w. (1998). qualitative inquiry and research design. thousand oaks, ca: sage. creswell, j.w. (2011). educational research: planning, conducting, and evaluating quantitative and qualitative research (4th ed.). upper saddle river, nj: pearson. doolittle, p.e., & siudzinski, r.a. (2010). recommended syllabus components: what do higher education faculty include in their syllabi? journal on excellence in college teaching, 20, 29-61. doyle, t. (2008). helping students learn in a learner-centered environment. sterling, va: stylus. eberly, m.b., newton, s.e., & wiggins, r. a. (2001). the syllabus as a tool for student-centered learning. journal of general education, 50, 56-74. fink, l.d. (2008). creating significant learning experiences. san francisco, ca: jossey-bass. garavalia, l.s., hummel, j.h., wiley, l.p., & huitt, w.g. (1999). constructing the course syllabus: faculty and student perceptions of important syllabus components. journal on excellence in college teaching, 10, 5-21. hake, r.r. (1998). interactive-engagement vs traditional methods: a six-thousand student survey of mechanics test data for introductory physics courses. american journal of physics, 66, 64-74. hinchliffe, g. (2011). what is a significant educational experience? journal of philosophy of education, 45, 417-431. king, a. (1993). from sage on the stage to guide on the side. college teaching, 41, 30-35. lambert, n.m., & mccombs, b.l. (1997). how students learn. washington, dc: american psychological association. marton, f., & säljö, r. (1976). on qualitative differences in learning: i-outcome and process. british journal of educational psychology, 46, 4-11. marzano, r.j. & kendall, j.s. (2007). the new taxonomy of educational objectives (2ond ed.). thousand oaks, ca: corwin press. brackenbury, t. journal  of  the  scholarship  of  teaching  and  learning,  vol.  12,  no.  4,  december  2012.   josotl.indiana.edu   28 maxwell, j.a. (2005). qualitative research design: an interactive approach (2nd ed.). thousand oaks, ca: sage publications. miles, m.b., & huberman, a.m. (1994). qualitative data analysis (2nd ed.). thousand oaks, ca: sage publications. moustakas, c. (1994). phenomenological research methods. thousand oaks, ca: sage publications. o’brien, j.g., millis, b.j., & cohen, m.w. (2008). the course syllabus (2nd ed.). san francisco, ca: jossey-bass. perkins, d. (2008). beyond understanding. in r. land, j. h. f. meyer, & j. smith (eds.) threshold concepts within the disciplines, 3-19. rotterdam: sense publishers. prince, m. (2004). does active learning work? a review of the research. journal of engineering education, 223-231. rhem, j. (1995). deep/surface approaches to learning: an introduction. the national teaching and learning forum, 5, 1-5. richardson, v. (1997). constructivist teacher education. new york: routledge. rushton, a. (2005). formative assessment: a key to deep learning? medical teacher, 27, 509513. savery, j.r., & duffy, t.m. (2001). problem based learning: an instructional model and its constructivist framework (tech. rep. no. 16-01). indiana university, center for research on learning and technology. stoll, c. (2008). 18 minutes with an agile mind. presentation at ted2006. retrieved march 30, 2012, from http://blog.ted.com/2008/03/27/clifford_stoll/ weimer, m. (2002). learner-centered teaching. san francisco, ca: jossey-bass. weimer, m. (2006). enhancing scholarly work on teaching and learning. san francisco: jossey-bass. yorke, m. (2003). formative assessment in higher education: moves towards theory and the enhancement of pedagogic practice. higher education, 45, 477-501. 1233chenneville final journal of the scholarship of teaching and learning, vol. 12, no. 4, december 2012, pp. 58 -75. civic engagement in the field of psychology tiffany chenneville1, susan toler2, and vicki t. gaskin-butler3 abstract: the purpose of this article is to describe the importance of, and recommendations for how best to promote, civic engagement among undergraduate psychology majors. in this article, we will describe how the goals of civic engagement are consistent with the specific curricular goals of undergraduate psychology programs. we also will (a) review the empirical support for civic engagement in the field of psychology and describe the implications of this method for teaching students about diversity; (b) discuss some of the challenges associated with incorporating civic engagement in psychology courses as well as provide strategies for overcoming these challenges; (c) discuss some of the unique ethical issues related to civic engagement in the field of psychology; and (d) provide recommendations, using specific examples, for how to incorporate service-learning activities as a means of encouraging civic engagement in psychology courses. keywords: civic engagement, psychology, service-learning i. introduction. working for the common good is a fundamental concept in the field of psychology and, in fact, in this article, we argue that the curricular goals of undergraduate psychology programs are consistent with the goals of civic engagement. promoting civic engagement in the field of psychology is supported by the literature, yet challenges exist when attempting to incorporate civic engagement as a central component of psychology courses. the concepts of service-learning and civic engagement are not new to the field of psychology. william james, considered by many to be the “father” of psychology in america, argued a century ago that youth should be encouraged to participate in community service as a means of mobilizing people by giving them a sense of common purpose. james argued that our society addressed the need for solidarity and a sense of common purpose by mobilizing military action rather than social action (james, 1910). john dewey, president of the american psychological association in 1899, was the founder of progressive education, pedagogy that became the foundation for what we know today as service-learning or civic engagement (rocheleau, 2004). dewey proposed that students need to struggle with social problems through the application of concepts and principles learned in the classroom. this emphasized the pedagogical method of using community service to enhance students’ understanding of facts and theories. both dewey and james emphasized the importance of theoretical relevance. that is, viable theories must be capable of addressing every day social and ethical problems. progressive education must encourage students to actively participate in the democratic process and tackle difficult issues related to social justice. 1 assistant professor of psychology, university of south florida st. petersburg, chennevi@stpt.usf.edu 2 smtoler@usfsp.edu 3 vgaskinb@mail.usf.edu chenneville, t., toler, s., and gaskin-butler, v. t. journal of the scholarship of teaching and learning, vol. 12, no. 4, december 2012. josotl.indiana.edu 59 as the progressive education movement evolved, dialogues returned to the responsibility of educators to produce citizens who had both the knowledge and the sense of civic responsibility to engage in community activism and social problem solving. this model emphasized nurturing a set of values and social ideals that produced citizens who participated in the democratic process and were civically engaged. the educational goal went beyond facilitating the acquisition of facts and theories to nurturing a set of values and social ideals that produced citizens who actively participated in the democratic process and were civically engaged. civic engagement, then, emphasized the educational goal of instilling democratic and social values. in contrast the service-learning model emphasized understanding facts and knowledge from the classroom through community service. near the end of dewey’s life, the pedagogical foundations of service-learning were criticized for lack of academic rigor and breadth. there was a growing sense that progressive education became technical education, preparing students for jobs rather than careers (rocheleau, 2004). additional criticisms of this movement focused on indoctrination of values that promoted a brand of liberal citizenship and community activism. it was argued that traditional education focused on imparting facts, knowledge and theories in a value neutral setting (fish, 2003). educational trends in the mid to late twentieth century shifted to a focus on careers and self-interest concurrent with a sense of social and political alienation (bellah, madsen, sullivan, swindler, & tipton, 1985). rocheleau (2004) has suggested that this social alienation and the call for relevance in the college classroom set the stage for a resurgence of service-learning and civic engagement near the end of the twentieth century. social crises such as the terrorist acts of september 2001, the economic meltdown of 2008 and the ethical and moral bankruptcy of our political and business leaders have given educators cause to reconsider their responsibilities to educate ethical and responsible citizens as well as scholars. it is no coincidence that president obama’s call for a new type of civically engaged american citizen emerged from his own experience as a community organizer in the south side of chicago and as a lecturer at the university of chicago. the university of chicago was the birthplace of john dewey’s philosophy of pragmatism and progressive education and home to the first psychology laboratory. a. service-learning versus civic engagement. “service-learning” and “civic engagement” often are used interchangeably and, while they are related, there are important distinctions between the two terms. the american psychological association (n.d.) defines civic engagement as “individual and collective actions designed to identify and address issues of public concern”. bringle and hatcher (1996) describe servicelearning as “a credit-bearing educational experience in which students participate in an organized service activity that meets identified community needs and reflects on the service activity in such a way as to gain further understanding of course content, broader appreciation of the discipline, and an enhanced sense of civic responsibility” (p. 222). while interrelated, these terms clearly signify different constructs. in this article, we conceptualize service-learning as a means by which citizen scholars are created. service-learning typically refers to a pedagogical approach that allows for students to better understand course material by providing a means by which to apply what they are learning in real life settings. in and of itself, in the absence of clearly specified goals, service-learning can be self-serving for the student. while community members may benefit in the short-term from chenneville, t., toler, s., and gaskin-butler, v. t. journal of the scholarship of teaching and learning, vol. 12, no. 4, december 2012. josotl.indiana.edu 60 the students’ participation in community settings, service-learning that is designed as a studentcentered approach may lack the components necessary to achieve significant changes within the student, the university, or the community. civic engagement extends beyond mere servicelearning. civic engagement reflects a reciprocal relationship between the student, the university, and the community. therefore, intention is important when designing service-learning activities. in this article, we discuss service-learning as a means by which to promote civic engagement, which should be the ultimate goal of all service-learning activities. b. learning outcomes in psychology. the american psychological association supports the idea that, as educators, we have a responsibility to inculcate our students with certain values. indeed, an overarching goal of a liberal arts education is to assist students in “recognizing, understanding, and respecting the complexity of sociocultural and international diversity” (american psychological association [apa], 2007, p. 210). suggested student learning outcomes include sensitivity to individuals from diverse backgrounds and abilities, heightened consciousness of prejudicial attitudes and discriminatory behaviors, and an understanding of how “power, privilege and oppression may affect prejudice, discrimination, and inequity” (apa, 2007, p. 20). what better classroom is there for understanding these social forces than in a homeless shelter, an inner city elementary school, or a battered women’s shelter? service-learning opportunities provide exactly these types of pseudo classrooms in which students not only extend their learning of concepts taught at the university, but also give back to the community in some meaningful way. the american psychological association identifies even more specific learning outcomes to educate psychology majors in the value of “empirical evidence and tolerance for ambiguity” as well as the importance of “assessing and justifying their engagement with respect to civic, social, and global responsibilities” (apa, 2007, p. 17). allowing students to engage in community service while at the same time giving our theories and facts the pragmatic test of realism offer powerful and sometimes challenging learning opportunities. in the sections to follow, we will discuss some of these learning opportunities in the field of psychology as well as some of the important considerations to be made when thinking about implementing civic engagement activities. ii. civic engagement in psychology. in this section, we will discuss the empirical support for civic engagement via service-learning in psychology courses and the implications of this method for teaching students about diversity. a. empirical support. while the benefits of service-learning have been well documented (e.g., prentice, 2007), there is a dearth of information about best practices for designing and implementing service-learning activities in higher education courses. and, most of what is available falls outside the field of psychology. however, there is some research that specifically addresses the benefits of, and recommendations for, implementing service-learning and civic engagement into the undergraduate psychology curriculum (e.g., kretchmar, 2001; fenzel, 2001; grimes, 1998; and mcdonald, caso, & dee, 2005). chenneville, t., toler, s., and gaskin-butler, v. t. journal of the scholarship of teaching and learning, vol. 12, no. 4, december 2012. josotl.indiana.edu 61 kretchmar (2001) reported that, when given a choice between participating in a servicelearning activity and completing a research project, over 88% of students enrolled in an introductory psychology course opted for service-learning. in this course, students were given the opportunity to secure their own placements from a list of possible community agencies (e.g., habitat for humanity, juvenile detention center). of those who participated in service-learning, the large majority (80%) reported positive results related to learning and service commitment. similarly, fenzel (2001) reported favorable course evaluations from the large majority (between 89 and 100%) of undergraduate students enrolled in one of three courses with a significant service-learning component (i.e., introductory psychology, child psychology, substance abuse and its effects). the service-learning activities varied by course. for example, students enrolled in introductory and child psychology courses participated in tutoring programs at various sites while students enrolled in the substance abuse course were required to teach a 6hour unit on drug use prevention to middle school students. most students (77%) indicated an increase in personal and social responsibility and many (59 to 88%) reported an increased interest in community service as a result of their service-learning experience. grimes (1998) also reported the positive results of incorporating service-learning into an applied psychology course. in this case, the service-learning component involved mentoring atrisk, african american middle school students. of the 14 students enrolled in the course, over 80% described the course as one of the best courses they had ever taken and believed the course should become part of the required curriculum. over 70% reported that the course increased their understanding of psychological concepts and their interest in public service. and, in fact, over 90% indicated an interest in continuing their work as a mentor. citing not only the benefits for students, but also for instructors and the community, mcdonald, caso, and dee (2005) describe the value of teaching and learning operant principles in animal shelters. as part of a service-learning requirement, students enrolled in a psychology of learning course volunteered as dog trainers in an animal shelter. in addition to enhanced selfesteem and an increased interest in social causes among students, both instructors and community members benefited from this requirement. b. teaching diversity. the importance of incorporating a focus on multicultural issues into psychology courses cannot be overstated. indeed, many undergraduate psychology curricula include specific courses designed to address issues related to diversity (e.g., multicultural psychology, psychology of women); and in addition, or at the very least, imbed issues of diversity by incorporating multicultural issues within all courses, regardless of the subject matter (e.g., discussing how theories of aging may differ across cultures in a developmental psychology course). the intersection between civic engagement and teaching diversity can be viewed from two different perspectives. first, promoting civic engagement among students via service-learning activities may serve as a vehicle for teaching students about diversity by exposing them to other cultures and sub-cultures within the community, regardless of the intended content of the course. for example, students may be exposed to the conditions of inner city schools while completing the service-learning component of a behavior modification course. second, civic engagement can be incorporated into courses specifically designed to teach issues related to diversity (e.g., psychology of women, multicultural psychology). chenneville, t., toler, s., and gaskin-butler, v. t. journal of the scholarship of teaching and learning, vol. 12, no. 4, december 2012. josotl.indiana.edu 62 mio, barker, and tumambing (2009) define multicultural psychology as “the systematic study of behavior, cognition, and affect in settings where people of different backgrounds interact” (p. 3). therefore, the pedagogical task of psychology courses containing multicultural content is to enable students to cross boundaries of race, gender, ability, class, and other differences, in order to better prepare them to make meaningful connections with others in our increasingly diverse society. multicultural psychology aids students in understanding the impact of culture on the development of worldviews, communication patterns, acculturation, and identity development. in addition, students in multicultural psychology classes are acquainted with issues related to stereotyping, prejudice, discrimination, and racism. cultural factors related to health and mental health also are concerns of multicultural psychology. the ultimate goal of multicultural coursework is to move students toward the development of multicultural competence (mio et al., 2009). the american psychological association states that civic engagement includes a range of civic acts and “[a]n engaged citizen should have the ability, agency, and opportunity to move comfortably among these various types of civic acts” (apa education directorate, 2009, p. 1). therefore, civic engagement is particularly important in multicultural classes. first, it affords students the opportunity to “address issues of public concern” (apa education directorate, 2009, p. 1). second, civic engagement activities enable students to expand their knowledge of and appreciation for the experiences of those from diverse communities (goodman & westolatunji, 2008; barrow, 2008). beyond incorporating civic engagement into multicultural psychology courses, multicultural content can be incorporated into all psychology courses through service-learning activities designed to enhance communication, collaboration, and critical thinking across race, ethnicity, and culture (grier-reed, detzner, poch, & staats, 2010). hurtado (2007) argues for linking diversity, education, and civic engagement as a means by which we can “advance students’ awareness of the origins of complex social problems and employ new forms of pedagogy involving dialogue, discussion, experiential learning, reflection, social critique, and commitment to change” (p. 186). in essence, incorporating civic engagement activities in psychology courses promotes the development of multicultural competence. laird, engberg, and hurtado (2005) noted that colleges and universities’ creation of programs that foster students’ “engagement with social diversity” (p. 448) prepare students to deal with a diverse society. furthermore, teaching courses that discuss multicultural content also increases the likelihood that students will work and live in diverse communities after graduating from college (laird, engberg, & hurtado, 2002). iii. challenges. despite the fact that research has demonstrated the value of civic engagement for undergraduate psychology majors (fenzel, 2001; grimes, 1998; kretchmar, 2001; mcdonald, caso, & fugit, 2005), barriers exist that can deter faculty from incorporating service-learning as a means of promoting civic engagement into their courses. these barriers, or challenges, can be categorized into three groups: administrative challenges, faculty challenges, and student challenges. while these challenges are discussed as distinct categories in the sections to follow, it is important to recognize that a significant amount of overlap exists between these areas. for example, chenneville, t., toler, s., and gaskin-butler, v. t. journal of the scholarship of teaching and learning, vol. 12, no. 4, december 2012. josotl.indiana.edu 63 administrative challenges cannot be separated entirely from faculty challenges as there clearly is a reciprocal relationship between administrative demands and faculty commitments. a. administrative challenges. it goes without saying that administrative support is imperative if efforts to promote civic engagement are to be successful. without a stated commitment, higher education institutions are likely to get sidetracked with competing initiatives. however, even at universities that include civic engagement as part of their mission statement, funding is likely to be an issue. especially in difficult economic times where budget cuts result in universities that are understaffed, the support necessary to sustain initiatives often is lacking. in addition to funding issues, a lack of enthusiasm for civic engagement may underlie resistance at the administrative level. further, institutions without a process for centralized decision-making and those that lack a strong sense of shared governance are likely to find it difficult to implement service-learning into the curriculum, much less promote civic engagement university wide (ward, 1996). similarly, lack of presidential support and poor faculty participation are likely to be barriers to institutionalizing service-learning initiatives that promote civic engagement. b. faculty challenges. faculty interested in incorporating service-learning components into their courses or otherwise promoting civic engagement face many challenges, all of which ultimately relate to time constraints. developing a curriculum that promotes civic engagement is time consuming. many university faculty juggle multiple roles and responsibilities (e.g., teacher, researcher, mentor, and/or faculty governance) making it difficult to devote the time and energy necessary to successfully launch a civic engagement initiative, much less sustain one, which requires ongoing supervision and documentation. bulot and johnson (2006) estimate that courses that contain service-learning require an extra two to ten hours per week (4.5 hours on average) compared to courses without a service-learning component. even at institutions where administrative support is adequate, which may reduce the demands placed on faculty with regard to managing civic engagement initiatives, faculty must prioritize competing demands on their time. this is especially true of faculty interested in tenure or promotion where the value of promoting civic engagement is not clearly articulated. a lack of faculty development related to service-learning and civic engagement also may present a challenge. even faculty who are intrigued by the concept of civic engagement may not believe they possess the knowledge or skills to successfully implement civic engagement activities. lacking self-efficacy with regard to civic engagement, faculty likely will avoid pursuing opportunities to incorporate service-learning in their courses. indeed, some faculty may not even realize they already promote civic engagement in their courses. in addition to adequately defining service-learning and civic engagement, faculty development efforts should emphasize the need for creating a balance between the needs of the course, the program, and the department with the needs of the students and the community agencies involved (osborne & renick, 2006). forging community partnerships is likely to be one of the greatest challenges faced by faculty members interested in civic engagement. identifying and contracting with community agencies can be both time consuming and confusing. legalities must be addressed, which can chenneville, t., toler, s., and gaskin-butler, v. t. journal of the scholarship of teaching and learning, vol. 12, no. 4, december 2012. josotl.indiana.edu 64 serve as a significant barrier. the faculty-community agency partnership must be crafted carefully to ensure that student involvement extends beyond mere volunteerism (bolut and johnson, 2006). without clear expectations, both parties may be less than satisfied with the experience. in a study of faculty implementing service-learning in undergraduate and graduate gerontology-related courses, bulot and johnson (2006) described the need to overcome stereotyped thinking among students as a significant barrier. specifically, the issue becomes how faculty can confront discriminatory attitudes among students without losing focus on other important course-related content. bulot and johnson (2006) also describe the challenge of balancing flexibility and rigidity, noting the importance of clear expectations but also acknowledging the need to take into consideration various student variables when designing service-learning activities. c. student challenges. various student variables may be perceived as challenges to the successful implementation and maintenance of service-learning and civic engagement initiatives. for example, the age, maturity level, and life experience of individual students may affect the extent to which certain servicelearning activities are useful. bulot and johnson (2006) suggest that nontraditional students (24 years or older) may pose unique challenges given the likelihood of competing responsibilities such as family obligations and work demands. on many campuses, especially in urban settings, even traditional students (under the age of 24) are working fullor part-time to put themselves through college. thus, instructors must be creative when incorporating service-learning into their courses. in fact, long, larsen, hussey, and travis (2001) stress the importance of allowing students choices when designing service-learning activities to avoid some of the difficulty associated with students’ outside commitments, which bulot and johnson (2006) refer to as the social situation. another challenge among students may be confusion about what is expected of them, which speaks again to the need for clearly articulated goals and expected outcomes as well as functional relationships between faculty/universities and participating community agencies. finally, students may simply lack interest in service-learning, which represents a challenge. d. distance learning. despite controversy about its effectiveness, distance learning is a current trend that must be acknowledged. while a discussion about the implications of distance learning on civic engagement are beyond the scope of this article, we would be remiss if we did not mention the inherent difficulties associated with incorporating service-learning into online courses. kenworthy-u’ren (2008) provides an excellent summary of the key implications associated with incorporating service-learning into online courses: (1) reflection activities must be adapted to an online environment; (2) service-learning projects will have to be re-structured and maybe even re-operationalized, taking into account logistical issues; (3) there may be a shift among key stakeholders (e.g., faculty, students, community partners) with some more comfortable than others with the inevitable changes that will be required; (4) service e-learning may require more or different resources from community partners, thus limiting the involvement of agencies incapable of meeting the technological needs of the course; (5) partnership opportunities may be strengthened by internet access given that students can locate potential placements; and (6) chenneville, t., toler, s., and gaskin-butler, v. t. journal of the scholarship of teaching and learning, vol. 12, no. 4, december 2012. josotl.indiana.edu 65 students may benefit from the convenience of electronic journaling. readers interested in learning more about opportunities to combine distance learning with civic engagement initiatives are referred to the section on distance learning in appendix a. e. overcoming challenges. clearly, incorporating service-learning activities into the undergraduate curriculum and, thus, promoting civic engagement among college students can be challenging for all of the reasons described above. however, many of these barriers can be overcome. creating a university climate that promotes citizen scholars will require action on the part of all stakeholders to include not only administrators, faculty, and students, but also community collaborators. both top down (i.e., administrative) and bottom up (i.e., students, faculty) initiatives are needed for civic engagement to be integrated successfully into higher education settings. civic engagement must be incorporated into the university’s mission before we can expect individual departments to embrace the idea of imbedding service-learning into the curriculum. this will require thoughtful discussions and collaboration between administrators, faculty, students, and community agencies. having a clear mission will set the stage for pursuing specific initiatives to create an environment where civic engagement is valued and practiced. creating such an environment may require the reorganization of university goals and priorities. because of the time demands associated with implementing service-learning activities, planning is needed to ensure that necessary resources are available and that stakeholders are not unduly burdened. organizational restructuring notwithstanding, there are some strategies that may ease the burden for individual faculty attempting to implement service-learning activities into their courses. for example, teaching assistants (ta) can help individual instructors manage activities related to service-learning/civic engagement. even when funds are not available to pay tas, many students are willing to volunteer as a ta for the experience it provides. iv. ethical issues. while morality and ethics are important across disciplines, professional ethics become a particular issue when incorporating service-learning/civic engagement activities into psychology courses. because service-learning activities in psychology courses are likely to expose students to vulnerable populations, ethical issues must be addressed. although a standard part of the curriculum in many graduate psychology programs, ethics is not a standard part of the curriculum in most undergraduate psychology programs. while ethics may be embedded into courses across the undergraduate curriculum, a course dedicated to ethics is not among the most frequently listed courses in undergraduate psychology curriculum (perlman & mccann, 1999). in this section, we will discuss some of the major ethical issues of concern when designing and implementing service-learning activities into psychology courses. while the topic areas correspond to some of the ethical standards described in the ethical principles and codes of conduct of the american psychological association (apa) (2002), it is important to understand that the apa does not accredit undergraduate psychology programs. clearly, undergraduate students majoring in psychology, even those who are student members of the apa, are not held to the same principles and standards expected of doctoral level psychologists. however, the principles and standards outlined in the apa ethical principles and code of conduct (2002) chenneville, t., toler, s., and gaskin-butler, v. t. journal of the scholarship of teaching and learning, vol. 12, no. 4, december 2012. josotl.indiana.edu 66 provide a framework for thinking about ethical issues that may arise when incorporating servicelearning in undergraduate psychology courses. a. competence. competence is difficult to define given that it is variable across time, task, and situation. however, if simplified, professional competence generally refers to one’s skills and abilities that affect performance and can be compared to a measurable standard. within the field of psychology, it is clearly understood that discipline-specific competencies vary across education and training level as well as specialty area. the expected competence of an undergraduate psychology major differs dramatically from the expected competence of a licensed psychologist or even psychology graduate student, with the standards much higher for the latter. however, as rosenthal, mcknight, and price (2001) demonstrate, confusion persists about the standard level of training for psychologists, even among undergraduate psychology students. therefore, it is the responsibility of faculty to educate both students and community partners about the appropriate role of undergraduate psychology students. for example, undergraduate students should never be sent out into the community to provide psychological services given that they lack the necessary training and experience. these limitations need to be clearly delineated to avoid potential confusion about what the student can and cannot do in a given placement. failure to do so could result in significant harm to the general public. b. privacy and confidentiality. confidentiality is important and privacy rights must be considered when designing servicelearning activities. for example, consider a child psychology course that places students in a local elementary school to volunteer (e.g., read to children) and requires students to keep track of their observations in a journal to be submitted to the instructor. students should be required to ensure the confidentiality not only of the children they observe and with whom they work, but also the teachers and other school personnel. while information shared between students and the individuals they serve through community placement is not considered privileged, respect for privacy should guide decisions about what to disclose and to whom. c. informed consent and assent. respect for individual autonomy also is an important issue when developing courses designed to promote civic engagement via service-learning activities. in the case of university-community partnerships, the obvious and primary concern is for the recipient of services provided through various community agencies (e.g., students in an elementary school, battered women in a domestic violence shelter). questions can arise with regard to informed consent or, in the case of children, informed assent. for example, should informed consent be obtained from parents whose children will be receiving classroom tutoring from a local university student? further, is assent from the children themselves necessary? assuming consensus can be reached about what situations warrant consent, which is a lofty goal in and of itself, then one must consider what information should be disclosed as part of the consent process. also relevant to this discussion is respect for the individual autonomy of students. specifically, what information should be provided to students enrolling for a course with a chenneville, t., toler, s., and gaskin-butler, v. t. journal of the scholarship of teaching and learning, vol. 12, no. 4, december 2012. josotl.indiana.edu 67 significant service-learning component or, further, to students majoring in psychology in a program whose mission dictates that civic engagement be fostered? handelsman and rosen (1987) describe the limitations associated with implying student consent based on course registration. that is, given variations in teaching philosophies and styles that, to a large degree, dictate assignments and grading, not to mention content, it may be unfair to imply consent based on course registration. handelsman and rosen (1987) remind educators that students’ decisions should be informed and, thus, adequate information should be provided prior to course registration. this is extremely important in the case of service-learning. service-learning activities often require a substantial amount of time outside of the classroom, which may be difficult or even impossible for some students. it, therefore, may be prudent to somehow notify students in advance about the requirements of the course so that an informed decision can be made. d. supervision. site supervision is critical for service-learning placement. ensuring that appropriate site supervision occurs will require that sites, themselves, be monitored as well as students’ progress at those sites. to this end, it is important to establish formal connections with site supervisors, when possible, to ensure that they understand the nature of the students’ assignment and the importance of their evaluations in determining students’ grades. e. research. research can be an integral part of service-learning initiatives designed to promote civic engagement. in fact, nigro and wortham (2006) encourage including action research as a part of service-learning activities. student research designed and conducted in collaboration with community partners can be extremely valuable, especially from the perspective of promoting civic engagement. however, the potential gains of research must be balanced against the risks and, even when the gains clearly outweigh the risks, precautions must be taken to ensure that risks that do exist are minimized. all of the ethical issues previously described apply to research activities. student researchers must be competent in the areas of research design and methodology and may require supervision to carry out research projects. appropriate actions must be taken to ensure the privacy rights and confidentiality of research participants. of particular concern when thinking about conducting research in community agencies via servicelearning placements is informed consent and assent. this issue becomes less problematic with “formal” research given that permission must be obtained from appropriate institutional review boards (irb) (e.g., university irb and hospital irb in the case of a service-learning placement in a hospital). more problematic is “informal” research. for example, consider the behavior modification course described in an earlier section. as described, students are placed in a local elementary school where they are required to apply learning principles when tutoring children and are required to track the children’s progress as part of a course assignment. essentially, the primary course assignment involves a single subject design. assuming only the instructor has access to the data collected, should informed consent/assent be obtained from children and their parents? what if others (e.g., classmates) have access to the data collection? or, what if a student wants to publish his/her findings? these are important questions that should be considered proactively. chenneville, t., toler, s., and gaskin-butler, v. t. journal of the scholarship of teaching and learning, vol. 12, no. 4, december 2012. josotl.indiana.edu 68 f. recommendations for addressing ethical issues. as demonstrated, many ethical issues arise within the context of promoting civic engagement among undergraduate psychology majors. and, although a discussion of liability is beyond the scope of this article, it is important to recognize that overlap exists between ethical standards and legal standards. policies and procedures should be established for responding to ethical issues as they arise. however, it is impossible to foresee what issues might arise in a given class or setting or for a particular student; and therefore, reactive approaches are limited. faculty, administration, and community partners are encouraged to be proactive in their approach to instilling the importance of professional ethics. one option is to incorporate a professional ethics course, specific to the field of psychology, into the undergraduate psychology curriculum and, further, to designate this course as a prerequisite to courses with a service-learning component. this option has significant limitations, though. among these limitations is the well documented fact that knowledge does not always translate into behavior. this means that even students who perform well in an ethics course may not demonstrate appropriate behavior outside the classroom. in his work on spirituality and psychotherapy, plante (2007, 2009) recommends a model of ethics that focuses on five central values: (1) respect, (2) responsibility, (3) integrity, (4) competence, and (5) concern. it seems this model fits well for all aspects of fostering civic engagement among undergraduate psychology majors from administrative policy to faculty course development to the establishment of community partnerships to the promotion of professional behavior among students. v. psychology course examples. service-learning, with the goal of promoting civic engagement among undergraduate students, can be incorporated into psychology courses. in this section, we will provide some examples of how civic engagement has been incorporated into psychology courses at a relatively small southeastern university with approximately 3,800 undergraduate students and 400 graduate students. the student to faculty ratio at this institution is 19:1. in the courses described, student enrollment averages anywhere between 30-50 students per semester. following the class examples is a discussion of how some of the ethical issues described above were addressed. a. child psychology. child psychology is an upper level course offered by psychology departments, either as part of the major or as a general education course available to all undergraduates. the course content generally includes an overview of cognitive, physical, emotional and psychological growth and development from birth through adolescence. typically, enrolled students have very little experience with children. most do not have children of their own. many do not have younger siblings, and the majority of students have limited exposure to children through employment or leisure activities. civic engagement has been incorporated into this course by requiring students to volunteer a minimum of two hours a week over the course of a semester in an elementary school or daycare setting. students are provided the opportunity to become classroom volunteers and observers of children across a wide range of developmental stages. throughout the semester, students are required to maintain a journal in which they reflect on specific developmental theories. this journaling exercise requires students to reconcile their personal observations and chenneville, t., toler, s., and gaskin-butler, v. t. journal of the scholarship of teaching and learning, vol. 12, no. 4, december 2012. josotl.indiana.edu 69 interactions with children with the theories introduced in the classroom. for example, journal topics focus on observations of the temperamental differences in children and reflections on the “goodness of fit” with their teacher to observations and reflections on vygotsky’s view of children’s use of private speech. due dates for journal entries are scheduled to coincide with lectures on specific topics. students are asked to discuss their experiences in the community, which often results in very lively and sometimes emotionally charged discussions. b. psychology of women. psychology of women is an upper level general education course often offered as an elective to undergraduate psychology majors. the course content includes providing a “challenging and thought provoking” (hyde, 2007, p. ix) stimulus for students to engage research related to the psychology of women. this course also enables students to understand women’s experiences resulting from biological and social/cultural factors. in addition, this course allows students to understand the psychology of women from a multicultural perspective. civic engagement has been incorporated into this course by requiring students to volunteer a minimum of ten hours over the course of the semester in a local community agency that provides services for women and/or girls. it is noteworthy that many sites do not provide services exclusively to this population; rather, many sites also provide services to men/boys (e.g., elementary school and middle schools, homeless shelters, women’s shelters, and women’s residences). students’ volunteer activities range in depth and breadth; some students merely observe the target population (i.e., the clients who receive services from a particular agency). however, others complete tasks that ultimately benefit the target population or, in some cases, provide direct service to the target population. students are required to have a site supervisor who verifies the completion of their volunteer hours and who evaluates their performance. during their volunteer experience, students are required to complete a journal in which they reflect on their experience. journal entries are completed after each site visit. at the conclusion of the volunteer experience, students are required to complete a research paper and conduct a presentation designed to integrate their experiences with recent psychological research on the psychology of women. students whose volunteer activities do not directly relate to the psychology of women are instructed to choose a research topic that relates to the agency’s role in providing services for women or girls. the ultimate goal of the assignment is to strengthen students’ knowledge of research in a particular area of interest and to critically engage the research in light of their civic engagement experience. students are assessed on how well they are able to integrate their knowledge of course material with their civic engagement activity. although students often report that completion of the volunteer experience, journal, and research paper are challenging, the majority of students recommend continuing to include civic engagement in future courses. students typically are challenged when their civic engagement experiences do not confirm stereotypes related to race, class, and gender. students often enter their settings with preconceived notions about the women or children served by particular agencies. however, by the end of their experience, most are able to understand that many of the circumstances in which women find themselves transcend race, class, and gender. as a result, students are more knowledgeable and more culturally competent at the conclusion of their civic engagement activities. chenneville, t., toler, s., and gaskin-butler, v. t. journal of the scholarship of teaching and learning, vol. 12, no. 4, december 2012. josotl.indiana.edu 70 c. behavior modification. behavior modification is an upper level course offered by psychology departments with prerequisites such as statistics and/or research design and methodology. this course provides an in depth introduction to behavioral analysis. it is designed to help students learn how to target and measure behavior, develop a behavioral intervention program, and implement a research design using the basic principles of behavior modification. students are responsible for completing a semester long behavior modification project as they volunteer in an underprivileged public school. each student is required to become certified as a “mentor” in the public school system because of the intensive involvement with elementary, middle or high school students. students enrolled in the course also are required to obtain parental consent for their work with one or more children. behavior modification students identify a target behavior that will enhance a child’s functioning in the school setting. they conduct a functional assessment of the behavior to understand environmental factors (antecedents and consequences) impacting the target behavior. a behavioral intervention is designed and implemented in the classroom, baseline and intervention data is collected and discussions focus on lessons learned over the course of developing and implementing the intervention. the final week of the semester culminates in written papers and oral presentations to fellow classmates regarding their behavioral interventions in the school setting. students require a great deal of oversight and encouragement as they attempt to apply behavioral principles to “real life” circumstances. faculty who attempt this level of service-learning need to be prepared to meet with student fear and resistance to what initially appears to be a daunting task. the leap from discussion of principles in the classroom to application in the schools requires a level of problem solving that most undergraduate students have not been required to face. as the students settle into their projects, active class discussion returns to understanding basic concepts, such as “reinforcement”, or “discriminative stimuli” that only an applied assignment can demand. as the semester nears an end, and projects are due, students generally experience the ultimate joy of accomplishing a truly challenging project. the best part of the semester is listening to students’ oral presentations, as they describe the successes of their own students, as well as challenges faced while working in the classroom setting. d. our approach to dealing with ethical issues. due to the ethical concerns associated with incorporating service-learning into undergraduate psychology courses, faculty within our department agreed to implement a requirement that students complete a course in professional ethics in psychology prior to enrolling in a course involving a service-learning component. the ethics course originally was designed for this purpose and specifically addresses issues related to competence, privacy and confidentiality, informed consent/assent, and research involving human participants. this approach, however, is not without problems. as is a problem with any prerequisite requirement, there is the issue of course availability and sequencing. the ethics course is taught only once a year, which can be problematic. we have dealt with this issue in a number of ways. in some cases, we have allowed students to enroll in the ethics course as a co-requisite (simultaneous with the service-learning course) as opposed to a pre-requisite. in other cases, the faculty member teaching the servicelearning course has incorporated ethics into the course curriculum/syllabus. for example, one semester, the faculty member who teaches the ethics course guest lectured in one of the servicelearning courses. while this approach addresses the need to expose students to professional chenneville, t., toler, s., and gaskin-butler, v. t. journal of the scholarship of teaching and learning, vol. 12, no. 4, december 2012. josotl.indiana.edu 71 ethics, we know that knowledge of ethical standards and principles does not always translate into ethical behavior. in addition to formal instruction on ethics, faculty members incorporating servicelearning into their courses do a number of things in an attempt to be proactive with regard to ethical behavior. first, the importance of working within one’s competence is stressed to students. for example, when volunteering at a community agency, students are instructed to behave strictly within the roles defined as part of the service-learning requirement and not to act like they know more than they do. second, we routinely address issues related to privacy and confidentiality by instructing students not to disclose identifying information about community members in class or in their coursework. for example, if there is a journal or other written course requirement, students are told to use first names only, initials, or pseudonyms to protect the identity of community members. third, issues of consent/assent are dealt with individually depending on the setting and the requirement. for example, in the child psychology course described above where students are required to volunteer in an elementary school setting, the school shares responsibility for consent/assent. that said, students are instructed to disclose their role to all concerned and to respect individual autonomy, addressing issues related to both competence and consent/assent. finally, students required to conduct research as part of a service-learning requirement must obtain approval through the university’s institutional review board (irb) and, where necessary, external irbs (e.g., k-12 school system irb, hospital irb). in closing, readers should recognize that the examples described in this section provide but a few of the possibilities that exist for incorporating service-learning into undergraduate psychology courses and, thus, promoting civic engagement among undergraduate psychology majors. for more examples, readers are referred to bringle and duffy (2006) who provide additional illustrations of how to integrate service-learning into psychology courses. vi. conclusion. in this article, we have attempted to demonstrate the utility of promoting civic engagement among undergraduate psychology majors via the inclusion of service-learning activities into relevant courses. as emphasized in this article, significant barriers exist, which include lack of administrative support, time constraints among faculty, and student resistance or complete lack of interest among students to engage in service-learning. however, with thoughtful planning, dedication and a commitment to the ideals of civic engagement, these challenges can be overcome. and, while unique ethical issues must be considered when incorporating servicelearning activities into psychology courses, professional and ethical behavior among students, faculty, and community collaborators can be encouraged if stakeholders are willing to be proactive. readers interested in learning more about this topic are encouraged to pursue the additional resources contained in appendix a. chenneville, t., toler, s., and gaskin-butler, v. t. journal of the scholarship of teaching and learning, vol. 12, no. 4, december 2012. josotl.indiana.edu 72 appendix a. civic engagement and service learning resources civic engagement and service learning journals: *journal for civic commitment *journal for civic engagement *michigan journal of community service-learning civic engagement and service learning websites: *international partnership for service learning: http://www.ipsl.org/ *national service-learning clearinghouse: http://www.servicelearning.org/ distance learning and civic engagement/service-learning resources: *bennett, g., & green, f.p. (2001). promoting service learning via online instruction. college student journal, 35(4), 491-497. *dailey-hebert, a., donnelli-sallee, e., & dipadvoa-stocks, l. (2008). service elearning: educating for citizenship. information age publishing: charlotte, nc. diversity and civic engagement/service-learning resources: *everett, k.d. (1998). understanding social inequality through service learning. teaching sociology, 26(4), 299-309. *marullo, s. (1998). brining home diversity: a service-learning approach to teaching race and ethnic relations. teaching sociology, 26(4), 259-275. other resources: *101 ideas for combining service & learning (separated by subject area). available at: http://www.ahead.org/uploads/2008/handouts/block8/8.2%20-%20handout%202%20%20101%20ideas%20for%20sl%20by%20subject%20area.pdf *howard, j. (2001). service-learning course design workbook. university of michigan: ann arbor. *service-learning in psychology: a resource sheet. available at: http://www.denison.edu/campuslife/servicelearning/psychology%20resource%20sheet%202009. pdf chenneville, t., toler, s., and gaskin-butler, v. t. journal of the scholarship of teaching and learning, vol. 12, no. 4, december 2012. josotl.indiana.edu 73 references american psychological association. (n.d.). civic engagement. retrieved from http://www.apa.org/education/undergrad/civic-engagement.aspx. american psychological association education directorate (2009). civic engagement and service-learning. washington, dc: american psychological association. retrieved august 10, 2009, from http://www.apa.org/ed/slce/civicengagement.html american psychological association. (2007). apa guidelines for the undergraduate psychology major. washington, dc: apa board of educational affairs task force on psychology major competencies. retrieved from www.apa.org/ed/resources.html american psychological association (2002). ethical principles and code of conduct. retrieved april 30, 2009 from http://www.apa.org/ethics/code2002.html. barrow, f.h. (2008). who is in charge of this: service-learning as a context for youth empowerment through policy participation in after-school program. the journal for civic commitment, 11. retrieved february 18, 2009, from http://www.mc.maricopa.edu/other/engagement/journal/issue11/barrow.s html. bella, r.n., madsen, r., sullivan, w., swidler, a, & tipton, s.m. (1985). habits of the heart: individualism and commitment in american life. berkeley: university of california press. bringle, r.g., & duffy, d.k. (2006). with service in mind: concepts and models for servicelearning in psychology. sterling, va: stylus publishing. bringle, r.g., & hatcher, j.a. (1996). implementing service learning in higher education. the journal of higher education, 67(2), 221-239. bulot, j.j., & johnson, c.j. (2006). rewards and costs of faculty involvement in intergenerational service-learning. educational gerontology, 32, 633-645. fenzel, l.m. (2001, april). enhancing the benefits of service-learning in undergraduate psychology courses. paper presented at the annual meeting of the american educational research association, seattle, wa. fish, s. (2003, may 16). aim low. chronicles of higher education, p. c5. goodman, r.d., & west-olatunji, c. a. (2009). engaging students in outreach as an extension of advocacy to achieve praxis. the journal for civic commitment, 12. retrieved february 18, 2009, from http://www.mc.maricopa.edu/other/engagement/journal/issue12/goodman.html. grier-reed, t., detzner, d., poch, r.k., & staats, s. (2010). toward multicultural community engagement. academic exchange quarterly, 14(3). retrieved from http://rapidintellect.com/aeqweb/cho4671.htm chenneville, t., toler, s., and gaskin-butler, v. t. journal of the scholarship of teaching and learning, vol. 12, no. 4, december 2012. josotl.indiana.edu 74 grimes, t.r. (1998, august). creating opportunities for service-learning through the applied psychology course. paper presented at the annual convention of the american psychological association, san francisco, ca. handelsman, m.m., & rosen, j.a.a. (1987). informed consent of students: how much information is enough? teaching of psychology, 14 (2), 107-109. hurtado, s. (2007). linking diversity with the educational and civic missions of higher education. the review of higher education, 30(2), 185-196. hyde, j.s. (2007). half the human experience: the psychology of women (7th ed.). boston: houghton mifflin. james, w. (1910, august). the moral equivalent of war. mcclure’s magazine, pp. 463-468. kenworthy-u’ren, a. (2008). creating paths forward for service elearning. in a. dailey-hebert, e. donnelli-sallee, & l.n. dipadvoa-stocks (eds), service-elearning: educating for citizenship, (pp. 9-22). charlotte, nc: information age publishing. kretchmar, m.d. (2001). service-learning in a general psychology class: description, preliminary evaluation, and recommendations. teaching of psychology, 28(1), 5-10. laird, t. f. n., engberg, m. e., hurtado, s. (2005). modeling accentuation effects: enrolling in a diversity course and the importance of social action engagement. the journal of higher education, 76(4), 448-476. laird, t. f. n., engberg, m. e., & hurtado, s. (2002). modeling the effects of a diversity course on students’ preparation for a diverse democracy. paper presented at the annual meeting for the study of higher education. sacramento, ca. long, a.b., larsen, p., hussey, l., & travis, s.s. (2001). organizing, managing, and evaluating service-learning projects. educational gerontology, 27, 3-21. mcdonald, t.w., caso, r., & fugit, d. (2005). teaching and learning operant principles in animal shelters: perspectives from faculty, students, and shelter staff. journal of instructional psychology, 33(4), 310-321. mio, j.s., barker, l.a., & tumambing, j. (2009). multicultural psychology: understanding our diverse communities (2nd ed.). new york: mcgraw-hill. nigro, g., & wortham, s. (2006). service-learning through action-research partnerships. in r.g. bringle & d.k. duffy (eds), with service in mind: concepts and models for service-learning in psychology, (pp. 161-170). sterling, va: stylus publishing. osborne, r.e., & renick, o. (2006). service-learning. in w. buskist & s.f. davis (eds), handbook of the teaching of psychology, (pp. 137-141). malden, ma: blackwell publishing. chenneville, t., toler, s., and gaskin-butler, v. t. journal of the scholarship of teaching and learning, vol. 12, no. 4, december 2012. josotl.indiana.edu 75 plante, t.g. (2009). five ethical values to guide professional behavior. in t.g. plante, spiritual practices and psychotherapy: thirteen tools for enhancing psychological health, (pp. 103-114). washington, dc: american psychological association. prentice, m. (2007). service-learning and civic engagement. academic questions, 20 (2), 135145. rocheleau, j. (2004). theoretical roots of service-learning: progressive education and the development of citizenship. in b.w. speck and s.l. hoppe (eds.), service-learning: history, theory, and issues (pp. 3-21). westport, ct: praeger publishers. rosenthal, g.t., mcknight, r.r., & price, a.w. (2001). who, what, how, and where the typical psychologist is…the profession of psychology scale. journal of instructional psychology, 28 (4), 220-224. ward, k. (1996). service-learning and student volunteerism: reflections on institutional commitment. michigan journal of community service-learning, 3, 55-65. microsoft word v7n2smith journal of the scholarship of teaching and learning, vol. 7, no. 2, october 2007, pp. 49 – 61. assessment of student learning about native american cultures in a team coordinated interdisciplinary freshmen course julie m. smith, greg jacob, and toeutu faaleava1 abstract: the purpose of this project was to examine whether students in three sections of a team coordinated interdisciplinary course received the same educational experience. an essay covering three aspects of native american history was evaluated for content and critical thinking. significant differences were seen between classes in describing cultural differences that lead to conflict between native americans and euro american settlers. additionally, regardless of instructor approach, many students tended to maintain common stereotypical views of native american cultures. i. introduction. in 1993, portland state university revised its undergraduate requirements to reflect a more integrated and holistic approach to general education. central to this new university studies program (unst) is a yearlong interdisciplinary course or theme that students complete in their freshmen year. themes are team-developed by three to five professors from diverse disciplines who together create common process and content-based learning objectives. though the content varies between themes, each theme teaches to the four overarching goals of the unst program: written and oral communication, the variety of human experience, ethics and social responsibility, and inquiry and critical thinking. students that complete a full year of this freshmen course receive credit for having taken a science, social science and english course. faculty models for theme development and delivery range across a continuum (davis, 1995) from a traditional approach to a more open format. the traditional model of team teaching is "a type of instructional organization, involving teaching personnel and the students assigned to them, in which two or more teachers are given responsibility, working together, for all or a significant part of the instruction of the same group of students" (shaplin and olds, 1964). in these courses, team members develop a common syllabus and use the same course activities and assignments. each team member presents a portion of the curriculum in each class. among the benefits of this format are that students experience different perspectives because all faculty are involved in curriculum development and presentation. the authors of this paper are instructors in the columbia river basin (crb) theme. our theme development model is best described as a “collaborative model using team coordination” (mcdaniel and colarulli, 1997, p.21) because, though we teach to common content learning objectives, team members do not share a common syllabus or present in each other’s courses more than one or two days per term. in order to provide cohesion between crb classes, we meet bi-monthly to exchange ideas and share readings and classroom activities. as a result, we often use similar activities, but with different classroom approaches. table 1 provides background information for each crb instructor and table 2 compares pedagogic approaches to teaching about native american (na) cultures. as seen in table 2, all three faculty employ classroom 1 center for science education, portland state university, p.o. box 751, portland, or 97207, smithj@pdx.edu; department of english, jacobg@pdx.edu; director for the mcnair scholar’s program, faaleava@pdx.edu. smith, j.m., jacob, g., faaleava, t. journal of the scholarship of teaching and learning, vol. 7, no. 2, october 2007. 50 discussion, but use different methods to engage students. this is also true for the major assignments. the benefit to this model is that the crb theme attracts faculty who can be discouraged by the restrictions of the traditional model (shaplin and olds, 1964), because it can “maintain more faculty autonomy, afford more individual pedagogical styles and require less interaction with faculty colleagues” (mcdaniel and colarulli, 1997). mcdaniel and colarulli also point out that this model can lead “to less curricular integration for students” (p. 21). however, in all unst themes instructors are assigned the same cohort of students for a full year. therefore students benefit by having a consistent faculty presence throughout the year who can draw connections between the many topics covered. nevertheless, because we do each use somewhat different teaching strategies, a question that periodically arises is whether students’ learning experiences are consistent between our classes. we were particularly curious about how the students construct meaning in light of perry’s (1970) model of cognitive development. though perry provides a more expansive continuum of his cognitive model, his work is often classified into the three broad categories of dualism, multiplicity and relativism (battaglini and schenkat, 1987). while dualists see the world in terms of either/ors, right or wrong, good or bad, multiplists acknowledge the existence of multiple perspectives, but are reluctant or unable to give weight to one over the other. students categorized as relativists “can internalize multiple points of view, reflect on them, and construct them into one’s own theory about oneself and one’s experience” (haynes, 2002). certainly in our approach we did not want to see student writing reflecting superficial knowledge and the illusion of learning. curiosity prompted us to design a post assessment that examined student understanding of native american history in the columbia river basin across the theme. we chose native american (na) history for a number of reasons. first, nas in the crb continue to struggle for treaty, gaming and education rights, so understanding the foundation of these issues is imperative. second, we each use na history and culture to address the unst goal of encouraging students to “appreciate the diversity of the human experience.” third on an anecdotal basis, we also have seen a tendency for entering students to see pre-contact na cultures as somewhat homogenous. this societal stereotype of a common na culture is one that has persisted over time (hischfelder, 1982). the paper explores the results of our assessment. table 1. background information for crb faculty. instructor years on crb team discipline gender 1 3 ethnic studies male 2 7 literature and rhetoric male 3 8 environmental science female ii. methods. a. assessment instrument. the assessment was administered at the start of the spring 2004 term, because by this time much of the na curriculum had been completed in all three classes. the crb learning objectives directed towards nas are: smith, j.m., jacob, g., faaleava, t. journal of the scholarship of teaching and learning, vol. 7, no. 2, october 2007. 51 1. describe native american lifestyle and history prior to contact with europeans in the region; 2. analyze the complex interactions between native americans and non-native peoples in the region after contact (circa 1775). table 2. instructor’s pedagogic approaches. activities instructor 1 instructor 2 instructor 3 readings richard white’s the organic machine; robert clark’s river of the west; plus readings from texts about other cultures robert clark’s river of the west robert clark’s river of the west; course reader: including readings by vine delorea and alexi sherman major assignments group project: investigation of student selected na cultural unit pre-contact; 1015 minute presentation to class in a multimedia format. individual: two-page reflective essay on student experience with the group project group project: investigation of student selected na cultural unit pre-contact; 20 minute presentation to class group project: investigation of student selected na cultural unit pre-contact; 20 minute presentation to class and annotated bibliography of sources individual: 10 page term paper on current na issue classroom activities student-led discussion on the readings where students pose key questions. smaller groups discuss the readings then share insights and critical analyses among the larger group. instructor contextualizes and summarizes the discussion. class discussion of readings 15-20 minute presentation by instructor and/or ta of content material followed by small group and then class discussion of readings outside activities students encouraged to attend na cultural activities or special events on campus for extra credit. students were encouraged to attend na cultural activities or special events on campus. students encouraged to attend na cultural activities or special events on campus. visited campus na cultural center. students were asked to respond to the following open-ended questions: q1: describe the cultural and environmental lifestyles of a native american tribe or nation living in the columbia river basin prior to contact. you do not have to name a specific tribe, but you might want to indicate if it is a river, coastal or plains culture. q2: how did the differences between native american and euro-american (ea) cultures lead to conflict after contact? (select two differences to discuss) q3: what outcomes do we see as a result of these conflicts? students were also asked to indicate, “what activity was most pertinent to your learning about native american cultures in this class?” and “are there other activities or information that smith, j.m., jacob, g., faaleava, t. journal of the scholarship of teaching and learning, vol. 7, no. 2, october 2007. 52 would be helpful to your learning in this area? please explain.” in addition, we collected demographic information on age, major, gender and ethnic background. the assessment was given in a computer lab so that, if students chose to type their answers, computers would be available. the goal was for each student to have 50 minutes with which to answer the questions. b. scoring assessment questions two scoring rubrics were created for each question, one that examined content understanding (con) and the other critical thinking (ct). we used perry’s categories of cognitive development, in part, to guide the scoring for critical thinking. content rubrics scored students ability to provide two specific pertinent examples. each rubric was based on a 5-point scale and we had six rubric categories: q1_con, q1_ct, q2_con, q2_ct, q3_con, q3_ct. an example rubric for question 2 is seen in figure 1. figure 1. content and critical thinking scoring rubrics for question two. q2_con: content 1 2 3 4 5 provides no clear differences between groups and/or does not provide appropriate background information vague or inappropriate or inaccurate to more specific selects two or more appropriate differences for discussion; provides appropriate background information q2_ct: critical thinking 1 2 3 4 5 provides no explanation of differences between groups vague or contains generalities dualistic multiplistic relativistic uses specific criteria to explain differences without generalities we chose not to categorize 2 as dualistic or 3 as multiplistic, because students might show both in an answer. the 1-5 point scale acted as a guide. figure 2 shows examples of student responses to question two and their scores to for critical thinking and content. sometimes it was challenging to place a numerical value on an answer. the response that received a 4 in table two is not too different from the 5, but what we observed was that as the scores decrease, student answers progressed from acknowledging a difference in the value each group placed on natural resources as a reason for conflict to ascribing characteristics of forcefulness and of being overbearing as the motivation for ea’s actions. c. inter-rater reliability. to avoid bias in scoring, three people independently scored each student’s paper. scorers included instructors two and three and an independent consultant. we calibrated the rubrics by scoring a subset of 10 papers, discussing the rubric’s effectiveness and fine-tuning our scale. smith, j.m., jacob, g., faaleava, t. journal of the scholarship of teaching and learning, vol. 7, no. 2, october 2007. 53 when scoring the post assessments, if there was a difference between scorers of more than two points, we discussed the differences and came to an agreement that would bring the scores into alignment. after all three scorers independently scored the post assessments we averaged the three scores after determining there was no significant difference between scorers. figure 2. student work samples for question. content and critical thinking = 5 the native americans of the crb viewed the land as their home, their way of life, and something they could not live without. they saw it as a part of them. eas, on the other hand, saw the land as a gold mine, so to speak. they saw the land for its natural resources and how those resources could bring wealth. this inherently lead to conflict with the native and eas. in order for the eas to profit from the land, they needed to relinquish it from the native americans. but the native americans did not always want to sell. then at times the land was taken by force from them. to my knowledge, most, if not all, wars between the native and eas were over land. unfortunately, the eas won most of the time and plundered the land once sacred to the native americans. content and critical thinking = 4 the native americans shared the land and they lived off the land that was provided. this included some sacred land, and land in which they would get their food. these aspects were really important to these people. the eas thought the land belonged to them. they were allowed to claim this land because no one had claimed it prior. this was an obvious misconception and caused many battles over this territory. the land was used by the settlers and they scared away a lot of the food that some of the natives would hunt. this not only caused a shortage of food for the natives but didn’t allow them to roam free any longer. they had lost all of their land to people they had trusted and even today only have a small section to claim as their own. content and critical thinking = 3 because the differences of lifestyles that began to collide and force to interact caused an extreme amount of hostility and dislike for each other. the eas lived a more eurolistic civilized life, meaning that they staked a claim to the property and lived on it all seasons. they also settled the land by farming and cultivating the property. the eas also sought out the land as a form of profit or revenue by mining, timber producing and etc. so when eas moved westward and onto the frontier they took claims to the land that belonged to certain villages and tribes. in doing so, it drove the native americans to remote regions and it caused a great deal of negativity toward the eas. eventually the euro movement caused enough frustration and negativity that it brought them to war against the eas. content and critical thinking = 2 the white man came in to the crb and just started to take over. they had no respect for the natives and what they were doing. whites saw this area as their land and nothing else. the natives saw it as their land and used it to the best of their ability not to harm it. when the natives saw how disrespectful the whites were being they got mad. but the whites being the overbearing people that they were didn’t see that and just kept abusing the land the people that were there before them. content and critical thinking = 1 when the eas came over they believed that they were the first to settle this “new” land, but in fact the native americans were first there. this caused major territory problems, and the eas had a lot better access to weapons and technology which lead them trying to take over and forcing some native americans in to slavery. smith, j.m., jacob, g., faaleava, t. journal of the scholarship of teaching and learning, vol. 7, no. 2, october 2007. 54 d. coding. together we created categories to code for common content and critical thinking themes. for example, a common theme in question one that affected critical thinking scores was that pre-contact nas had a simple peaceful existence. we also coded demographic information and student responses to questions about class activities. e. potential biases and confounding factors. due to unforeseen circumstances, instructor three’s class was allotted only 30 minutes to complete the assessment. the following class period, students were given an extra 20 minutes to make any additional comments. seven students chose to write more. also, to motivate students to take the assessment seriously, instructors one and three attached a point value to the assessment, 3% and 3.5% respectively of the students’ overall course grade. all students who completed the assessment received full credit. instructor two assigned no point value to the assessment. instructor one’s background in ethnic studies could positively skew data for his class. additionally, instructor three’s teaching assistant was a third year native american studies major. iii. results. a. demographics. across the crb theme, 31 women and 43 men (n=74) completed the assessment. the gender make-up of each class can be seen in table 3. ages ranged from 18 to 38. the average age was 20, with the majority (52) being traditional first year students aged 18 and 19. sixty-one students self identified as caucasian, nine as asian and one each as native american, latino and hispanic. no student identified him or herself as african american. there were nine “english as a second language” (esl) students, including international students and first generation immigrants. esl students were evenly distributed between classes. table 3. gender by crb course. instructor females males total (n) 1 9 17 26 2 11 15 26 3 11 11 22 total 31 43 74 b. between class comparisons. our original goal for undertaking this project was to determine whether students had the same learning experience in each class. we did not perform a pretest, so we do not know what students knew about this topic before the class, however, we can examine what information they took away. we began by comparing the six individual content and critical thinking scores between classes using a one-way. we also performed a kruskal-wallis test on the grouped content scores smith, j.m., jacob, g., faaleava, t. journal of the scholarship of teaching and learning, vol. 7, no. 2, october 2007. 55 for all three questions, the critical thinking scores for all three questions, and the grouped scores for all six rubric scores. we found a significant difference between the three classes on q2_ct (p=0.042). otherwise there was no significant difference between classes. we grouped the data from all three classes to look for trends and influencing factors across the crb theme. we found that overall, students scored higher on q2 (p=0.051) than on q1 and q3. there were no differences between the grouped content questions and grouped critical thinking scores. age had no significant influence on students’ scores, but gender did. women tended to score higher than men on content q1_con (p=0.040), q2_con (p=0.035) and q2_ct (p=0.054, mannwhitney). c. question one content and critical thinking analysis. the most common content topic for all three classes related to natural resource usage. all classes showed an understanding of the interconnectedness between tribal survival and their appropriate management of available resources. interestingly, there were differences though in how each class used this topic to contextualize various aspects of na life. instructor one’s class often connected this topic to concepts of tribal mobility and the communal nature of land usage. plateau tribes living in areas of scarcer or seasonal resources tended to be more “purposefully” nomadic, while those living closer to the lower columbia river where resources were more stable tended to have stationary homes. instructor two’s class connected natural resource usage to fishing technology and instructor three’s class to tribal and family structure. student answers to this question were also often shaped by their independent research projects. a common theme between classes was the sacred nature of both land and salmon to nas. forty-three of the 79 responses contained references to salmon, including salmon’s connection to tribal survival, ritual, religion, story telling, myth and structure. respect for the environment was mentioned in 55 of the 74 responses. in terms of critical thinking, twenty-two of the 74 responses alluded to the cultures of pre-contact nas as being simple. the majority of these often contained a single sentence, such as, “before contact with outside cultures, native americans led simple and productive lives” (student 46), followed by a detailed and correct description of a complex cultural process such as natural resource management. these responses showed pronounced dualistic thinking. a few responses, though, like the one below showed an obvious misunderstanding of na cultures. before the euro americans settled in north america the native americans had a wonderful life, living off the land and keeping in touch with nature and the wildlife. they were always good to the land, their culture and made sure to give thanks to the animal gods. not to mention no real territorial problems have ever been recorded before the movement of the white man. they were all sentimental people who just had a great love for their tribes and their land. (student 19) the assumption of simplicity was the most common stereotype, but a few others were seen: nas worshipped salmon or were primitive or uncivilized compared to eas. for our paper, we chose the jussim et al (1989) definition of stereotype that “stereotypes constitute people’s beliefs about groups – beliefs that may be positive or negative, accurate or inaccurate” (p. 6). smith, j.m., jacob, g., faaleava, t. journal of the scholarship of teaching and learning, vol. 7, no. 2, october 2007. 56 d. question two content and critical thinking analysis. differences in how nas and eas viewed land ownership, natural resource management and religion were the three most common content topics in question two. there were little content differences between classes, except that instructor three’s class was more likely to list multiple differences as compared to the other instructors. most students scored well on the content portion of this question. by contrast, though, there were significant differences for critical thinking between classes on question two. the stereotypical views of nas as simplistic that arose in question one, were less evident in question two. however, eas did not fair so well. in our data analysis, derivatives of the words push (pushed, pushy, pushing), force (forced, forcefulness) and greed (greedy, greediness) were the most common attributes ascribed to eas. though students could list and describe the issues that lead to conflict, 41 students attributed ea’s motivation, in part or whole, to greed and aggressiveness. fifteen of instructor two’s students scored less than a three on this question. this was due to short undeveloped responses or the presence of multiple generalizations. e. question three content and critical thinking analysis. students in all three classes scored lower on question three than on the first two questions. though students could provide outcomes such as loss of tribal lands and cultural lifestyles (the two most common answers between classes), they could not expand on the implications for the tribes today. again, instructor three’s class provided more outcomes than the other instructors, but all students’ explanations were limited. content differences between the classes were that, in addition to the outcomes listed above, instructor one’s students listed reduced numbers of nas, instructor two’s listed treaty issues and instructor three’s listed resentment between nas and eas. the primary critical thinking issue that arose from question three was a tendency by the students to see the problems of nas as in the past: skirmishes with calvary and settlers, signing treaties, going to reservations, death from communicable diseases, loss of tribal hunting and gathering grounds. these are problems that do have implications for today, but these implications were not reflected in the student’s responses. though students could list outcomes of cultural conflicts, their explanations did not show confidence in their understanding of these issues. for example, the two most common answers were “loss of culture” and “loss of land.” these were indeed direct outcomes, but most students did expand on what this means to nas in the context of their current lifestyle. there was, also, a tendency in many papers to see outcomes as past events. f. other analyses. because salmon figured so prominently in the answer to question one (pre-contact), we coded the answers to question three (post contact) for how the subject of salmon was incorporated into their answers. only nine students mentioned salmon, most frequently in connection to resource management issues (5) and treaty conflicts (3). there were no significant differences between classes, gender, age, or esl status in terms of who would be more likely to have generalities about na or ea in their papers. smith, j.m., jacob, g., faaleava, t. journal of the scholarship of teaching and learning, vol. 7, no. 2, october 2007. 57 g. student feedback. a review of student comments on activities were most pertinent to their learning showed that instructor one’s and two’s students overwhelmingly chose projects, where they worked in groups to investigate a tribe and presented the information to the class. instructor three’s students were almost evenly split (7/5) between writing assignments and class discussions as most pertinent. interestingly, answers to the follow-up question about why the activity was pertinent were similar for students whether they chose group projects or writing. students reasoning included: enjoying independent research, selecting their own topic of personal interest to research, “digging deeper” into aspects of na cultures, discovering and learning a range of information, learning from each other while sharing the work and being actively involved in scholarship. the primary difference students gave for why group projects and writing were pertinent was that group projects allowed students to work collaboratively with their peers, while writing papers was an individual effort. across the classes, answers to what activities would improve their learning were similar. the most often suggested activities were guest speakers (19), followed by field trips (11), movies and documentaries (8), books and more readings (6), individual projects (3), mentor session (2), more time (2), discussion (2) and getting involved in na activities (2). iv. discussion. as we scored and coded the responses, themes began to emerge. over the years, we had seen a tendency for entering students to share a common ea stereotype of na cultural homogeneity prior to contact (hischfelder, 1982) and during the 2004 term, we sought to explicitly reduce this stereotype through independent research projects. students were responsible for sharing their research findings about different cultural groups with the class, thereby exposing all students to the diversity of columbia basin cultures. through their responses to question one, we felt that the students made gains in this area. however, we were surprised to see other stereotypes emerge, most notably the tendency to see na cultures as simple and peaceful. unfortunately, we do not know whether these were preconceptions students brought with them to class or if they were value judgments formed as a result of their independent research. the stereotype of na cultural simplicity is a common theme throughout ea history that hundorf (2001) describes as the postmodern individual’s desire to seek a less complicated existence. perhaps, in the midst of finals and end of term papers, na life did seem simpler to some students. compared to the two responses referring to na’s as uncivilized, the stereotype of simplicity and peacefulness were not seen as negative attributes by the students. we do know, though, that in the students’ more formal research assignments, we did not see these particular generalizations. this may be due to differences in how the formal assignments and the assessment were conducted. the formal assignments gave students more time to revise and reflect on what they wanted to communicate. in the timed exercise, students may have fallen back on stereotypes to expedite the explanation of what happened between the two groups. this may also be an example of what wegner (1994) suggests as a rebound effect. macrae et al (1994) demonstrated that when asked to suppress stereotypes students could do so. however, when this restriction was removed, stereotypes emerge, even to a greater extent than those who were not asked in prior writings to suppress their stereotypes. smith, j.m., jacob, g., faaleava, t. journal of the scholarship of teaching and learning, vol. 7, no. 2, october 2007. 58 the image of the nas changed in question two from simple and peaceful to the victims of ea greed and pushiness. in terms of content, students scored highest on this question. this was not surprising, as we each focus a lot of time on the period of contact and students may have had prior exposure to this information in high school. but, while their responses accurately reflected differences in land ownership and natural resource management, students tended to categorize ea’s motivations rather negatively. there was little discussion of the ea cultural and historical events that lead up to western settlement. this indicates that we need to revise how we teach the critical thinking aspect of this content area. the three classes also scored significantly different on question two critical thinking. instructor two, who had a lower score, replaced a term paper on post contact na culture with a paper on women of the west. additionally, instructor three, who scored higher, had several class discussions related to ea motivations for expansion. instructor three’s class had few generalizations about eas. instructor two, also, did not attach any point value to the assessment and this may have influenced the outcome on this question. it is interesting to note that many students’ families have lived in oregon for generations. most speak of this with pride and do not see their own great grand parents as greedy or pushy. this is perhaps another example of dualistic thinking. by the time students reached question three, overt stereotypes began to disappear. nas were on reservations, the rivers were polluted and the salmon were gone. student answers to question three were short with much less detail than their answers to questions one and two. this was the last question and students may have been tired, but most finished before the allotted 50 minutes. the quality of student responses probably reflects a weakness in the curriculum of all three classes. interestingly, though all three of us use treaty rights and salmon fishing to contextualize current issues, only 11 students’ papers touched on these topics. as stated previously, we coded for salmon in the first and last question. salmon was mentioned 43 times in response to question one but only nine in question three. in general, we attempted to bring these issues into the present, but our students didn’t make that jump with us. implicit in many students answers was the feeling that the outcomes of the conflict between these two groups was in the past, another common misconception (huhndorf, 2001). we were interested in the degree to which the students’ “internalized multiple points of view and arrived at knowledge as relative to and constructed within a context” (hayes, 2002). we had hoped the students would avoid excessive generalizations and dualistic thinking, and yet across all crb classes there were tendencies for students to make generalizations about both eas and nas. students scored lower on the critical thinking aspects of all three questions. even though each faculty attempted to dispel stereotypical perspectives of nas and eas, many students either maintained these attitudes or incorporated them into their new understandings in a dualistic manner, i.e. yes, nas had elaborate cultural structures regarding fishing, but their lives were simple. one cannot attribute this to the student’s age, because there was no corresponding decrease in stereotyping with increasing age. this may indicate a more pervasive societal perspective of nas. but, the body of research shows that breaking stereotypes can be challenging (schneider, 2004). one instructor experienced this first hand. he commented, what disturbs me is that even after a year of taking my class, three students used stereotypes to characterized native americans as “simple” and “peaceful.” two of the students were on a group whose presentation on the chinook included generalizations and errors that i had corrected in emails to them after their smith, j.m., jacob, g., faaleava, t. journal of the scholarship of teaching and learning, vol. 7, no. 2, october 2007. 59 presentation, however, one of the students tried to defend the error by pointing out that the information was on a “website” about the chinook. this experience suggests that i need to confront errors and stereotypes directly. so, how do we make progress in this area? as a team we need to better understand the preconceptions and stereotypes of nas that students bring to class and then present evidence that contest and challenge their preconceptions and stereotypes. constructivist research shows that when instructors do not uncover students’ preconceived ideas about a subject and discuss them openly, that even the best curriculum many not dispel these notions from the students’ minds (national research council, 2000). students will simply place the new information in context of their prior beliefs. in our case, many students could explain the complexities of na of life, but still saw their particular lifestyle as simple or idealistic. we must show how such preconceptions and stereotypes lead to unfair and unequal treatment of nas. hewstone (1989) explores the role of disconfirming information in changing stereotypes, where a “new instance of stereotype-discrepant information modifies the existing stereotype” (p. 208). based on an extensive literature review, he suggests that successfully using this technique be “(1) linked to typical outgroup exemplars; (2) presented to highly motivated perceivers; and (3) provided under conditions that do not induce intergroup anxiety” (p. 220). this technique has the potential for success, in that the students themselves suggested more visual, hands-on activities like speakers, movies and field trips, thus showing a willingness and interest to learn more. additionally, in a student-centered classroom, students may feel more at ease about hearing “disconfirming information.” na speakers and visits to reservations and other na spaces are effective ways of changing how students see and understand na. in short, we continue to educate our students about nas in ways that respect their complexity, diversity and humanity. but, we must also extend these educational practices to ea history, as students held an equal number of negative perspectives about eas. it is difficult to read early american history and not come away with some anger towards the injustices inflicted on nas. the assessment indicates a need to incorporate into the curriculum, as early as possible, a means of soliciting and discussing these ideas. in a practical sense, we found that class discussion over the assigned readings did not go far enough in breaking down stereotypes. we need to provide a structured framework to help students to contextualize the readings. these included giving students questions prior to reading, assigning summary statements, and exploring more deeply the significant difference between na and ea values toward land and life style. also, as a result of this project, another interesting trend appeared. in our study, gender appeared to be intervening factor on several content rubric scores. clinchy (1989) uses the term “connected knowing” to describe an observed phenomenon related to women’s ways of knowing. women are more likely to listen and think about others perspectives and compare and contrast them to their own thoughts before voicing their own ideas. it could be that the student centered pedagogic approaches used in the crb theme of class discussions and group work connect more with a woman’s way of knowing. feedback on what students enjoyed about the classes did indicate that class discussions and working with other students was pedagogically important to them. we found that independent research played a key role in all courses, whether instructors assigned it as an individual or group project. when responding to the question of why the assignment was pertinent to their learning, terms like “in-depth” and “delve deeper” appeared repeatedly in students’ answers. while other researchers (ruwe and leve, 2001) have found that smith, j.m., jacob, g., faaleava, t. journal of the scholarship of teaching and learning, vol. 7, no. 2, october 2007. 60 students disconnect in team-developed courses, we found that our students felt more connected to the course content and goals when offered the opportunity to explore a self-selected topic of interest about a particular na group. our original goal in undertaking this project was to determine whether our students were receiving the same learning experience within our three crb classes. our collaborative model of using team coordination for course development and delivery differs from the traditional model of team teaching. we found that unlike one study’s claim that team teaching rarely finds a middle ground (bartlett, 2002), our experience revealed that our classes might be more alike than even we anticipated. we learned through this process that though students may learn different content, they still leave the course with similar understandings. indeed, our classes may have found too much middle ground, based on the persistence of stereotyping in all three of our classes. we were, however, successful in dispelling the myth of a homogenous culture in the columbia basin prior to contact. overall, the assessment was invaluable to the improvement of our course designs and we encourage all faculty in team taught or collaboratively developed courses to undertake similar assessments. references battaglini, d. j. and schenkat, r. j. (1987). fostering cognitive development in college students – the perry and toulmin models. urbana, il: eric clearinghouse on reading and communication skills. (eric document reproduction service no. ed284272) bartlett, t. (2002). students become curricular guinea pigs. chronicle of higher education, 48(35), 1-7. clinchy, b. m. (1989). on critical thinking and connected knowing. liberal education, 75, 1419. davis, j. r. (1995). interdisciplinary courses and team teaching: new arrangements for learning. westport, ct: oryx press. eisen, m.j. and tisdell, e.j. (2000). team teaching and learning in adult education. san francisco: jossey bass. haynes. c. (ed). (2002). innovations in interdisciplinary teaching. westport, ct: oryx press. hewstone, m. (1989). changing stereotypes with disconfirming information. in d. bar-tal, c. f. graumann, a. w. kruglanski, and w. stroebe (eds.), stereotyping and prejudice: changing conceptions. new york: springer-verlag. hischfelder, a. b. (1982). american indian stereotypes in the world of children: a reader and bibliography. metuchen, nj: the scarecrow press, inc. huhndorf, s. m. (2001). going native, indians in the american cultural imagination. ithaca, ny: cornell university press. smith, j.m., jacob, g., faaleava, t. journal of the scholarship of teaching and learning, vol. 7, no. 2, october 2007. 61 jussim, l. j., mccauley, c. r., and lee, y-t. (eds.). (1989). stereotype accuracy: toward appreciating group differences. washington, d.c: american psychological association. macrae, c. n., bodenhausen, g. v., milne a. b., and jetten, j. (1994). out of mind but back in sight: stereotypes on the rebound. journal of personality and social psychology, 67, 808-817. mcdaniel, e. a. and colarulli, g. (1997). collaborative teaching in the face of productivity concerns: the dispersed team model. innovative higher education, 22, 19-36. national research council. (2000). how people learn: brain, mind, experience and school. washington, d.c.: national academy press. perry, w. g. jr. (1970). intellectual and ethical development in college years. new york: holt, rinehart and winston. ruwe, d. and leve, j. (2001). interdisciplinary course design. clearing house, 74(3) 117-118. schneider, j. d. (2004). the psychology of stereotyping. new york: the guilford press. shaplin, j.t. and olds, h.f., jr. (eds). (1964). team teaching. new york: harper and row, publishers. wegner, d. m. (1994). ironic processes of thought control. psychological review, 101, 34-52. 503 service temporarily unavailable 503 service temporarily unavailable nginx/1.14.1 503 service temporarily unavailable 503 service temporarily unavailable nginx/1.14.1 v4n1banker journal of scholarship of teaching and learning, vol. 4, no. 1, may, 2004, pp. 35 – 40. e-journals: reflections and communication improve learning outcomes teresa g. banker1 abstract. this report describes how journaling and email were used to benefit both the teacher and her students on a college campus in georgia. the report outlines how decisions were made about the content of the journals, submission deadline considerations, and concerns about the e-journal process. benefits to the teacher/professor are enumerated and discussed as are benefits to the students. ejournaling has been a successful experiment, which, seemingly, has improved the education process for these students and others as well. keywords: journaling, communication, student success i. introduction. journaling is not a new phenomenon; it is used very successfully in a plethora of situations. it comes as no surprise that the national council of teachers of mathematics (nctm) has long recommended journaling in the teaching and learning of mathematics (nctm, 1989, 2000). the communication (process) standard, in particular, recommends journaling for the following reasons: 1) journaling helps students organize and consolidate mathematical thinking through communication. 2) journaling helps students to communicate mathematical thinking coherently and clearly to peers, teachers, and others. 3) journaling helps students to use the language of mathematics to express mathematical ideas precisely. (nctm, 2000, p. 268) 4) journaling affords students opportunities for using new vocabulary or notation and for reflecting on their understanding of mathematics. mathematics is a precise language full of abstract symbols and notation that have no meaning without careful consideration of their purpose. thus, if students are to communicate mathematically and use mathematics in a productive way, they must find meaningful understandings for the symbols and notation associated with the language of mathematics. the act of communication contributes strongly to connecting intuitive ideas about mathematics to the abstract symbols and notation that constitute the language of mathematics. very often communication sharpens the understandings that students have about the language of mathematics and the mathematics itself. the understandings of the language of mathematics are two-fold. the first is the myriad symbols with their applications and the vocabulary that we use to stand for mathematical ideas and concepts. attaining familiarity with this aspect of the language is often daunting in itself. the second, of course, is understanding the concepts and bringing meaning to their applications. building clear understandings of mathematics is one of the goals of the mathematics classroom, and communication is a fundamental tool of the classroom where students are asked to think, 1 the author is at kennesaw state university, kennesaw, ga, and may be contacted at teresabanker@mindspring.com. an earlier version of this paper was presented at the georgia mathematics conference in october, 2003 banker, t. g. journal of scholarship of teaching and learning, vol. 4., n. 1, may, 2004. 36 reason, and “express results of their thinking orally and in writing” (nctm, 2000, p. 268). this type of communication can foster a classroom environment that encourages students to explore and refine their knowledge of the language of mathematics. such a classroom environment is desirable at all grade levels and is even beneficial in the college or university classroom. those teachers who provide opportunities for students to communicate about mathematics cultivate a supportive, non-threatening environment that can deepen the understandings that students need to make mathematics meaningful and to connect informal ideas about mathematics to the symbols and notation in the language of mathematics. ii. e – journal experiment. in the courses that i teach, which are freshman mathematics (e.g., mathematical modeling/college algebra) and the critical content for elementary education majors, the students bring with them an undercurrent of attitudes about mathematics that tend to hinder their success. students often come into these courses with very little confidence in their mathematical abilities. therefore, one of my instructional goals is to encourage better attitudes toward the study of mathematics and, consequently, improve student success, and the use of journals has contributed to meeting that goal. williams and wynne (2000) stated that journal writing allowed students to clarify their mathematical thinking by explaining their ideas about mathematics concepts. williams and wynne recommended a three-part approach to journaling: decide what the students write, decide when the students write and how long, and decide the writing format. williams and wynne worked with high school mathematics classes to develop their approach to journaling. to tailor their recommendations to the university venue, i chose to use email for the journals because it was convenient for both student and me because each of us could choose the time to complete the composition of or response to the e-journal. in principles and standards of school mathematics, nctm states that teachers can help students reflect on their learning by asking the students to “write commentaries on what they learned in a lesson or a series of lessons and what remains unclear to them.” (p. 272) this statement guided the formulation of what and when the students would write, the first and second recommendations of williams and wynne. i designed the e—journals to be in the form of reflections that addressed the following three questions: 1) what mathematics did i learn this week? 2) what was easy and why? 3) what was difficult and why? the second recommendation of williams and wynne was decide when the students write and how long. i decided that the activity could be accomplished outside of class, thus, not using valuable instruction time in the classroom, and students could choose a time to write the journals. because of the differences in student populations and class meeting schedules, i asked my students to send the email by midnight of the second day after the last class, (e.g. the mondaywednesday classes by midnight on friday). i set this time deadline because the university population is quite different from that of a high school. many of the students in my classes are non-traditional students who go directly to work and would be unable to submit an e-journal by 5 pm. also, i have found it beneficial for students to “mull over” the content of the week before writing their journals. this period before the deadline gives them time to work on homework and identify with what concepts or parts of concepts they are still struggling. the length of the reflections was to consist of fifty words or more. i decided that fifty words would be a minimum banker, t. g. journal of scholarship of teaching and learning, vol. 4., n. 1, may, 2004. 37 if students were truly reflecting on their learning. this approach seemed more reasonable than setting a time limit since the students choose the time that is convenient to them. lastly, williams and wynne recommended that the teacher decide the format of the journal. this aspect of their recommendations was a natural outgrowth of what the students would write, based on the questions for reflection that i designed for my students, with the additional stipulation that the writing use complete sentences. iii. benefits to teachers/professors. students who conscientiously reflect on their learning provide benefits not only to themselves but also to the teacher/professor. i have found that three basic benefits for the professor arise from student reflections. the first benefit is the correction of misconceptions. as students communicate what mathematics they have learned, many times the narrative they use will highlight a misunderstood mathematics term or concept. this misconception can be identified for the student and corrected immediately via reply email and, hopefully, get the student back on track. the timeframe that i use for replying to reflections is before the next class meeting. this timeframe is more likely to accomplish the desired outcomes (improved student learning and self-confidence and assisting the teacher/professor with planning). the second benefit that i have perceived is directly related to planning instruction of the course content. when several students in the same class report difficulties with the same concept, then i know that particular concept needs to be attacked from another direction to aid students’ understanding of the concept. this approach to planning instruction promotes a dynamic classroom based on student needs. finally, the third benefit has to do with student success. once students become accustomed to the e-journaling process and how they can be helped by speedy replies and suggestions for correct thinking, they are less reluctant to contact the teacher/professor with questions at other times not related to journaling. i am very diligent about checking email to answer student questions when a test/quiz/graded homework is scheduled for or is to be turned in the next class meeting. this “conversation” provides even more opportunities to help students deepen their understanding of mathematics and promote student success. iv. benefits to students. students (pseudonyms were used to identify each student quote.) involved in e-journaling report personal benefits that enhance the learning experience. identifying strengths and weaknesses and focusing study time were explicitly cited by amy and casey: …the reflections, what a powerful tool to make us “think” about what we’ve learned, tell [the professor] about it, and evaluate where we need to go… (amy, in-service middle grades teacher) i have benefited greatly from doing the weekly reflections…[they] make me think about the math that i did that week…[it] shows me where my weaknesses lie. it helps me see where i need to focus more study time…in order to succeed in math i must go to class, review my notes, and reflect to identify my weaknesses. i love the weekly reflections; it is a wonderful tool to help me with mathematics. (casey, early childhood major) banker, t. g. journal of scholarship of teaching and learning, vol. 4., n. 1, may, 2004. 38 a lack of confidence in mathematics abilities is very often an attribute the professor must deal with in certain student populations. lora, sarah, and matthew report that the reflections actually build confidence in their mathematics knowledge and abilities: by completing the reflections every week, i have found that i am better able to assess my math skills on my own…this knowledge gives me a better sense of confidence in my mathematics abilities. math has never been my best subject, so the reflections encourage me when i see that i really am learning the material… (lora, mathematical modeling student) the reflection website has been very helpful to me since spring semester…i need to be able to explain in english the concept that i learned instead of using numbers. this helps me put the information in simple terms to show that i understand it and can explain it to another person…my favorite question is the one where i explain what i’ve learned. it is so rewarding to me when i have grasped a concept and i can explain it in my reflection.(sarah, early childhood major) …this weekly procedure is hardly a sacrifice; it’s a medium for me not only to communicate my thoughts [to the professor], but also to communicate my thoughts with myself. math has always been a big scary, lonely place where numbers fly around in my head. by “talking” about these numbers and definitions, i’m able to grasp math as a whole much more easily. (matthew, mathematical modeling student) a third benefit that students nearly always report is the immediate feedback that is possible via email. they cite the “clearing up” of misconceptions, affirmation of correct thinking, and the “clarifying, supportive, and encouraging” communication between student and professor. [the professor] always responds to our reflections, often clearing up a misconception. i love the weekly reflections; it is a wonderful tool to help me with mathematics. (casey, early childhood major) i am able to let my professor know immediately when i realize that i do not understand something, and my questions are answered almost as soon as they arise. (lora, mathematical modeling student) when i put down the concept i am struggling with, the professor always replies with an example, suggestion, or a question that clarifies my confusion. she helps me understand where my misconception is and explains the concept…(sarah, early childhood major) in addition to clarification of thought in definition, the communication aspect between student and professor is heightened in this medium…i can be assured of a response from [the professor] either clarifying, supportive, or both! this pleases me not only because of the answers to questions but also the encouraging words and support, which i find invaluable. (matthew, mathematical modeling) banker, t. g. journal of scholarship of teaching and learning, vol. 4., n. 1, may, 2004. 39 finally, many students report that the reflections help them “see” the mathematics from a different perspective that allows for better understanding and contributes to their success as mathematics students. april says it very well: i am a firm believer that reflections help me better understand mathematics. to write a reflection, i must go through my notes and find key points and think about the importance of them and understand how they fit into the big picture. often times, i may not realize how things fit together and what is important until expressing these ideas via reflections. also, i have to remember specific points, reflect on them, and form them into my memory to express them as a reflection. the reflections keep the information fresh in my mind and help me better understand the concepts. (april, mathematical modeling student) v. concerns and cautions. with every strategy we devise in our endeavors to better educate students in mathematics or any other discipline, there are always concerns and tradeoffs that must be considered. implementing e-journals is not immune to the arena of concerns. i have currently identified three major concerns. the first concern involves the amount of time to respond to student e-journals. it is somewhat time consuming to compose and write thoughtful responses to student journals, and if a professor attempts to integrate e-journals for all students, this time factor may be prohibitive. the second concern involves dealing with computer/internet provider downtime. policies have to be thoughtfully considered in these circumstances because students very often do not have any control of when these circumstances occur nor can the professor actually verify that the problem really existed. the third concern involves the group of students who will abuse the opportunity in one of two ways that i have identified to date. the first abuse is the inevitable “gripe session” journal where the student constantly criticizes what has been done by the professor in teaching the concepts of the week, with no thoughtful suggestions to improve the situation. of course, the professor controls the privilege to delete the journal or withhold credit for the journal, but, unfortunately, one still has to listen to the “griping.” the second abuse is the student who just does as little as possible in the journaling process to qualify as a journal. in those instances, i have found that stated policies address the problem very well. in conclusion, e-journaling has been a successful experiment with my students. the process has proven beneficial to both the teacher and the students in several ways that, seemingly, has improved the education process for these students and others as well. ejournaling appears to disarm many misconceptions before they lead to more serious problems with other concepts, to provide opportunity for a dynamic classroom based on student needs, and to build students’ confidence in mathematics abilities that results in better student success. references national council of teachers of mathematics. (1989). curriculum and evaluation standards for school mathematics. reston, va: nctm. national council of teachers of mathematics. (2000). principles and standards for school mathematics. reston, va: nctm. banker, t. g. journal of scholarship of teaching and learning, vol. 4., n. 1, may, 2004. 40 williams, n. b. & wynn, b. (2000). journal writing in the mathematics classroom: a beginner’s approach. mathematics teacher, 93, 132 – 135. microsoft word knowltonfinal.rtf journal of scholarship of teaching and learning, vol. 6, no. 1, august, 2006, pp. 75 – 87. rapid prototyping as method for developing instructional strategies for supporting computer-mediated communication among university students dave s. knowlton1 abstract: because rapid prototyping results in the quick development of curriculum, materials, and processes, it is a form of design that could be particularly useful to professors in higher education. yet, literature documenting the use of rapid prototyping in higher education is scarce. this paper offers a case example of rapid prototyping being used as a design process. after presenting the case, the author points to necessary considerations for other faculty members who are considering using rapid prototyping. these considerations include the need to gain perspective on the roles of instructional strategies and computers within teaching and learning processes, understand the distinction between traditional research rigor and design rigor, and the importance of approaching design systematically. i. introduction and purpose designing meaningful learning experiences is difficult for professors. true instructional design often is too expensive of a process to be viable in higher education; and while carefullyconstructed constructivist learning environments are becoming more widely used across the academy, such environments, too, require a high level of detailed planning, particularly when computers are involved. college professors simply cannot enter each semester with a solid constructivist design of all assignments and course activities. sometimes, then, the best a professor can do is to design “something” as a part of a new course preparation and tweak it over time. in general, this process of designing and tweaking is referred to as rapid prototyping (reiser, 2001). while a professor’s efforts to use rapid prototyping can result in the quick development of instructional materials or activities (resier, 2001), the quality of resulting materials and activities often is suspect. why? models of rapid prototyping are surprisingly complex and are largely based on “progressive refinement”—“putting a first version of a design into the world” and then revising that design “until all the bugs are worked out” (collins, joseph, & bielaczyc, 2004, p. 18). the process is not complete, then, when materials have been developed. revising implies a detailed and systematic process; it is the iterative nature of designing that makes rapid prototyping a successful design approach (jones & richey, 2000). in fact, rapid prototyping often involves an entire support team to manage the design process (cf., lohr, javeri, mahoney, gall, li, & strongin, 2003), but most professors do not have access to such a level of human capital. the purpose of this paper is to offer a case example of the rapid prototyping process that i used to develop assignment guidelines for supporting students’ use of an online discussion board. notably, this paper emphasizes the systematic development of the assignment guidelines 1 assistant professor of instructional design & learning technologies at southern illinois university edwardsville, www.siue.edu/~dknowlt. knowlton, d.s. 76 across three semesters of implementation. perhaps this type of case example can be illustrative for other professors who need to systematically prototype assignments, sans formal training in rapid prototyping and support team. this paper begins with a description of the context in which the assignment was prototyped and then temporally describes the prototyping process. in the last section of this paper, i offer generalized principles for using rapid prototyping to develop assignments for the higher education classroom. ii. context in which rapid prototyping occurred part of my teaching responsibilities within a school of education at a midwestern university included serving as a member of a faculty team that supported the efforts of preservice teachers (undergraduate students majoring in elementary or secondary education) in a two-year, field-based teacher-certification program. the preservice teachers who were enrolled in this program were assigned to k-12 classrooms in partnership schools. because this was only the second implementation of the entirely field-based certification program, much of the context supporting the program was still developing. during the first semester of the two-year program, the preservice teachers often assumed a periphery role within the classroom—serving more as a teacher’s aide than as a practicing teacher. during the last semester of the two-year program, though, the student teachers participated in a formal “student-teaching” experience. the fieldbased program was designed to support the preservice teachers’ development from aide to professional teacher. throughout the two years, a team of university faculty supervised weekly content seminars. within the seminars, faculty members sometimes resorted to lecture as a means of orienting the preservice teachers to various educational theories and methods; more often, though, within these seminars, preservice teachers were given opportunities to discuss their experiences in the classroom. during each of the first three semesters of their field experience, the preservice teachers were enrolled for one credit hour of educational psychology—the content that i was responsible for overseeing. in principle, though, “courses” were non-existent. instead, each courses’ content was integrated into seminar activities and discussions. while the faculty team and preservice teachers came together for the weekly seminars, communication throughout the rest of the week was difficult. most of the professors on the faculty team had other responsibilities that prevented them from spending substantive time within the partnership schools, and the preservice teachers were placed in a variety of schools across three school districts. therefore, the team of faculty determined that since webct’s (the university’s approved online course management tool) discussion board depended on neither face-to-face communication nor real-time interactions, it would be a useful and efficient tool to help the preservice teachers stay connected with each other and with the faculty team. the rapid prototyping process that is the basis of this paper involves the design of strategies to support the effective use of bulletin board discussions. table 1 provides (a) an overview of the factors that influenced the development of each version of the discussion assignment, (b) the characteristics of each version, and (c) a summary of evaluation findings for each version. iii. the first-semester use of the electronic bulletin board the first-semester guidelines supporting the preservice teachers’ use of the bulletin board proved ineffectual as a tool for promoting communication, much less learning. within this knowlton, d.s. 77 table 1: factors contributing to assignment design, assignment characteristics, and evaluation. first semester version second semester version third semester version factors influencing design factors influencing redesign factors influencing redesign • need for flexible and efficient communication tool • emerging nature of the field experience • lack of information about the preservice teachers’ knowledge and skills • need to introduce basic educational psychology principles • first version was ineffectual • shifting responsibilities of preservice teachers • changes to the use of weekly seminar time • need for preservice teachers to gain skill in using web-based communication tools • evaluation of revised version • elimination of seminar time for educational psychology • continued shifting responsibilities of the preservice teachers initial design characteristics characteristics of redesign characteristics of redesign • laissez-faire • preservice teachers were simply made aware that discussion board existed. • preservice teachers assigned to two groups • discussion based on three-week cycles • discussion centered on student-initiated problems and proposals for practical solutions • addition of a privacy statement and job aid emphasizing conventions of cmc • additional direction to focus on “instructional problems”; more scaffolding to support “good” contributions • added reflection writing and self report form evaluation of first evaluation of second evaluation of third • ineffectual and rare use • preservice teachers reported that they didn’t see practical value of using cmc • problems were narrow in scope • interaction among the preservice teachers was limited • grading was cumbersome • perservice teachers noted workload was heavy and contrived • scaffolding of third week contributions did broaden the types of input from the preservice teachers knowlton, d.s. 78 section, the factors contributing to the first-semester guidelines and a description and evaluation of those guidelines are discussed. a. factors contributing to the first-semester guidelines commonly, careful analysis of both the educational context and learners precedes rapid prototyping (jones & richey, 2000). because the context of the partnership school was still emerging, analysis was based largely on generalization. from a macro-perspective, it seemed that the unique context of the field experience would continue to emerge as implementation progressed. this symbiosis between context and implementation required that i give the preservice teachers plenty of latitude in their use of the bulletin board, which might include their decision not to use it at all. furthermore, as the initial guidelines needed to be in place the day that i met the preservice teachers, i had no knowledge of the skill of the learner for which i was designing. had they used a bulletin board before? did they even know how to find the university’s webct site and log on? i did know that these preservice teachers had never before taken educational psychology. some content, then, needed to be transferred to these preservice teachers. in a pedagogical age of open-ended learning environments and within the context of a field experience, i recognize the vulgarity of suggesting the need for knowledge transfer. nevertheless, because of both the school of education’s accreditation process and certification tests that the preservice teachers would need to pass, the preservice teachers needed to obtain a basic understanding of educational psychology concepts and principles. this requirement further accentuated the need to de-emphasize the use of the bulletin board (which often is more wellsuited for promoting open exploration than for supporting direct concept attainment) and emphasize activities and assignments that were more likely to promote direct knowledge transfer. b. characteristics of the first-semester guidelines at the start of the first semester of the field-based program, the faculty team simply made the preservice teachers aware that webct had a discussion board where they could share ideas with each other and ask questions in a forum that would expedite communication. once the preservice teachers were aware of webct, i informally suggested that they might use the bulletin board to collaboratively make sense of assigned readings and prepare for seminar activities. admittedly, this laissez-faire approach contradicts much of the practical advice for using bulletin board discussions. some literature suggests that if professors do not scaffold the “hows,” “whens,” and “whys” of using asynchronous discussion then students will not use it effectively, or even at all (e.g., knowlton, knowlton, & davis, 2000). c. evaluation of the discussion board’s first-semester use predictably, the bulletin board was used rarely. when it was used, the contributions were most often in the form of close-ended questions: “what chapters are we supposed to have read by next week’s seminar?” several preservice teachers noted that it was nice to know the bulletin board was available, but they did not have a need to use it often. that is, they did not see how sharing ideas on the bulletin board would help them prepare for their day-to-day activities in the k-12 classrooms. after all, their argument went, they daily had access to their mentor teachers— knowlton, d.s. 79 the full-time teachers in the classroom to which each preservice teacher was assigned—who could guide them in their decision-making processes. iv. the second-semester use of the electronic bulletin board the laissez-faire approach to support learning through the bulletin board was not effective. to aim for more educational effectiveness, i shifted the emphasis toward a computermediated communication (cmc) assignment by designing instructional strategies that would more likely secure the preservice teachers’ participation. the formalized design showed some promise, but evaluation suggested the need for refinements to the assignment’s design. a. factors contributing to the second-semester guidelines both the preservice teachers’ “readiness” for a higher level of professional thinking and their shifting responsibilities in the classroom necessitated formalized guidelines to support the use of the computer-mediated discussion. during the first semester of the partnership program, i had assigned readings from the adopted educational psychology book (see eggen & kauchak, 1997). these readings served the purpose of introducing the preservice teachers to the large issues that fall within the domain of educational psychology. once the preservice teachers had been exposed to key educational psychology concepts, they needed experience applying those concepts by making connections between textbook theory and real-world classrooms. such connections can be useful in supporting students’ problem-solving efforts in field experiences (beckett & grant, 2003). this shift from “knowing” to “applying” seemed further appropriate because it paralleled the preservice teachers’ shift within the partnership school. the preservice teachers slowly were moving from serving as paraprofessionals—by taking class attendance and organizing materials, for example—to participating as true professionals—by designing lesson plans and teaching the entire class. a second contextual factor also created the need for more exact guidelines to support the use of the discussion board. the team of faculty members who supervised the weekly seminars decided that more organization was needed within the seminars. no longer would the faculty team collectively guide discussion and facilitate activities; rather seminar time was divided among content areas—“today is an educational psychology seminar; next week will be a reading methods seminar.” such a shift was problematic because it violated one of the very foundations of a field-based program—that content should be integrated and directly based on the preservice teachers’ field experiences (cf., beckett & grant, 2003; scanlon & ford, 1998; weber, 1996). successful professionals must learn to think holistically about their experiences, not about “courses” from a program of studies. designing and implementing more exact strategies to support cmc served as a means for prompting the preservice teachers to continue making integrated connections, even though seminar time was less integrated. b. characteristics of the second-semester guidelines participants were divided into two groups and the electronic discussion was based on three-week cycles of sharing and response. assessment of students’ efforts was integrated across the cycle. at the end of each cycle, roles were reversed so that preservice teachers in group one performed the responsibilities of the preservice teachers in group two and vice versa. this general approach has been supported elsewhere in the literature (cf., knowlton, 2002). knowlton, d.s. 80 week one of the discussion cycle. preservice teachers assigned to group one were responsible for describing a problem that they were experiencing within their partnership school. the assignment guidelines noted that the “problem might be interpersonal (e.g., a conflict with a mentor teacher or parent); instructional (e.g., students not meeting the objectives of a lesson); behavioral (e.g., a student who constantly ignores classroom rules); or contextual (e.g., a lack of adequate textbooks or other supplies).” this emphasis on a variety of problem types was purposeful. as i have noted, one goal of the assignment was to help the preservice teachers see the ways educational psychology was integrated with other “content,” such as cultural foundations of education, instructional methods, and classroom management. if the scope of the problems that students shared was broad, then opportunities for connections to content beyond educational psychology equally would be broad. week two of the discussion cycle. preservice teachers in group two were responsible for using the index and table of contents of the educational psychology textbook to theoretically frame the problems that had been shared during week one. the textbook, then, became a learning-on-demand resource, where preservice teachers were self-selecting readings that would most likely contribute to an analysis of the problem-at-hand. in addition to making connections between the problem and educational psychology, preservice teachers were encouraged to make connections to content areas that were the basis of their weekly seminars. these connections were designed to help the preservice teachers understand that neither the problems that they encountered nor educational psychology were discrete. instead relationships existed among problems encountered in classrooms, educational psychology, and other content areas. week three of the discussion cycle. all of the preservice teachers—regardless of whether they were assigned to group one or group two—were responsible for three contributions to the computer-mediated discussion. the assignment guidelines dictated that not all three contributions should be posted on the same day of the week. the purpose of this criterion was to build in reflection time for the preservice teachers. they were to consider the discussion in its entirety before adding further to the discussion. the assignment guidelines directed the preservice teachers’ efforts with week three contributions by noting that they should “further define and work to solve the problems-at-hand through dialogue.” they should “read what [their colleagues had written] within a ‘thread’ of discussion and interact by responding to [each other’s] ideas.” assessment across the discussion cycle. the assignment guidelines noted that the preservice teachers would “receive most credit for the number of contributions that [they offered].” the rest of the credit would be earned by meeting the stated purpose of each week’s contribution. for example, a stated purpose of week two and week three contributions was that the preservice teachers should theoretically frame the problem with citations. the assignment guidelines did offer a caveat to this loose assessment structure, however: “after the first two cycles, if we all feel that we are doing more than ‘going through the motions,’ then the assessment criteria can stay equally ‘loose.’ that is, we all should be working as professionals to help classmates solve real problems. if, however, i sense—or we agree as a class—that the criteria are not rigorous enough to foster collaborative problem solving, i will [offer] additional criteria (for example, specifying the length of contributions) to enhance the educational potential of this assignment.” knowlton, d.s. 81 c. evaluation of the second-semester design two data sources served as a basis for evaluation. first, my assessment of the preservice teachers’ efforts served as a basis for determining additional changes that were needed to improve the efficacy of the assignment. second, during a weekly seminar, i engaged the preservice teachers in a discussion about the use of cmc. assessment as evaluation data. as i assessed the preservice teachers’ participation in the computer-mediated discussion, i made judgments about the design of the assignment itself. this approach of combining assessment with evaluation to determine the educational viability of cmc is not unprecedented. in fact, “only the integration of assessment [with] evaluation can produce a clear picture of an online discussion’s educational viability” (knowlton, 2001, p. 164). numerous observations suggested the need for additional change. first, i found that students were relatively successful at articulating problems that they were experiencing, but the problems were extremely narrow in scope. of the approximately ninety posted problems, most focused on discipline problems among the k-12 students. one or two of the posted problems focused on interpersonal conflicts, such as conflicts with their mentor teachers or a parent. two of the posted problems focused on instructional concerns. second, most contributions during week three of the discussion were replies to the original problem posted during week one. in other words, the preservice teachers were not discussing the problems by interacting; they merely continued to offer solutions to the original problem. in fact, it often was unclear as to whether or not the preservice teachers were reading the threads of discussion in which they were responding. while repetition of various ideas across contributions within the same thread of discussion was common, consensus building and synthesis of ideas were scarce. third, as i assessed the preservice teachers’ participation, i recognized the inordinate amount of time i was spending on grading, as opposed to engaging in activities that were related to assessment but more productive toward creating continued learning among students—such as reacting to their discussion contributions, highlighting common themes among their interactions, and offering contributions to the discussion as an authentic participant. certainly, it was within my purview to grade the preservice teachers’ efforts, but grading should not dominate the assessment process (bauer & anderson, 2000). input from preservice teachers. at the end of the second semester, i solicited input from the preservice teachers about the use of cmc. i used two questions as prompts to promote feedback from the preservice teachers. notably, these prompts emerged as i assessed the preservice teachers’ success during the second semester: • how could the assignment guidelines be adjusted to emphasize the value of collaborative analysis and inquiry in an attempt to solve real-world problems? • how might the assignment guidelines be structured to foster an environment where the preservice teachers “forget” that cmc is being used as an “assignment” that will be graded and instead remember the need to act professionally and help their colleagues, even if that means offering more input than the minimum expectations require? during this discussion, two points emerged. first, the preservice teachers noted that many of them still usually were engaged in activities that did not directly relate to teaching. certainly, they felt that by the third semester of their partnership experiences they would have shifted even more into a key role as the “teacher” of the class. this shift, they argued, would make it easier knowlton, d.s. 82 for them to participate in the discussion because they would have richer experiences on which to base their contributions to the discussion. second, the preservice teachers noted that criteria governing week two and week three contributions were hindering their participation. they urged me, for example, to reduce the number of required contributions in both weeks two and three. this would give them more time to research and find appropriate resources to support the perspectives that they offered within the discussion. one preservice teacher noted that there were so many contributions to each thread that there was nothing left to add for late-comers to the discussion; reducing the number of required contributions would solve this problem. another preservice teacher agreed and noted that she did not read the threads before contributing because she did not want to know whether she was duplicating ideas that had already been offered. for similar reasons, students suggested the need to eliminate any criterion that specified on what days of the week they should participate. once they planned their contribution they returned to the discussion board only to find that someone else had offered their idea. also, several preservice teachers noted that they were printing out discussion contributions and sometimes even entire threads of discussion and reading them. so, while their actual contributions might come on a single day of the week, they were spending time considering the discussion across numerous days of the week. v. the third semester cmc assignment the third-semester version of the assignment included several changes from the previous semester. notably, these changes were based on feedback from the preservice teachers, which was reported in the previous section of this paper. in this section, i describe the milieu that contributed to the development of the third-semester assignment guidelines, the changes that were implemented, and evaluation. a. factors contributing to the third-semester design the feedback that i had solicited from the preservice teachers did contribute to the prototyping of the third-semester design, but other factors contributed, as well. notably, the format of the weekly seminars once again changed during the third semester of the partnership. it was determined that certain content areas—educational psychology being one such area—would not be given any formal emphasis during seminars. yet, i was still accountable for assessing the preservice teachers and giving an educational psychology grade to each of them at semester’s end. because of this dilemma between, on the one hand, needing to assess the preservice teachers and, on the other hand, not having formal seminar time to assess them, continuing to formally use cmc seemed appropriate. b. characteristics of the third-semester cmc guidelines the assignment was still based on the idea of the preservice teachers sharing real problems that they were experiencing and cycles of theoretically framing and solving those problems. three changes to the assignment guidelines were made in an effort to overcome some of the weaknesses evident in the second-semester version. the first was an administrative change. the second was a change in the types of problems that the preservice teachers should knowlton, d.s. 83 share. the third change concerned guidelines governing week two and week three discussion contributions. administrative change. during the second-semester version of the assignment, i had spent large amounts of time grading the preservice teachers’ contributions, as opposed to assessing and providing participants with the types of authentic feedback that could improve their problem-solving and analysis skills. to shift my own role within the discussion from a grader—which is often viewed as punitive—to a facilitator—which offers the opportunity to be more constructive—i developed a self-report form. at the end of each cycle of discussion, the preservice teachers completed and submitted this self-report, which allowed them to report factual information about their participation. for example, they could list the subject line of the threads in which they participated and cite the various resources that they used in theoretically framing a problem to which they responded. when the preservice teachers submitted their report, i had a list of threads in which i could find their contributions. this made the process of “grading” less time consuming. changes to the types of problems offered for discussion. the third-semester version of the assignment guidelines required that all problems contributed to week one of the discussion cycle must be “instructional problems”—as opposed to the type of behavior and discipline problems that dominated the second semester. specifying that the problems should be “instructional” in nature was designed to broaden the preservice teachers’ thinking regarding what constitutes a classroom problem that was worthy of analysis. changes to guidelines governing week two and week three contributions. i reduced the number of required contributions during week two of each cycle from three to two. the preservice teachers had advocated for the need to lower the number of required discussion contributions. they argued that lowering the quantity of required contributions would allow them to be more thorough in their analysis of the problems contributed to the discussion. while skeptical of such claims, i implemented this change in the hope that my skepticism would be unfounded. also, i specified that week three contributions had to be replies to week two contributions, not replies to the original problem discussed during week one of each cycle. i hoped that this criterion would improve interaction among the preservice teachers within the computer-mediated discussion and promote a deeper analysis of the issues embedded within the problems, not just continued (and often redundant) “solutions” to the original problem. because of this more specific purpose of week three contributions, i developed a list of possible strategies that the preservice teachers might use as they contributed to the discussion during week three. (see table 2.) c. evaluation of the third-semester prototype evaluation consisted mainly of the preservice teachers completing an open-ended survey about their views of using cmc. space limitations prevent a full explication of the survey results. i focus here, though, on feedback that directly related to changes made in prototyping the final version of the cmc assignment. changes governing the types of problems shared during week one. several preservice teachers noted that describing an instructional problem was more difficult than describing problems with student behavior or interpersonal conflicts. many of them acknowledged, though, that being asked to describe instructional problems forced them to look at their own curriculum development and instructional practices in a more detailed way. as one preservice teacher noted, knowlton, d.s. 84 table 2: strategies for replying to week three contributions as you write contributions to the discussing during week #3, you should work to add a deeper level of analysis to the discussion, not to simply repeat what has already been offered. if you are stuck for ways to contribute during week #3, consider the following possible strategies. • pick two replies to the same problem and discuss why you think one would work better than the other. • pick a reply to a problem and discuss the strengths and weaknesses of the proposed solution • pick a theory that someone mentioned as a help to understanding week #2 and apply that theory differently (or more thoroughly). • discuss your experiences with how a solution has/has not worked in the classroom. • write a summary of responses to your own problem and describe what the biggest things that you are taking away from your problem are. “i was surprised that it was more difficult to relate an instructional problem to ed[ucational] psych[ology]. the behavior problems stood out more and the connections were more obvious. because we had to share instructional problems, i think that i learned how interwoven ed[ucational] psych[ology] and teaching truly are.” changes to week two and three contributions. the preservice teachers on average tended to agree that the changes to week two and three contributions were positive. one preservice teacher noted that the changes to the guidelines allowed her to “actually relate different theories and sources of information to the problems [that] others [were] experiencing.” her point was that by being required to offer fewer contributions she could consider those contributions more carefully. other students seemed to indicate that the suggestions for week three responses were useful. many students noted that from these possible responses they realized that they could share their own experiences to a problem. one preservice teacher phrased it this way: “the most helpful responses were not the ones that said, ‘on page 276 of the text book, it states.…’ instead, responses that described what [others] were doing in their classrooms to help with similar problems were . . . much more helpful.” from this and several similar comments, i infer that the suggestions for week three responses (as shown in table 2) were useful to students in guiding them toward offering more salient contributions to the cmc discussion. vi. implications of this prototyping approach in this paper, i have offered a case example of rapid prototyping as a design approach for developing a cmc assignment for the higher education classroom. the details of such an example should provide faculty members with new perspectives about the iterative nature of development processes. specifically, several implications of this case cut across many higher education scenarios and are worthy of comment. knowlton, d.s. 85 a. instructional strategies influence learning noticeably absent from the case example is a discussion of the importance of customizing webct to improve the educational utility of the discussion. instead of focusing on the prototyping of improved media, this case focuses on the development of instructional strategies. such a focus is fully appropriate, as it is consistent with a view supported in the literature. namely, instructional strategies, not computers, are the cause of learning (e.g., clark, 1983, 1994a, 1994b). admittedly, such a perspective is not without detractors (e.g., kozma, 1991), but even these detractors agree that there is no credible evidence to suggest that computers influence learning. professors who are using rapid prototyping to design media-based assignments would do well to consider their own philosophy among media, instructional strategies, and positive learning outcomes. if, in fact, computers do not influence learning, then prototyping should focus on strategy development more so than on media development. b. consideration of design rigor for readers of this article who come from a traditional empirical background, this case example may have proved a frustrating read. no method of data collection and analysis was offered and applied, and no discussion of “significant” results was provided. but edelson (2002) distinguishes between traditional research and design research. with this distinction comes a distinction in approach. for the professor who is interested in achieving a level of understanding to justify change within a course or assignment, empirical rigor is not needed, and may even be misleading. instead, pedagogical rigor can provide insights sufficient to adjust assignments so that they promote a stronger opportunity for learning among students. the point is that faculty members across disciplines should gravitate toward design processes that allow for functional revision of assignments, even if such gravitation limits one’s ability to publish more scientific claims that are indicative of traditional research. c. nature of systematic design this third implication builds largely from the second. to suggest that empirical rigor indicative of the positivist research paradigm is unnecessary is not to suggest that design is haphazard and non-systematic. the case example noted here serves as a worthy model for professors across disciplines because it illustrates the relationship between the prototyping process and a dependence on inputs and outputs, which is one characteristic of “systematic” design. consideration was given, for example, to the macrocontext of the field-based program. consideration was also given to the changing needs of the preservice teachers. as professors adopt rapid prototyping procedures, they, too, should consider the role of context as a factor that influences and informs their design. as a learning context evolves, design practices must become increasingly iterative and flexible. furthermore, the evaluation of design is one unique stage of the design process that is particularly dependent on inputs and outputs. professors who are prototyping assignments across semesters or even within a single semester should plan for evaluating the quality of their own designs. as can be noted within the case described in this paper, the professor’s judgment was involved in evaluating the assignment, but the prototyping of the assignments did not stand on the professor’s judgment alone. student input was a part of the evaluation process and the assignment was prototyped—at least to some extent—based on that student input. within the knowlton, d.s. 86 case reported in this paper, perhaps i had an obvious advantage in that my students (i.e., the preservice teachers) remained the same across each semester of implementation. this allowed me to develop a rapport with them, and they perhaps felt more invested in assisting with the prototyping of the strategies, since they knew that they would be engaged in cmc discussion in future semesters. professors who do not have such an advantage might have to go to greater lengths to account for student input as they are prototyping assignments. the use of additional formal surveys, focus groups, or other opportunities for student input may be useful to this end. references bauer, john f., and anderson, rebecca s. (2000). evaluating students’ written performance in the online classroom. in renee e. weiss, dave s. knowlton, and bruce w. speck (eds.), principles of effective teaching in the online classroom (pp.65-72). san francisco: jossey-bass. beckett julia, and grant, nancy k. (2003). guiding students toward solutions in field experiences. in dave s. knowlton & david c. sharp (eds.), problem-based learning for the information age, (pp. 67-72). san francisco: jossey-bass. clark, richard e. (1983). “reconsidering research on learning from media.” review of educational research, vol. 53, no. 4, pp. 445-459. clark, richard e. (1994a). “media will never influence learning.” educational technology research and development, vol. 42, no. 2, pp. 21-29. clark, richard e. (1994b). “media and method.” educational technology research and development, vol. 42, no. 3, pp. 7-10. collins, allan, joseph, diana, and bielaczyc, katerine (2004). “design research: theoretical and methodological issues.” the journal of the learning sciences, vol. 13, no. 1, pp. 15-42. edelson, daniel c. (2002). “design research: what we learn when we engage in design.” the journal of the learning sciences, vol. 11, no. 1, pp. 105-121. eggen, paul, and kauchak, donald (1997). educational psychology, 5th ed. columbus, oh: prentice hall. jones, toni s. and richey, rita c. (2000). “rapid prototyping methodology in action: a developmental study.” educational technology research and development, vol. 48, no. 2, pp. 63-80. knowlton, dave s. (2001). “determining educational viability in online discussions: a student-centered approach.” academic exchange quarterly, vol. 5, no. 4, pp. 162-168. knowlton, dave s. (2002). “promoting liberal arts thinking through online discussion: a practical application and its theoretical basis.” educational technology & society, vol. 5, no. 3, pp. 189-194. knowlton, d.s. 87 knowlton, dave s., knowlton, heather m., and davis, camela. (2000). “the whys and hows of online discussions.” syllabus: new directions in educational technology, vol. 13, no. 10, pp. 54-58. kozma, robert b. (1991). “learning with media.” review of educational research, vol. 61, no. 2, pp. 179-211. lohr, linda, javeri, manisha, mahoney, chris, gall, jim, li, kathy, and strongin, dawn (2003). “using rapid application development to improve usability of a preservice teacher technology course.” educational technology research and development, vol. 51, no. 2, pp. 41-55. reiser, robert a. (2001). “a history of instructional design and technology: part 2: a history of instructional design.” educational technology research and development, vol. 49, no. 2, pp. 5767. scanlon, patricia a., and ford, michael p. (1998). grading student performance in real-world settings. in rebecca s. anderson and bruce w. speck (eds.). changing the way we grade student performance: classroom assessment and the new learning paradigm, (pp. 97-105). san francisco: jossey-bass. weber, alan (1996). “professional development schools and university laboratory schools: is there a difference?” the professional educator, vol. 18, no. 2, pp. 59-65. microsoft word wiersemafinal the journal of scholarship of teaching and learning, vol. 7, no. 1, may 2007, pp. 16-33. developing responsible learners: the power of intentional mental processing janice a. wiersema and barbara l. licklider1 abstract: most college students do not habitually do the deep thinking that will allow them to take their learning beyond the acquisition of knowledge. this paper examines the findings from a phenomenological study of eight students involved in an interdisciplinary community focused on developing responsible, life-long learners. students identified both the myriad ways the learning opportunities had been structured to engage them in thinking and the impacts of the resultant deep reflection on their learning. implications are that educators must do more asking than telling until students learn to ask and answer the challenging questions on their own. keywords: reflection, intentional mental processing, deep thinking. i. introduction. learning from experience is powerful for most individuals, but rarely will they “extract all the potential meaning that is implicit or move beyond their current meanings without being challenged” (caine and caine, 1997). one key to helping students use their experiences to engage in deeper learning is active processing, which, according to caine and caine (1994), is: …the consolidation and internalization of information, by the learner, in a way that is personally meaningful and conceptually coherent. it is the path to understanding, rather than simply to memory…the pervasive objective is to focus on the process of our learning and extract and articulate what has been explored and what it means. in effect, the learner asks in as many ways as possible “what did i do?”, “why did i do it?”, and “what did i learn?” (p. 156-157) (note: because of common misconceptions associated with the phrase “active learning,” we are using “intentional mental processing” in place of caine and caine’s phrase: “active processing.”) the more questions the individual asks and answers, the deeper the learning is likely to be as a result of the experience. as perkins points out in the article “learning as biological brain change” (leamnson, 2000), “learning is a consequence of thinking—it’s less the doing than the thinking, the reflecting on that doing that counts” (p. 37). in their book connecting leadership to the brain, dickman and stanford-blair (2002) refer to a similar kind of thinking within a discussion about reflective intelligence. if information patterns are the currency of intelligence, reflection is the compounding of returns on the original investments in their construction. that is, reflection is the ultimate stringing together of patterns of information through serious consideration—a conscious 1 department of information assurance center, iowa state university, 2413 coover, ames, ia 50011-3060, janw@iastate.edu and department of educational leadership and policy studies, blicklid@iastate.edu. wiersema, j.a., and licklider, b.l. the journal of scholarship of teaching and learning, vol. 7, no. 1, may 2007. 17 bending back—of constructed knowledge to proactively explore further configurations, implications, and applications thereof. in effect, the reflective qualities of your brain engage in examination of how that which is mentally constructed might best be invested– exploited might be a better word–to the advantage of survival interests. (p. 95) while a full discussion of the reflective nature of intelligence is beyond the scope of this paper, the basic notion of engaging students in meaningful reflection fits with the active processing suggested by caine and caine (1994). it further supports rephrasing that notion as intentional mental processing. in addition to the questions of “what did i do?,” “why did i do it?,” and “what did i learn?,” dickman and standford-blair probably would add “how did i do it?” and “what if… ?” indeed, from our own research we would add “what was/am i thinking?” and “why was/am i thinking that?” the challenge for educators is to help students learn to ask and answer these kinds of questions consistently for themselves for most experiences—to engage in intentional mental processing as a habit of mind. this study examined the experiences of students in a program that was designed specifically to get students to take responsibility for their own learning by developing intentional mental processing as a habit of mind. ii. background for the study. recently, we had the opportunity to combine our knowledge of human learning and the principles of learning organizations in a program for students at a land-grant university in the midwest. as part of the national science foundation (nsf) scholarship for service (sfs) program, students are awarded full scholarships in exchange for two years of federal government work following graduation. the nsf sfs program is an interdisciplinary effort involving students and faculty in computer engineering, computer science, mathematics, political science, management information systems, and education. fellowship recipients participate in a two-year leadership development program in addition to the requirements of their majors. the program is designed with an emphasis on: (a) learning about learning, (b) learning about self, (c) purposefully developing community, (d) deliberately practicing and refining skills to support and encourage the growth of self and others, (e) practicing metacognition, and (f) engaging in intentional mental processing. the knowledge, skills, and dispositions desired as outcomes of this program are certainly outside the experiences and comfort zones of these students’ previous educational encounters. by the end of the first year, it was clear students not only were taking responsibility for their own learning, but they also were developing into a productive community of learners. we needed to find out what contributed to these developments. iii. methodology. a. methods. the desire to understand, from the words of the participants, key factors that contributed to their learning and development made this an ideal phenomenological research study (colaizzi, 1978; moustakas, 1994). we experienced the phenomenon of learning in community with the participants and witnessed their growth and development into avid learners and worthy team members. therefore, we engaged in the phenomenological epoch process (moustakas) to wiersema, j.a., and licklider, b.l. the journal of scholarship of teaching and learning, vol. 7, no. 1, may 2007. 18 identify our own beliefs, biases, and assumptions and set them aside to focus on the lived experiences of the participants. b. data collection and analysis. although the eight participants in the study were in the fourth semester of the nsf sfs program, the research was designed to focus on the learning experiences during the first two semesters. consistent with the goal of phenomenological research to collect rich, meaningful information that accurately depict the participants’ interpretations of the phenomenon (merriam, 2002), data sources included a focus group interview comprised of all eight participants, individual interviews, journals, and written self-assessments. all interviews were audio taped and transcribed verbatim. data were analyzed and interpreted following the steps suggested by colaizzi (1978): read all data, extract significant statements, formulate meanings, organize into clusters of themes, integrate into an exhaustive description, and formulate the exhaustive description in as unequivocal a statement of identification of the phenomenon’s fundamental structure as possible. data were validated by returning the findings to the participants and asking for feedback. also known as member checking (merriam, 2002), this step provided no new data. iv. findings. as referenced earlier, the last stage in the analysis of the data is to organize the themes into an exhaustive description of the phenomenon and formulate the exhaustive description in as unequivocal a statement of identification of its fundamental structure as possible (colaizzi, 1978). in general, the themes that emerged were: self-identified growth and development, continuous reflection, metacognition, high expectations for addressing challenging tasks, interdependence, accountability, and supportive environment. therefore, the fundamental structure of the phenomenon of learning in community, as perceived by these eight participants, is a self-recognized transformative development resulting from engaging learners in intentional mental processing before, during, and after challenging them with, and holding them accountable for, addressing complex, meaningful tasks in an interdependent and supportive environment over time. this complex statement gives rise to a multitude of possibilities of exploration with implications for educators and students. this paper explores the impacts of intentional mental processing in this transformative experience. while it is not surprising that students would identify intentional mental processing as being key to their learning, what was surprising was the number of different ways they identified that we had structured the learning experiences so they had to think. we understand, as identified by leamnson (2000), that learning is a very private matter and happens only in the mind of the individual. this means external agents cannot actually cause learning, but can only influence the likelihood of it. therefore, faculty do have a responsibility to guide and direct the thinking and actions of students, but it is the individual learner who “must think deeply and repeatedly about something” (p. 37) and in multiple ways before learning occurs. although we are purposeful with every learning opportunity planned for students, revelations from the participants identified just how much our own habits of mind related to thinking causing learning guide almost all of our actions and behaviors related to students and their learning. in this paper we uncover, through the words of the participants, the kinds of expectations and experiences that contributed to the development of intentional mental processing as a habit of mind. in addition, we provide wiersema, j.a., and licklider, b.l. the journal of scholarship of teaching and learning, vol. 7, no. 1, may 2007. 19 insights that can be used by other faculty who seek to have their students become independent, intentional learners. we do this by first explaining overall expectations that made a difference for engaging students in thinking. next, we describe specific experiences and identify the components of those related to intentional mental processing, thereby, providing insights for those in charge of developing intentional learners. a. expectations. we view the leadership program as developmental in nature. according to douglas robertson (2001), development is a process of adding something, such as thoughts, feelings, or behaviors, to what was there already and, as that something is integrated, having the whole that it is joining, such as a perspective or frame of reference, be transformed. students enter the nsf sfs program with different sets of thoughts, feelings, and behaviors based on unique experiences. we do not expect that they will all progress in the same way or at the same rate. we do, however, expect all students to take responsibility for their own growth and development– that they do their own thinking and own their choices of actions. we understand this as becoming intentional learners. every learning opportunity is planned with that outcome in mind, and students are reminded continually that the assignments are for them, not the professor. the words of the participants reveal differences in our expectation and the kinds of expectations they were accustomed to meeting in order to get a desired grade: in the vast majority of my other classes, i show up for class, jot down a few notes, do the homework, and take the tests. i am not forced to share thoughts of my own or involve myself with others. indeed, most college classes allow students to work in isolation which is a grievous error. leadership class is dramatically different. i am required to voice my own values and perform teamwork. i am not allowed to just sit back and listen. i must synthesize my own thoughts and express them. when we were given the assignment i dreaded it. i knew it was going to be hard and that it was not something i could just do and get over with. without a doubt, meaningful learning is hard work, and we do expect students to fully engage in the hard work required for deeper learning. our overall expectation that students take responsibility for their own growth and development is more fully clarified via the specific expectations of changing habits, thinking, coming to know self, and engaging in metacognition. changing habits. most students (and many faculty) are deeply entrenched in the paradigm of learning where students come to class expecting to be told exactly what to do and how to think, check assignments off a list, take tests that measure how much information has been stored in (short-term?) memory, and then dump the information before moving on to the next class. the good grades received from such practices don’t always translate into abilities to solve ill-defined, real-world problems students will face the rest of their lives (huba and freed, 2000). faculty have a responsibility to help students break through that paradigm and to learn to do the thinking required to become intentional learners, but breaking old habits of being told exactly what to do and then checking things off a list to meet requirements for earning a grade does not come easily: wiersema, j.a., and licklider, b.l. the journal of scholarship of teaching and learning, vol. 7, no. 1, may 2007. 20 i have been having quite a bit of trouble with the interview assignment. i sort of wish there were clearly defined guidelines on what we are supposed to learn, because i’m used to that. this is sort of stepping outside my comfort zone, having to decide what i want to learn and all. i feel much more comfortable now in these settings than i did in the past, and the reason is because i have had numerous opportunities to practice them. i think an important distinction is that i was not forced to do so once or twice because of assignments, after which i could breathe a sigh of relief and never worry about doing so again. knowing that it would be more or less a weekly requirement to share my personal feelings with a larger group, i realized that it was not something i just had to get through, it was something i had to become better at. to be sure, breaking old habits is not easy and it does take time. the time invested, however, reaps rewards as students begin to realize that learning is, in fact, a developmental process: over time, i’ll think of things that i couldn’t today, or later we’ll learn stuff that i can apply that i didn’t know to think about today. i think that is one thing about our class that is really important, and pivotal to our continual growth and learning. by assigning us to go back and look at things we’ve done or somewhat completed, reflect upon them and then apply new things to them we are learning more and getting more out of topics and materials. we don’t just read a chapter and then never go back, we are required to apply those things learned to new assignments later on. it’s a great way to commit those ideas, principles and facts to memory… those words written in a journal entry during the last week of september reveal that kim was beginning to internalize two important concepts about deep learning—it is never finished (developmental) and thinking is critical. thinking. as referenced earlier, it is not the doing that causes the learning. it is the thinking about the doing that causes learning (leamnson, 2000). the challenge, then, for faculty is to help students develop a habit of thinking about all learning opportunities. the words of the participants reveal their understandings that we do expect them to engage in thinking before, during, and after class: coupled with the knowledge i have gained about how i am best able to prepare for such things [interactions during class] (journaling, jotting notes, some kind of critical thinking beforehand), i now feel much more at ease sharing things in this particular group. …if we have an idea we've been talking about all class, i might reflect on it at a certain angle and that's another reason i like the go-rounds. hearing what other people, i always like those go-rounds that say 'what are you going to think about more as a result of this class, this particular session', and just hearing what other people got out of it. sometimes it's completely different than what i did, and it gives me something else to think about that i wouldn't have started thinking about otherwise. ever since wednesday, i’ve been thinking about the group activity we did with the poker chips, surviving the cave… wiersema, j.a., and licklider, b.l. the journal of scholarship of teaching and learning, vol. 7, no. 1, may 2007. 21 as students do begin to change their old habits and being thinking before, during, and after classes, as they begin to develop reflection as a habit of mind, our next challenge is to get them to engage in deeper reflection—to move them toward intentional mental processing. the goal of intentional mental processing must always be for deeper thinking and ultimately deeper learning. it is not just the final stage in a lesson or a time of reflection following an experience (caine and caine, 1997). it includes thinking critically, asking and answering probing questions, exploring alternative perspectives, solving real-world problems, and searching for big ideas and broad applications of new concepts. intentional mental processing leads to deeper understanding, relevant insights, and mastery of the discipline. the following excerpts from students’ journals reveal that they were beginning to do the type of thinking required for intentional learners: i recognize the worth and impact this project had on me and that is the start to further growth and understanding. i learned that there is so much out there to be learned and i understand that in the grand scheme of things there is so much i do not yet understand. expressing beliefs and values is more challenging than just thinking them. it takes more time and effort. you are discussing/reflecting on an event in the past and must then make future decisions based upon your new decisions (or affirmed beliefs). you can’t be stagnant, you have to try and learn from the past and all the thinking that you did afterwards. the potential for growth and gaining a better understanding of yourself is huge; you just need to make sure you take advantage of that. as marti’s words suggest, intentional mental processing leads to insights not only about experiences, but also about self. coming to know self. once students become accustomed to a deeper kind of reflection, we try to “help them take advantage” of this opportunity to learn more about themselves by moving them deeper into intentional mental processing—investment of resources in personal analysis, critical thinking, and application of new knowledge to daily living. the next step is introspection, a detailed mental self-examination of feelings, thoughts, and motives: fortunately, reflecting regularly has put me better in touch with how i am feeling. i realize i have an innate inclination to withdraw. ergo, i should be able to fight the urge and keep making forward progress. i did not have a great sense of what my own values were in the early stages of my leadership training. i rarely ever stopped to reflect upon my learning experiences. i just moved on from one experience to the next without a second thought. as a result, my personal growth was very slow if not nonexistent. as students come to understand more about themselves, they seem to like the control they have over their own growth and development. they become more willing to manage their learning and to own their choices and behaviors—to become the intentional learners they need to be: my writings allowed me to get a handle on why i think and feel the way that i do. plus, journals were an outlet for me to scrutinize my strengths, my weaknesses, my success, and my failures. once i had that information, i could begin to work on bettering myself. wiersema, j.a., and licklider, b.l. the journal of scholarship of teaching and learning, vol. 7, no. 1, may 2007. 22 eventually they begin to think more about their own thinking: interestingly enough… i have discovered something about myself. i have always thought of myself as a traditionalist in many ways. i never pictured myself as one of those new age thinkers… not that i find anything wrong with the new way of thinking about things, but it is really not the way i thought i thought about things. according to huitt (1997), metacognition, thinking about one’s own thinking, is an essential skill for learning how to learn. that supports our notion that intentional learners must practice metacognition as a habit of mind. engaging in metacognition. apparently, students recognized our expectation that they practice metacognition: actually this, if i recall correctly, is a big objective of the whole leadership class idea. most people (myself included) don’t really think enough about their thinking. this can, and does, result in people doing a lot of things for reasons they’re not really sure of themselves… i really think i have a much better understanding of why it is important to think about your thinking than i did before. metacognition means more than just thinking about our own thinking. it requires the wisdom to know one’s ignorance and how one’s patterns of thought and action inform as well as prejudice understanding (wiggins and mctighe, 1998). these words of a student from a self-assessment provide evidence of this kind of thinking: i think this process [metacognition] represented a gradual growth in my awareness of my unawareness. what i mean is that i think i began to realize that i didn’t think enough about my thinking about the world around me… near the end of the semester, i no longer seemed to be thinking as much about what we were doing as about what i was thinking. not to say that i didn’t have thoughts about what we were doing, but often i’d stop and think about where that thought was coming from. indeed, the analysis of the data confirmed our observations that by the end of the first semester students were beginning to question their old paradigms of learning and to embrace a different understanding of what it means to learn. they were well on their way to taking responsibility for their own growth and development. going back to the data revealed numerous specific experiences that contributed to this phenomenon: we have used reflection in a number of ways. the most prominent way is our journal, but we have done other reflection in class, such as jotting down thoughts after an activity or coming up with praise and suggestions for the second year students’ security sessions. throughout all of these activities, i have realized how much my understanding of the topic at hand improves after i have completed some reflection. often the reflection brings up new questions or ideas that i had not originally considered, and these lead to the possibility of even deeper understanding. wiersema, j.a., and licklider, b.l. the journal of scholarship of teaching and learning, vol. 7, no. 1, may 2007. 23 as stated earlier, similar statements from the participants revealed just how much our own habits of mind related to thinking causing learning had influenced almost all of our actions and behaviors related to students and their learning. all our expectations did, in fact, require thinking. our ultimate goal was for that thinking to evolve into intentional mental processing as a habit of mind. all learning opportunities were planned with that end in mind. the analysis of our data, in the words of the participants, revealed the following kinds of experiences that seemed to have the greatest impact on moving the students toward our goal: go ’rounds, team activities, dialogues, application of skills, self-assessments, and journals. b. experiences for each kind of experience, first, we provide a brief explanation of the learning opportunity. next, we offer illustrations, from the voices of the participants, to give insight about the impact for helping students develop intentional mental processing as a habit of mind. and, finally, we offer suggestions for post-secondary educators in charge of student learning. go ’rounds. to encourage both thinking and contributions early, we start and end every meeting with a go ’round. this is one of our favorite interaction strategies. the facilitator poses a question or a notion to elicit a response, and after individual think time, each person is expected to speak. a volunteer is selected to start the go ’round and to determine the direction around the circle following the first response. if an individual is not ready to speak when it is his/her turn, that person may pass until everyone else has spoken. discussion is discouraged during the actual go ’round to provide all individuals the opportunity to reveal their thinking publicly without fear of being ridiculed or judged. during the focus group interview participants were reflecting on those first classes and the early go ’rounds. one participant recalled how difficult it was to meet the expectation of speaking, “when you required us to talk, i was scared to say anything. i would always pass; well, not always, but a lot of the times.” another one recognized how much his willingness to speak up in other groups had changed. when questioned about the reason, he replied, “having go ’rounds every week, you know, always having to say something once or twice every class.” kelly valued those go ’rounds that gave him the opportunity to hear what others were thinking because it often gave him more to consider: …if we have an idea we've been talking about all class, i might reflect on it at a certain angle and that's another reason i like the go-rounds. hearing what other people [are thinking], i always like those go-rounds that say 'what are you going to think about more as a result of this class, this particular session', and just hearing what other people got out of it. sometimes it's completely different than what i did, and it gives me something else to think about that i wouldn't have started thinking about otherwise. as students practiced the thinking required to express thoughts openly, they learned to think more deeply, to challenge the notions they were studying: i have since become convinced that i am better off to form my own leadership philosophy rather than wholly adopt someone else’s. that realization has caused me to think more critically about what i am learning. for example, in the last opening go-round wiersema, j.a., and licklider, b.l. the journal of scholarship of teaching and learning, vol. 7, no. 1, may 2007. 24 i questioned whether a person actually moves through stages of tolerance towards enlightenment as exploring leadership suggests. seemingly, intolerance is a learned behavior. in the beginning of the semester, i did not think nearly so critically. i accepted what i read and heard at face value. helping students move from being afraid to voice their opinions to the group to being willing to critically challenge concepts they have studied takes time. be patient. set ground rules for go ’rounds and hold everyone to them. ground rules we have found most useful include: • each person has the opportunity to talk without any responding. • honor each person’s thinking. • an individual has the right to pass. if one chooses to pass, go back to that person after the go ’round has completed the circle. • no interruptions. • no sidebar conversations. students will watch the facilitator closely; therefore, it is important to model expected behaviors. it is especially difficult, as the facilitator, not to respond to contributions, thereby breaking the first ground rule. instead of commenting as each person finishes, address them by name and thank them. this honors the response and indicates that the go ’round is moving on. only when necessary or helpful, ask probing questions for clarification or to redirect. the question or the notion posed to elicit a response will determine the type of thinking for the student. it is important here to be purposeful. during the first go ’rounds our purpose is simply to get students to speak. it is important to keep the contributions as non-threatening as possible—so everyone will have a response and no one else can suggest it is wrong. our first one is usually, “tell us your name, where you are from, and something you are pretty good at doing.” this allows everyone in the group to start gathering information about their peers and it allows us to guide the next activity toward what they know about learning—“how did you learn or get good at what you shared?” from this point on, all the go ’rounds should serve a purpose and engage students in the type of thinking you desire of them. team activities. for years the study of learning was dominated by a psychological view that focused only on the individual and his/her thinking alone (brandt, 1992), but current cognitive scientists consider learning to be largely a social process (caine and caine, 2001). undeniably, much learning occurs through social interaction. this notion of learning through social interaction means more than just having students practice and recite terminology together (caine and caine, 2001; leinhardt as highlighted in brandt, 1992; wiggins and mctighe, 1998). it means providing them the opportunity to make their implicit knowledge explicit—giving them the chance to explain their thinking to each other, listen to each other, and help each other explain. the words of the participants confirm the notion that it is not the activity itself (the doing) that causes the learning as much as it is the reflection (thinking) following the activity that causes the learning: at the beginning of the semester i was pretty skeptical of the idea of doing icebreaker games for leadership training. i had never had an experience where i left an icebreaker game feeling like i had gained anything (maybe with the exception of the names of the people in the group) from the experience. for the first time in my life, i feel like i have learned from this type of small group activity… each of these activities was designed to wiersema, j.a., and licklider, b.l. the journal of scholarship of teaching and learning, vol. 7, no. 1, may 2007. 25 teach the members something about leadership, and each of these points has stayed with me. i believe this is true because i reflected on the activities. after reflecting on the exercise, i have learned a bit more about myself as it pertains to my contributions in a group setting. as team activities became more complex, students realized the value of discussion for promoting deeper thinking: i also need to work in a group to be at my best. of course, group work needs to come after i have had some alone time so that i feel adequately prepared. once i am prepared, though being in a group allows me to bounce ideas off of others. in a group setting, my train of thought gets revised. i incorporate other people’s thinking into my own thought process and a synergy takes place. i get a much broader and clearer picture when i am put in a team. eventually they came to appreciate the importance of interaction not only during the activity, but also as they discussed interactions during the activity and set goals for the future. kim’s words reveal movement from simple reflection toward intentional mental processing with a focus on analysis: for sure the way the three hour class periods are spent, because they’re spent interactively talking most of the time, performing activities that are fun and then after you’re done with the activity, talking about everything that happened in the activity, that was something that i never would have done before and that was probably the most valuable key as a group, was definitely the most valuable thing that we did. just having to talk about everything, analyze what other people did, analyze what you did… putting students into teams and telling them to work together does not mean that they know how to interact or that they will do so even if they do know how (johnson, johnson, and smith, 1991). we have found three critical components for increasing the chances that having students work in teams will promote learning: 1) students must have a reason to interact, 2) they need to learn skills that will allow them to interact effectively, and 3) they must process the interaction. therefore, it is important to be purposeful in planning interaction, to deliberately teach specific interactive skills, and to develop specific questions to guide processing. as with go ’rounds, we like to start simple and plan for success. our favorite strategy is the turn to your partner (ttyp), which we have adapted from the work of johnson et al. (1991). the purpose of a ttyp is to engage the brain of the learner. general steps for using this strategy include: • purposefully plan a question to cause the type of thinking desired. • allow time for students to formulate responses individually. • ask students to share their responses with their partner and listen to the response of their partner. • encourage students to reach toward a deeper understanding through discussion. • hold students accountable for their discussions by calling on them at random. an appropriate interactive skill for students to practice as they engage in the ttyp is active listening. since we expect students to practice effective interactive skills, we do take time wiersema, j.a., and licklider, b.l. the journal of scholarship of teaching and learning, vol. 7, no. 1, may 2007. 26 to teach the skills—to help them understand our meanings and the steps involved for each skill. we also expect students to practice using the skills during class interactions and outside of class. as students become comfortable working with a partner, we increase the number of individuals on the teams and the complexity of the activities. regardless of the size of the team or the purpose of the interaction, it is critical to be always mindful that it is the thinking that causes the learning. students must engage in thinking about the team activity and must learn how to engage in discussion about the interaction. faculty must continue asking the questions that will cause the students to engage in the kind of thinking and discussion desired until the students learn to ask and answer their own questions. this does take time, but the time invested reaps rewards as students learn to engage in deeper discussions with less structure. meaningful discussion. according to bandura (1977), environments that support and promote interpersonal interaction are more likely to result in greater reflection. the social interaction may increase motivation, prolong on-task engagement, produce more information, and stimulate additional ideas—all contributing to deeper thinking about the experience. our challenge is to facilitate a discussion following the social interaction that encourages students to reveal their thinking to others. during the focus group interview participants revealed that they had come to value such discussions: one thing that i would like to see not change is just how much group discussion there is on everything, whether it be the readings or creative writings or the go-rounds. i like the amount we just talk together. i kind of enjoyed the current events, talking about things, especially if you have something like an election going on, i think to hit that and really discuss what really is going on, takes yourself out of the classroom aspect and more into the world aspect, and anytime you do that i think it's effective… not only did participants seem to value the discussion, they also came to understand that experiencing “interesting, intellectual discussions” may lead to an increase in confidence, resulting in a willingness to take more risks: i feel like i’ve developed more confidence through the course of the program. it is the first real opportunity i’ve had, or at least taken advantage of, to be involved in interesting, intellectual discussions… doing some of these activities from week to week has built up a confidence in me such that i know i am capable of doing these things, even if i do not always feel up to it. getting past the fear of putting myself out in front of everyone, and of being responsible for other people’s learning, has been a big hurdle for me to overcome. i feel that in the future i will be more willing to take risks in these areas… as students embraced the notion “of being responsible for other people’s learning,” our jobs became easier. students learned to challenge one another to think more deeply: in addition to their good advice, my classmates completed their job of making me reflect deeper. i believe adam asked me why i was bothered. after all, someone else acting out has no bearing on me. i had to think hard about that one, but i think i have an answer. wiersema, j.a., and licklider, b.l. the journal of scholarship of teaching and learning, vol. 7, no. 1, may 2007. 27 on top of improving my own thinking, i have helped my classmates think more deeply. in talking about my own struggles as a leader earnestly, i gave my classmates problems to which they could relate. they were then able to think about my dilemmas and determine what they might do in my situation. by causing my cohorts to think more deeply, i received excellent advice from them. i have come a long way from the first few leadership sessions in respect to challenging other peoples’ thinking. i only posited my own ideas rather than eliciting higher thinking from others in the beginning. without a doubt, students were beginning to move toward the kind of interactions senge (1990) believes important for leaders in education, business, and industry who seek ways to turn their institutions into learning organizations, “where new and expansive patterns of thinking are nurtured, where collective aspiration is set free, and where people are continually learning to see the whole together” (p. 3). helping student learn how to engage in such meaningful discussions takes time. as with the go ’round and team activities, start simple and plan for success. structure opportunities where students will experience a supportive environment as they risk making their thinking public. engage them in discussions about reading assignments where they have had time to prepare responses prior to class. ask the type of question that will elicit the type of thinking desired. challenge their thinking by asking them to support their claims. continue asking questions until they learn to challenge themselves and others. the key, then, to helping students move toward intentional mental processing by engaging in meaningful discussion is the same as learning any skill—having the opportunity to use the skill. application of skills. a major aim for undergraduate education is for students to use all they learn in their post-secondary education to solve the problems they will face in the future. this transfer of learning—using concepts in a situation different from where they were learned—is one of the most powerful principles of learning (sousa, 2001). however, it is critical to understand that while the brain does many things automatically, transfer is not one of them (national research council, 2000). using learning in a situation different from that in which it was learned requires intentional mental action on the part of the learner. faculty must help students learn how to do the kind of thinking required for transfer of learning. as students worked to change their old paradigms of learning, they seemed to understand the impact of deliberate practice of new skills on their own growth and development: one of the most important aspects of all the development i have experienced is that i was given the opportunity to practice different skills many times. rather than just reading or talking about a conflict resolution skill, or how to improve group interactions, we actually had quite a few chances where we either explicitly focused on practicing, a skill, such as some of the in-class activities, or we were given other tasks to do, like security education sessions, service learning projects, and while working on those tasks we were able t o put the things we learned about into effect. a practice strategy we like to use to promote transfer of learning is our adaptation of roleplaying. we refer to it as “working a role.” we ask students to work a role as honestly and sincerely as possible to help all of us learn as much as we can. students come to know the value of such practice: wiersema, j.a., and licklider, b.l. the journal of scholarship of teaching and learning, vol. 7, no. 1, may 2007. 28 i can see the value in role-playing certain controversial situations in order to learn more about myself. i hadn’t previously realized that fairness was such a strong value to how i view things. to encourage students to practice new skills outside of class, we set the expectation that they will practice, and we hold them accountable for doing so. asking students to commit to their own growth and development during a final go ’round is a simple way to set the expectation, “tell us one thing you will do during this next week to enhance your own learning.” the opening go ’round during the following week can be used for accountability, “tell us what you did and how it went.” a typical response might be: i sometimes have trouble remembering things, especially where school is concerned, so after last week’s class i was ready to try activating my semantic memory lane and really focusing on remembering more things i learned in class. surprisingly, it seemed to work fairly well. another favorite strategy we use to encourage students to transfer their learning is a “practices inventory.” learners record insights about learning, identify habits or behaviors that are congruent with the insight, identify habits or behaviors that are incongruent with the insight, and develop plans to use the insight to enhance their own growth and development: i did an okay job on the practices inventory. i filled in as much as was required of me, but i did not go any further beyond that. i did spend a lot of time thinking and writing about the insights that i had. consequently, i filled the practices inventory up with a brutally honest look at my behaviors. the amount of thinking i did behind the writing was probably the most beneficial to me. michael’s last statement reveals the key—“the amount of thinking i did behind the writing was probably the most beneficial to me.” critical for helping students learn to use skills in new situations—to transfer their learning—is to engage them in intentional mental processing. faculty owe it to students to help them learn to ask and answer the following questions: • how is this similar to something i already know? how is it different? • how have i used this kind of information in the past? • in what other situations might this be useful? • what implications are there for me as a professional? • how might this knowledge or skill impact my professional development? • what impact does knowing this have on my future? as students learn to transfer their skills into real-world situations, they begin to think of themselves as professionals—to identify strengths and to set goals for improvement: admittedly, i have spent time in introspection and outlining goals for myself but have a tendency only to see what’s in front of me. i do, however, see the value of looking at how far one has come. positive reflection on past successes gives one a sense of confidence, making future goals seem less intimidating and more surmountable. wiersema, j.a., and licklider, b.l. the journal of scholarship of teaching and learning, vol. 7, no. 1, may 2007. 29 this willingness to engage in self-examination sets the stage for students to learn to critically assess their own growth and development. self-assessments. an emphasis on self-assessment is consistent with our goal for students to take responsibility for their own growth and development. not only do students need to learn to do the thinking that will empower them to manage their own learning, they need the ability to critically assess themselves as professionals when they enter the work force. therefore, we require students to formally assess their own learning and progress at the end of each semester. an excerpt taken from one student’s self-assessment written at the end of the year reveal the ability to think critically about self growth and development: the final major area of development worth noting is my journaling. after reading through last semester’s journal entries and comparing them to this semester’s, i noticed quite a difference. the first is my topic of writing—last semester it seemed pretty sporadic, and now the focus has improved quite a bit, as noted in my analysis. i also do not recall ever writing about current leadership issues in last semester’s journal, and i have learned it can be useful to keep up to date with both good and bad leadership examples, and be able to recognize the differences between them through journaling and critical thinking. other improvements include thinking through decisions and rationalizing their outcomes, and increasing my depth of reflection overall. once again, the words of the participants reveal the key to self-assessment—intentional mental processing. students need to engage in deliberate thought about what they are learning and how they are learning it (national capital language resource center, 2004). this kind of reflection allows them to step back from the learning process and think about their own learning strategies and their own progress—an important step in becoming independent learners. self-assessment at the end of each semester is a critical strategy for moving students in this direction. when completing the semester self-assessment, students typically find that their journals provide solid evidence of their development as learners: one of the best places to look for evidence of learning and growth is my reflection journal. at the beginning of the semester, many of my entries did not contain much depth. i would simply write down an idea and not think fully about all the possible viewpoints. however, as time progressed and i reviewed the rubrics, i slowly began to use a more in-depth process in my decision-making and justification of ideas. “we know the power of self-reflection to deepen learning for adults… one of the strongest motivators is the opportunity to look back and see progress” (chappuis, p. 42, 2005). chappuis’ words support our findings that being required to record thoughts regularly in a journal is one of the most powerful experiences for helping students develop intentional mental processing as a habit of mind. journals. faculty typically ask students to do mental processing in different ways, such as responding to teacher-directed questions, discussing with team members following activities, or sharing their thinking during large group discussions, but it has more meaning for students if they actually record their thinking on paper: wiersema, j.a., and licklider, b.l. the journal of scholarship of teaching and learning, vol. 7, no. 1, may 2007. 30 journal writing connects students with their emotional selves and core values. through writing, students become aware of the relevance of their belief systems. through writing, they begin a healthy habit of reflecting on moral values as they consider problems and issues that come up in their studies and in their daily lives. i have found that students want to discuss topics that touch on important moral questions. (wanket, 2005, p. 74) this is the purpose of the reflection journal as students take responsibility for their own learning and development within the community of learners. students often engage in deeper thinking while recording thoughts in their journals. in addition to framing and guiding their thinking throughout the course, the journals provide evidence of growth and development along the journey. for many students in the nsf sfs program keeping a journal was a new experience. it was necessary for us to persist longer than they resisted before they would realize the benefits of doing the thinking and investing the time required to record their thoughts: when i first heard we were going to be doing a journal i was apprehensive and a little disconcerted. i have never done any journaling before, and i was a little reluctant to do it fearing i wouldn’t be able to put my thoughts to paper. however, as it turned out, it is a great learning tool for me, more so than i would have thought initially. i figured at first that it would be something i’d slog through and do as well as i could, but i didn’t really expect great returns on the time i invested. in the case of the leadership journal, at first i did it simply because it was a requirement of being in the class. over time it developed into a valuable tool for me, but the problem is that i wouldn’t have done it in the first place if i wasn’t “forced” to. eventually, students were able to identify specific examples of how writing in their journals contributed to their growth and development as intentional learners: additionally, the encouragement of reflecting on different things happening inside and outside of class helped me to solidify my thinking more, and think more critically about the things that happened. i think the hardest part of reflection is making yourself do it, but making it an integral part of the leadership development courses has helped motivate me to put my thoughts on paper. …by forcing myself to sit down and come up with nearly a page or more of writing about a semi-focused topic, i definitely develop that idea more than i would with only mental thoughts; i think it is easier to push oneself to find more insight in this manner. now that i have spent a semester keeping a regular journal about various issues relating to our coursework… i have developed a great appreciation for the value of putting thoughts down on paper and giving more critical consideration to them. there were many journal entries that i started writing with one idea in mind, and by the time i finished i had come to a completely new, unexpected realization. students will bring a myriad of experiences with keeping journals and a variety of attitudes. some students will welcome the challenge, but others likely will resist. most important wiersema, j.a., and licklider, b.l. the journal of scholarship of teaching and learning, vol. 7, no. 1, may 2007. 31 in overcoming the resistance is setting the expectation that students will keep a journal and holding them accountable for doing so. once students understand that they will be held accountable for recording their thoughts in journals they will likely need direction to develop skills leading toward intentional mental processing. two simple suggestions offered by wanket (2005), a high school english teacher, are applicable for learners of all ages: date every entry, and write without ceasing. the journals will become logs of their thinking. students will learn to read through their journals and track their own growth. there will be times that dates of an entry are important to them. in addition, early on, students likely will need to force themselves to make entries. specific dates will be reminders of minimum expectations set either by self or by instructor. “write without ceasing” helps remove the burden of perfect writing. some students are likely to be inhibited by their perceived skills as writers. encouraging them to go wherever their minds wander usually will lead to more original insights and creative thinking. once the students get used to the idea of recording their thoughts in journals, they will be more receptive to additional guidance. our experience has been that one of the most important notions to help students internalize is the idea that they are keeping the journal for themselves – not for us. students have become so accustomed to having instructors tell them how they did and how to improve that they often hesitate to think for themselves. it is not until the student truly embraces that idea of ownership that he/she is ready to explore and expand her/his own thinking. at this point, providing prompts to inspire deeper thinking is important: • how can i use this? • why did i react that way? • how is this similar to something else i understand? how is it different? • what other applications might there be? what are the implications thereof? • what does this mean for me as a professional? v. conclusion. indeed, the students recognized that they were developing intentional mental processing as habit of mind: everything we do from our journaling to class participation to our interview projects has involved diligent reflection. and because of this i have been able to weigh and consider all of my actions, thoughts, beliefs and the information gathered from outside sources to enhance my learning and growing experience… this understanding about the importance of reflection is something i can use in almost any situation in the future, especially in difficult times, to explore my thoughts and understandings further. faculty have the responsibility to help their students develop abilities to solve ill-defined, real-world problems they will face the rest of their lives. this requires the ability to learn continuously and to think critically. moving students from simple reflection to intentional mental processing (identified above as diligent reflection by one of the participants) as a habit of mind will better equip them to have a positive impact on the world. in doing our own intentional mental processing about what these participants had to say, we realized the key to helping students become responsible learners is in doing whatever it takes wiersema, j.a., and licklider, b.l. the journal of scholarship of teaching and learning, vol. 7, no. 1, may 2007. 32 to get them to think. in reflecting about “whatever it takes,” we concluded the foundation of every strategy we employ is asking questions and expecting students to develop answers. even if faculty feel ill-equipped to engage students in some of the learning experiences we have described, they can work on purposefully planning and asking the questions to promote deeper thinking in students. key questions might include: what do you think about…? why do you think that? how is this similar to…? how is this different…? what did you do? why did you do it? what do you conclude about…? what is your evidence? why does it matter? how does this connect to…? what have you learned about…? what is your evidence that you have learned it? what are the implications of…? what difference will this make in the/your future? eventually, students will learn to ask and answer challenging questions on their own, and their emerging intentional mental processing as a habit of mind will support their development as responsible learners. references bandura, a. (1997). social learning theory. englewood cliffs, nj:prentice-hall. brandt, r. (executive editor). (1992). learning about learning [videotape]. alexandria, va: association for supervision and curriculum development. caine, g., and caine r. n. (2001). the brain, education, and the competitive edge. lanham, md: scarecrow press, inc. caine, r. n., and caine, g. (1994). making connections: teaching and the human brain (rev. ed.). menlo park, ca: addison-wesley publishing company. caine, r. n., and caine, g. (1997). education on the edge of possibility. alexandria, va: association for supervision and curriculum development. chappuis, j. (2005). helping students understand assessment. educational leadership, 63(3), 39-43. colaizzi, p. f. (1978). psychological research as the phenomenologist views it. in r. valle and m. king (eds.) existential-phenomenological alternatives for psychology (pp. 48-70). new york: oxford university press. dickman, m. and standford-blair, n. (2002). connecting leadership to the brain. thousand oaks, ca: corwin press, inc. huba, m., and freed, j. (2000). learner-centered assessment on college campuses: shifting the focus from teaching to learning. needham heights, ma: allyn and bacon. huitt, w. (1997). metacognition. educational psychology interactive. valdosta, ga: valdosta state university. retrieved september 10, 2005, from http://chiron.valdosta.edu/whuitt/col/cogsys/metacogn.html. wiersema, j.a., and licklider, b.l. the journal of scholarship of teaching and learning, vol. 7, no. 1, may 2007. 33 johnson, d.w., johnson, r.t., and smith, k.a. (1991) cooperative learning: increasing college faculty instructional productivity. ashe-eric higher education report no. 4. washington, d.c.: the george washington university, graduate school of education and human development. leamnson, r. (2000). learning as biological brain change. change: the magazine of higher learning, 32(6), 34-40. merriam, s.b., and associates. (2002). qualitative research in practice: examples for discussion and analysis. san franciso: jossey-bass. moustakas, c. (1994). phenomenological research methods. thousands oaks, ca: sage. national capital language resource center (2004). assessing learning: peer and self assessment. retrieved april 10, 2006, from http://www.nclrc.essentials/assessing/peereval.htm. national research council (2000). how people learn: brain, mind, experience, and school. washington, dc: national academy press. robertson, d.r. (2001). beyond learner-centeredness: close encounters of the systemocentric kind. journal of faculty development, 18(1), 7-13. senge, p. (1990). the fifth discipline: the art and practice of the learning organization. new york: doubleday/currency. sousa, d. (2001). how the brain learns [2nd ed.]. thousand oaks, ca: corwin press, inc. wanket, m. o. (2005). building the habit of writing: journal writing is strength training for the mind. educational leadership. september, 74-77. wiggins, g. and mctighe, j. (1998). understanding by design. alexandria, va: association for supervision and curriculum development. 503 service temporarily unavailable 503 service temporarily unavailable nginx/1.14.1 503 service temporarily unavailable 503 service temporarily unavailable nginx/1.14.1 503 service temporarily unavailable 503 service temporarily unavailable nginx/1.14.1 microsoft word young final journal of the scholarship of teaching and learning, vol. 8, no. 2, may 2008, pp. 1-10. metacognitive awareness and academic achievement in college students andria young and jane d. fry 1 abstract: the researchers examined the metacognitive awareness inventory (mai) (schraw and dennison,1994) to determine how it relates to broad and single measures of academic achievement in college students. correlations were found between the mai and cumulative gpa as well as end of course grades. scores on the mai significantly differ between graduate and undergraduate students. professors’ use of the mai as a potential screening tool to identify students requiring metacognitive strategy intervention is discussed as well as implications for future research. keywords: metacognition, metacognitive awareness, gpa, grade. college professors today are faced with classrooms full of students who come to them with varying levels of knowledge about how they learn. some students are active, self directed learners who know how they learn and are able to apply what they know to various learning situations. others may be average students who work hard and who have awareness of their learning strengths and weaknesses, but who may not adequately regulate their learning. still others may be passive learners who have little awareness of how they learn and how to regulate their learning. in essence, professors are faced with classrooms full of students who come to them with various levels of metacognitive skills. metacognition is generally defined as the activity of monitoring and controlling one’s cognition. it can further be defined as what we know about our cognitive processes and how we use these processes in order to learn and remember (ormrod, 2004). researchers further conceptualize metacognition by breaking down metacognition into two subcomponents, metacognitive knowledge and metacognitive regulation. these two subcomponents have been theorized to be related to one another (brown, 1987; flavell, 1987; schraw and dennison, 1994). metacognitive knowledge can be described as what we know about our own cognitive processes. declarative, procedural and conditional knowledge may all be considered subcomponents of metacognitive knowledge (schraw and moshman, 1995). declarative knowledge involves what we know about how we learn and what influences how we learn. procedural knowledge is our knowledge about different learning and memory strategies/procedures that work best for us. conditional knowledge is the knowledge we have about the conditions under which we can implement various cognitive strategies. as a whole, our knowledge of cognition refers to what we know about how we learn; what we know about the procedures and strategies that are the most effective for us; and, what we know about the conditions under which various cognitive activities are most effective (schraw and moshman, 1995). 1 school of education and human development, university of houston –victoria, 3700 n. ben wilson st., victoria, tx 77901, younga@uhv.edu. young and fry journal of the scholarship of teaching and learning, vol. 8, no. 2, may 2008. 2 metacognitive regulation in contrast to metacognitive knowledge may be thought of as the actual activities in which we engage in order to facilitate learning and memory (schraw and moshman, 1995). metacognitive regulation can be broken down into three component activities. these include planning, monitoring and evaluating. planning involves just that, planning out a cognitive task by selecting appropriate strategies and cognitive resources. monitoring involves the awareness of our progress through a cognitive task and our ability to determine our performance. finally, evaluating involves taking a look at the outcome and determining if the learning outcome matches our learning goals and if the regulation processes we used were effective (schraw and moshman, 1995). it stands to reason that if students have well developed metacognitive knowledge and metacognitive regulatory skills and they use their metacognition they will excel academically. consequently, it is important to be able to assess metacognition of college students to determine if this knowledge and skills are related to academic achievement. if we can say that metacognitive knowledge and skills are related to measures of academic success then professors can use various techniques to assess their students’ metacognition and develop means by which to improve students’ metacognition when necessary. i. metacognitive assessment and academic achievement. researchers have examined metacognition and how it relates to measures of academic achievement. in these studies metacognitive skills are measured in terms of metacognitive regulation, metacognitive knowledge or both of these components. however, these components are measured differently within the literature. some researchers use self report inventories to assess metacognitive skills and relate them to achievement measures (schraw and dennison, 1994; sperling et al., 2004). other researchers examine metacognitive judgments in the form of monitoring accuracy as a measure of metacognitive regulation on various tests (everson and tobias, 1998; nietfeld et al., 2005; schraw, 1994). monitoring accuracy is measured in terms of what is considered calibration of performance. calibration of performance judgments are made at the local and global levels. local judgments are made after each item on a test. local monitoring accuracy is determined to be the average difference between the actual answer of each test question and the students’ judgment of how well they answered each question. global judgments are made after the entire test is completed. students are to judge how well they think they did on the test as a whole. global monitoring accuracy is determined to be the difference between the overall test score and the students’ judgment of how they did on the test. local monitoring accuracy is thought to be a measure of ongoing metacognitive regulation during testing and global monitoring accuracy is thought to be a measure of cumulative metacognitive regulation (nietfeld, et al 2005). the following is a brief review of studies utilizing both survey and measures of monitoring accuracy to assess metacognitive knowledge and/or metacognitive regulation. everson and tobias (1998) were interested in knowledge monitoring accuracy. this skill is thought to be involved in metacognitive regulation. they developed a means to assess students’ knowledge monitoring ability (kma) by examining the difference between students’ estimates of their knowledge in the verbal domain and their actual knowledge as determined by performance on a standardized verbal test. they found the greatest relationship to be between the kma and students’ end of course grade in english, then the humanities and the students’ overall young and fry journal of the scholarship of teaching and learning, vol. 8, no. 2, may 2008. 3 gpa. they also found that this measure of metacognitive regulation, the kma, was related to academic achievement in college and it was a good predictor for success in college. schraw (1994) was interested in the relationship between metacognitive knowledge and metacognitive regulation. he measured metacognitive knowledge by asking students to rate how well they thought they could monitor their accuracy on a series of multiple choice reading tests. he measured metacognitive regulation at both the local and global levels by having students rate accuracy for each question then rate their accuracy after completing the tests. based on the results of his study, schraw suggested that adult students may differ not so much in their metacognitive knowledge skills but in their metacognitive regulation skills. he further suggested that metacognitive knowledge may develop independently of metacognitive regulation. finally, schraw found that actual test performance was significantly correlated with judgments of test performance made before testing, a measure of metacognitive knowledge. test performance was also correlated with metacognitive regulation in that he found correlations between performance and local and global judgments. nietfeld et al (2005) examined metacognitive regulation by measuring monitoring accuracy at the local and global level on a series of multiple choice tests given as a part of a semester long course. they found that monitoring accuracy remained stable across tests throughout the semester. they also found that students were more accurate in their global predictions than their local predictions. they found that student performance on the tests was related to local monitoring accuracy. schraw and dennison (1994) developed the metacognitive awareness inventory (mai) to assess metacognitive knowledge and metacognitive regulation which they referred to as the knowledge of cognition factor and the regulation of cognition factor. the mai consists of 52 questions tapping into these two components of metacognition. they found that there was strong support for the knowledge of cognition and regulation of cognition components and that these two components were related as had been suggested in the research (brown, 1987). schraw and dennison (1994) also tested the convergent validity of the mai by comparing mai scores with other measures thought to be related to metacognition such as pretest monitoring ability, actual test performance and the ability to accurately monitor test performance. they did not find a significant relationship with regard to monitoring accuracy and the mai or between pretest judgments and monitoring accuracy. they found the knowledge of cognition factor of the mai was related to higher test performance and the regulation of cognition factor of the mai was not.they also found that knowledge of cognition as measured by pretest judgments was related to the mai. pretest judgments were also related positively to test performance. sperling et al (2004) utilizing the mai to determine college student metacognitive awareness, found a significant correlation between the knowledge of cognition factor and the regulation of cognition factor. they also were interested in whether the mai would be correlated with other measures of academic achievement such as sat scores and high school average. they found no relation between scores on the mai and measures of academic achievement. they were surprised to find a negative correlation between sat math scores and the mai scores. overall, the findings in the research reviewed above regarding the correlation of metacognition with academic and achievement measures indicate that when regulation of cognition is measured by having students estimate their performance on either a local or global young and fry journal of the scholarship of teaching and learning, vol. 8, no. 2, may 2008. 4 level, regulation of cognition is related to test performance, domain specific gpa scores and overall gpa scores (everson and tobias, 1998; nietfeld et al, 2005; schraw, 1994). it appears that when metacognition is assessed through calibration of performance measures there is support for the relationship between metacognitive skills and measures of academic achievement. unfortunately, determining monitoring ability and monitoring accuracy at the local and global level to assess metacognitive knowledge and regulation skills is a labor intensive endeavor. this situation is especially true for students who are assessed in their actual college classes and not a laboratory or contrived setting. students monitoring their accuracy on a local and global level must take the time to answer the test questions and then respond to how confident they were about their performance on each question. this process can be a time consuming and possibly stressful task for students while taking tests that will count toward their end of course grades (nietfeld, 2005). it is important to assess students in a less intrusive manner in order to ascertain their metacognitive awareness and skill level. a less intrusive assessment such as a questionnaire, will allow instructors to quickly identify struggling students early on and assist them in developing effective metacognitive skills. in a departure from utilizing metacognitive judgments as a method to determine metacognitive skills schraw and dennison (1994) developed the mai as a quick and easy means to assess metacognitive awareness. as reported above they found the mai correlated with reading comprehension test performance, a measure of academic achievement, only on the knowledge of cognition factor. sperling et al (2004) did not find a correlation with more comprehensive measures of academic achievement such as sat scores or high school gpa. obviously the results of the studies in which the mai was used to assess metacognition are mixed. the mai, needs to be examined further and in a broader context. instruments used to assess metacognition must be sensitive to comprehensive measures of academic achievement that require a variety of cognitive skills in addition to general verbal ability. assessments must be easy to administer and score so professors can use the information to help students over the course of a semester. metacognitive assessments must also be comprehensive assessments of the theorized components of metacognition, namely metacognitive knowledge and metacognitive regulation. the purpose of the present study was to further examine the relation between metacognition and broad based measures of academic achievement within a natural classroom setting. the mai was chosen because it is an easy to administer survey for adults, which can be delivered in both face to face and online classes. additionally, the mai taps into the two component model of metacognition, metacognitive knowledge and metacognitive regulation cited in the research (brown, 1987; schraw and dennison, 1994). furthermore, with the mai researchers can analyze relationships between metacognitive skills and specific academic skills such as scores on classroom tests, reading comprehension tests etc.researchers can use the mai to analyze for relationships between metacognitive skills and broader measures of academic achievement such as cumulative gpa, sat scores and other standardized scores. the final purpose of the study is to add to the body of knowledge regarding the validity of the mai in terms of the statistical relationship between metacognitive knowledge and regulation; and in terms of the convergent validity of the mai with measures of academic achievement. given the findings thus far regarding the mai and academic achievement measures the primary goal of this study was exploratory in nature. the researchers were interested in correlations between the mai and end of course grades; the mai and cumulative gpa; and the young and fry journal of the scholarship of teaching and learning, vol. 8, no. 2, may 2008. 5 mai and single tests within a semester long course. furthermore, the researchers were interested in whether scores on the mai would distinguish between experienced and less experienced students as measured by class standing as graduate or undergraduate. ii. method. a. participants. undergraduate and graduate education students at a small upper level (junior, senior and graduate level) institution located in southeast texas were invited to take part in the study. junior and senior level students in undergraduate teacher education classes in reading and human learning were asked to voluntarily complete the metacognitive awareness inventory (mai) during summer and fall semesters. additionally, graduate students in master’s education programs taking core classes were asked to voluntarily complete the mai during summer and fall semesters. the mai was offered in a total of 15 classes. two classes were delivered face to face, while the remaining classes were delivered online. the requirements of the fifteen classes from which students were drawn included multiple choice tests, online discussions with specific content criteria, projects and lesson plans. students in both face to face and online classes had access to the mai online through webct. they were told they could take the mai at any point during the semester in which they were enrolled in the participating class. one hundred and seventy eight students completed the mai. forty five or 25.3% were graduate students and 133 or 74.7% were undergraduate students. one hundred and fifty eight or 88.8% of the respondents were enrolled in online classes. the remainder was enrolled in face to face classes with access to an online component of the class. see table 1 for student characteristics. table 1. student characteristics. gender age level of college education credit hours taken semester of participation gpa male 3.9% 20-25 yrs 32.8 % <2yrs. 1.7% 3-6 35% 4.0-3.5 46.9% female 96.1% 26-30 yrs 22.6% 2yrs. 17.4% 7-12 36.3% 3.49-3.0 32.8% 31-35 yrs 15.3% 3yrs. 20.8% 13-18 17.5% 2.99-2.5 15.3% 36-40 yrs 5.6% >3yrs. 34.3% 19-21 2.1% 2.49-2.0 5.1% 41-45 yrs 14.7% bachelors 22.5% >21 .9% over 45 9.0% masters 3.4% young and fry journal of the scholarship of teaching and learning, vol. 8, no. 2, may 2008. 6 b. materials. the mai (schraw and dennison, 1994) with permission of the first author was used to measure students’ metacognitive awareness. the mai consists of 52 statements which students rate as being false or true on a five point likert scale. the two components of metacognition discussed above are represented within the scale, metacognitive knowledge and metacognitive regulation. within the mai these are referred to as the knowledge of cognition factor and the regulation of cognition factor. within the inventory there are 17 questions related to the knowledge of cognition factor for a possible point total of 85.there are 35 questions related to the regulation of cognition factor for a possible point total of 175. the factor scores are calculated by adding the scores on questions related to each of the factors. higher scores correspond to greater metacognitive knowledge and greater metacognitive regulation. in addition to the knowledge of cognition score and the regulation of cognition score a mai total score is derived by summing responses to all 52 questions. the instrument was designed for use on adult populations. the mai was transformed into a web format so it could be completed by students online. c. procedure. in each of the fifteen classes a link to the mai was set up on webct for students to access. students in face to face classes had an online component to their classes so they too had access to the mai via webct. the letter within the link explained the mai and the purpose of the study. students were asked to consent to complete the mai and to provide their names on the mai so their end of course grades could be associated with their score on the mai. students were not provided incentive in the form of additional points to complete the mai as this extra credit would skew their end of course grades and confound the results of the study. consequently, each class had approximately a 50% response rate. students were told they could complete the mai at anytime during the semester. the experimenters downloaded the mai responses only after the end of course grades were submitted to the registrar in order to avoid bias in assigning end of course grades. iii. results. a. correlations between mai and measures of academic achievement. for the 178 respondents the mean mai score was 206.85. the mean score for the knowledge of cognition factor and regulation of cognition factor was 68.69 and 138.16 respectively. see table 2 for means and standard deviations of the mai. table 2. means and standard deviations of the mai. m sd mai total 206.85 20.99 knowledge of cognition factor 68.69 7.28 regulation of cognition factor 138.16 14.94 (n=178) young and fry journal of the scholarship of teaching and learning, vol. 8, no. 2, may 2008. 7 in order to determine if there were relationships between the knowledge of cognition and the regulation of cognition factors, as well as correlations between scores on the mai and achievement measures of gpa and end of course grades spearman’s rho, nonparametric correlation analysis was completed. there was a significant correlation between the knowledge of cognition factor and the regulation of cognition factor r = .73, p<0.01. see table 3. table 3. correlations between mai scores and broad measures of achievement. course grade gpa mai total regulation factor knowledge factor grade 1.00 0.36** 0.19* 0.19* 0.20** gpa 0.36** 1.00 0.23** 0.20* 0.26* mai total 0.19* 0.23** 1.00 0.97** 0.87** knowledge factor 0.20** 0.26** 0.86** 0.73** 1.00 regulation factor 0.19* 0.20** 0.97** 1.00 0.73** ** correlation is significant at the 0.01 level *correlation is significant at the 0.05 level there was a correlation between the total score of the mai and end of course grades. breaking this down into the two factors of knowledge of cognition and regulation of cognition a correlation was found between each of these factors and end of course grades. there was a correlation between gpa and the knowledge of cognition factor and also between gpa and regulation of cognition factor. see table 3. albeit relatively modest correlations, these results show a relation between both the knowledge of cognition and regulation of cognition factors of the mai and broad measures of academic achievement. a subset of 65 students from the current sample was drawn in order to examine whether the mai would correlate to single tests within semester long courses. subjects from four undergraduate sections of an online course titled, “human learning and its application to education” were drawn for this purpose. this sample was chosen from the larger sample as it contained the largest number of students within the sample taking the same course; each section had the same tests; and course delivery was consistent across sections. three multiple choice tests were given throughout the semester. multiple choice questions covered course content and required students to know facts and be able to synthesize and apply information. spearman’s rho nonparametric correlation analysis was completed on the data. there were no significant correlations between test 1 and scores on the mai; nor were there significant correlations between test 2 and scores on the mai. there was a correlation between test 3 and the knowledge factor r= 0.26, p<0.05 and the regulation factor r = .27, p<0.05 of the mai. b. mai scores and individual differences. the researchers were also interested in whether there were differences in scores on the mai between more experienced graduate student learners and less experienced undergraduate student learners. consequently analysis of variance (anova) was performed. the independent variable was whether a student was a graduate or undergraduate student and the dependent variables were the regulation of cognition and knowledge of cognition factor scores. there was not a significant difference between the two groups on the knowledge of cognition factor. there young and fry journal of the scholarship of teaching and learning, vol. 8, no. 2, may 2008. 8 was a difference between graduates and undergraduates with regard to the regulation of cognition factor f(1,177) = 4.13, p<0.05. the mean score on the regulation of cognition factor for graduates was 142.04 and 136.85 for undergraduates. iv. discussion. the purpose of the present study was to further explore the mai and its relationship to broad and single measures of academic achievement. as was expected and found in previous research (schraw and dennison, 1994; sperling et al, 2004) there was a significant correlation between the regulation of cognition factor and the knowledge of cognition factor. significant correlations were found between the mai and broad measures of academic achievement. the knowledge of cognition factor of the mai was correlated with gpa and end of course grades. the same is true for the regulation of cognition factor. these results also provide support for the validity of the mai as it relates to academic measures. within this study significant differences were found between graduate and undergraduate students with regard to their scores on the regulation of cognition factor of the mai but not the knowledge of cognition factor. this supports the authors’ contention that if the mai is a good measure of academic achievement then it should yield scores that distinguish between more and less experienced students. graduate and undergraduate students do not differ in relation to knowledge of cognition, they do differ in terms of their regulatory skills. this finding supports that of schraw (1994) who found that adult learners tend to differ with regard to the use of metacognitive regulatory skills and not so with regard to metacognitive knowledge skills. the results of the correlations between the mai and single test scores within a course were unexpected. in the current study, the mai is better correlated to broad measures of academic achievement such as gpa and end of course grades rather than single measures. it seems there may be other factors that confound the relation between the mai and single test performance. single test performance grades may be impacted by many variables other than one’s utilization of metacognitive regulation and knowledge skills. these confounding factors may be physical illness, variations in personal motivation, and, potential problems with the technology required for the online class. broad measures such as gpa and end of course grades which are measures of academic performance over time are much less sensitive to these vagaries of everyday life. this possibility is one that warrants further research to determine how factors other than an individual’s metacognitive abilities temper learning as measured by single test scores. the results of this study are promising. given the positive correlations between the mai and end of course grades as well as gpa it can be a tool for professors to use to screen students in need of direct instruction related to metacognition. this may become especially important in large classes as well as online classes where professors have little opportunity to get to know their students on an individual basis. professors can flag students who obtain low scores on the mai and then use the mai as a means to determine what type of metacognitive knowledge and regulatory skills the student reportedly utilizes while learning. the mai is set up so professors can complete an item analysis for low scoring students. each of the 52 items within the mai is a statement about one’s knowledge of learning or the activities one must undertake to regulate learning. for example, “i understand my intellectual strengths and weaknesses.” and “i have control over how well i learn.” (schraw and dennison, 1994, p. 473) are examples of questions related to metacognitive knowledge. “i pace myself young and fry journal of the scholarship of teaching and learning, vol. 8, no. 2, may 2008. 9 while learning in order to have enough time.” and “i set specific goals before i begin a task.” (schraw and dennison, 1994, p. 473) are examples of questions related to metacognitive regulation. professors can examine responses to statements like these and specifically pinpoint areas students are reporting weaknesses. professors can then tailor instructional intervention related to metacognitive knowledge and regulation to meet the needs of individual students. the use of the mai as a screening tool and a tool to identify specific metacognitive weaknesses merits further research. v. future research. in the future the goal will be to further examine the relation between the mai and measures of academic achievement with larger, random samples of students. the intent behind this is to determine if more robust correlations can be obtained when sampling is random and sample sizes are larger. in addition, future research will focus on using the mai in the applied setting of a classroom to identify and assist students in developing their metacognitive skills. the mai will be administered to two sections of the same class, one class of students will serve as the control and the other class of students who score low on the mai will be identified for metacognitive instruction. analysis will be completed to determine if students with similar mai scores in the two classes differ on course related achievement measures when the experimental class members with low scores receives metacognitive instruction and the control class members with low scores receives no metacognitive instruction. vi. limitations. students participating in the study may not be representative of all adult learners as they were primarily education majors. additionally, students self selected for the study by volunteering to participate. thus, higher performing students may have been overrepresented in the sample. finally, associations between various measures within this study may be confounded by additional variables that were not measured such as motivation, students’ personal time constraints etc. references brown, a. (1987). metacogntion, executive control, self-regulation, and other more mysterious mechanisms. in f. weinert and r. kluwe (eds.), metacognition, motivation and understanding (pp. 65-116). hillsdale, nj: erlbaum. everson, h. t. and tobias, s. (1998). the ability to estimate knowledge and performance in college: a metacognitive analysis. instructional science, 26, 65-79. flavell, j.h. (1987). speculations about the nature and development of metacognition. in f. weinert and r. kluwe (eds.), metacognition, motivation and understanding (pp. 21-29). hillsdale, nj: erlbaum. young and fry journal of the scholarship of teaching and learning, vol. 8, no. 2, may 2008. 10 hacker, d. j., bol, l. horgan, d.d. and rakow, e.a. (2000). test prediction and performance in classroom context. journal of educational psychology, 92(1), 160-170. nietfeld, j. l., cao, l. and osborne j. w. (2005). metacognitive monitoring accuracy and student performance in the postsecondary classroom. the journal of experimental education, 74(1), 7-28. ormrod, j.e. (2004). human learning. upper saddle river, nj: pearson prentice hall. schraw, g. (1994). the effect of knowledge on local and global monitoring. contemporary educational psychology, 19, 143-154. schraw, g. and moshman, d. (1995). metacognitive theories. educational psychology review, 7(4), 351-371. schraw, g. and dennison, r. s. (1994). assessing metacognitive awareness. contemporary educational psychology, 19, 460-475. sperling, r. a., howard, b. c., staley, r. and dubois, n. (2004). educational research and evaluation, 10(2), 117-139. 503 service temporarily unavailable 503 service temporarily unavailable nginx/1.14.1 v4n1mckinney journal of scholarship of teaching and learning, vol. 4, no. 1, may, 2004, pp. 13 – 22. learning sociology: successful majors tell their story kathleen mckinney1 abstract. the focus of this study is to begin to answer the question of how sociology majors come to learn their discipline. in this article, i report on the findings from the first study in a multi-method project on this topic. i conducted a group interview with honors sociology majors from around the united states. students discussed several questions related to learning sociology. themes that emerged, and which may be useful in a number of disciplines, included the need to make connections, the importance of other people, talking about the material, experiencing varied pedagogies, and the active construction of knowledge. keywords: learning, honors students, student perceptions i. introduction. the central question i pose in this research is “how do sociology majors successfully learn sociology?” the study reported here, using a focus group or group interview of honors sociology majors, is one study in a multi-method project investigating this question. the objective of the focus group phase of the study is to obtain exploratory, qualitative data from the point of view of these honors students on how they believe they learn sociology. in this phase, learning of sociology is not directly measured, rather i assume that these honors students, selected to attend the american sociological association annual meetings, are by definition successful learners of the discipline. i am also gathering and analyzing data from learning logs, qualitative interviews, and self-administered quantitative questionnaires with sociology seniors at one, mid-sized public institution. the primary objectives of the full research project are the following: 1. to uncover the learning strategies, both inand out-of-class, alone or with others, in terms of study skills or other behaviors, which distinguish more and less successful learners of sociology, 2. to uncover any demographic, learning style, or motivational correlates of these strategies and of learning, and 3. to share this information with colleagues in sociology and in other disciplines in order to positively impact pedagogy, curriculum, and learning. 1 kathleen mckinney is the cross endowed chair in the scholarship of teaching and learning, and professor of sociology at illinois state university. this research was funded by a grant from the american sociological association teaching endowment fund and is part of a larger, multi-method project supported by the author’s selection as a 2003-2004 carnegie scholar. the author gratefully acknowledges the contributions of the nine honors students who participated in the focus group and kerry strand who assisted in organizing the focus group at the 2003 american sociological association annual meetings. thanks, also, to tom gerschick and michael loui for their comments on earlier drafts. mckinney, k. journal of scholarship of teaching and learning, vol. 4, no. 1, may, 2004. 14 this line of research is significant for a number of reasons. theoretically, the findings will have implications for several areas of higher education scholarship and for the literature on teaching and learning in sociology. on a practical level, faculty, staff, and students can apply the results at the course and department levels to improve learning. in addition, the results will have implications for faculty and students in related disciplines, as well as staff involved in support services for student learning in higher education. based on the literature i reviewed, little past empirical work has been done that focuses on how students learn from the viewpoint of students and, specifically, for the discipline of sociology. for example, experiments or quasi-experiments have been conducted assessing how different study strategies, note-taking strategies, or assignments impacted learning in psychology courses (e.g., hartlep and forsyth, 2000; kreiner, 1997; watson, hagihara, and tennery, 1999). novices and experts in a discipline, such as physics, have been compared in their approach to problem-solving (e.g., chi, feltovich, and glaser, 1981). some researchers have used interviews, case studies, focus groups or “think alouds” to assess students’ study strategies or views of learning in disciplines other than sociology or across multiple disciplines (e.g., albaili, 1997; calder, 2002; jacobs, 2002; johnson, 1994; laurillard, 1979; light, 2001; nelson, 1998; van etten, freebern, and pressley, 1997; yaworski, weber, and ibrahim, 2000). other research has analyzed quantitative data from questionnaires or institutional data sets about students’ academic views and behaviors (e.g., dietz, 2002; entwistle and tait, 1990; jacobs, 2002; nist, et. al, 1991; paulsen and feldman, 1999; prosser, walker, and millar, 1996; vermetten, lodewijks and vermunt, 1999). questionnaires have also been used to assess the role of demographic variables, academic background, interest, or motivation in disciplinary learning (eckstein, schoenike, and delaney, 1995; meeker, fox, and whitley, 1994; neuman, 1989; paulsen and gentry, 1995; szafran, 1986). only four of these studies are in the discipline of sociology and the focus of all of these was on introductory level students, not specifically majors. in addition, quantitative measures were used in all four studies. over fifteen years ago, szafran (1986) studied factors influencing prior knowledge and grade in the introductory course. his research showed that year in school, gpa, and parents’ education all significantly related to course pretest score, and gpa and pretest score were significantly related to course grade. a few years later, neuman (1989) extended szafran’s work. neuman writes “this study confirms szafran’s finding that pretest scores and gpa predict posttest (final exam) scores with no direct effects from demographic, family background, or prior course work variables. both studies found few effects on pretest score, course grade, or learning from gender, high school sociology courses, or age” (p. 25). neuman also looked at pre-post test score differences (amount learned in the course) and reports that “students learn more if they enter the course knowing less, have a higher gpa, and studied a foreign language” (p. 25). more recently, in a study on developing the sociological imagination by catholic and non-catholic students at a private, catholic institution, eckstein, schoenike, and delaney (1995) found significant relationships between some student demographic variables and successful development of the sociological imagination. more specifically, non-catholic students and students from less privileged backgrounds (measured by both social class and income) were more likely to successfully develop the sociological imagination. finally, dietz (2002) defined success in the large introductory sociology course as total points earned in the course. factors significantly and positively related to total points included attendance and reading the required materials. factors unrelated to success included self-reported study time and use of virtual mckinney, k. journal of scholarship of teaching and learning, vol. 4, no. 1, may, 2004. 15 learning tools. interestingly, study group participation was negatively related to total points earned. in summary, the extant theories on learning in higher education emphasize, to varying degrees, biological mechanisms, individual development and learning preferences, the role of interpersonal variables, and particular experiences or contexts as related to learning. past empirical work has used a wide range of research methodologies. learning was defined and measured in many ways in this past research including test score improvement, course grade or points earned, student perceptions, and understanding of the sociological imagination. this work supports the idea that more and less academically successful students do vary somewhat in terms of their study and learning behaviors. furthermore, some demographic and academic background variables are related to learning. finally, the efficacy of various study strategies, even for the same students, appears to be context specific. in the study i report here, nine honors sociology majors, in a group interview, discuss the factors and strategies that impact their learning of sociology. ii. methodology. i obtained the focus group by soliciting volunteers (using fliers and an announcement by asa staff) from the group of 33 honors undergraduate sociology majors attending the 2003 american sociological association annual meetings in atlanta. nine of these students (27%) volunteered and met for over two hours to discuss the questions. the focus group consisted of eight females and one male; all appeared to be caucasian. of the nine, seven were traditional and two were nontraditional students. they were all juniors or seniors. the nine students were from diverse types of schools from around the nation. i make the assumption here that these nine students are examples of successful learners of my discipline. participation was voluntary and confidential, and verbal informed consent was obtained. as an incentive, i provided the students with food. for a variety of reasons (nine voices, semipublic place, greater confidentiality), i chose not to audiotape the focus group. i took detailed notes, using abbreviations, on the student responses to the focus group questions. the following questions guided the conversation of the focus group. the students each conceptualized learning in their own way. 1. why did you become interested in sociology? how did you get into the sociology major? 2. tell me about the strategies you have used that you believe have helped you learn sociology, understand the sociological imagination, and so on. 3. we often learn things outside of class. what types of out-of-class learning experiences have been helpful in learning sociology? 4. think about an example of a difficult moment in learning sociology. how did you manage to get beyond this moment of difficulty? 5. is there anything else you would like to share on how you learn sociology? immediately after leaving the focus group, i reviewed my notes and added clarification and details. the notes were then typed up question by question yielding about four pages of notes and brief notations when something was repeated multiple times. i analyzed the responses on a question-by-question basis looking for similar or common phrases and ideas in response to each question. my goal was to find themes for each question that had been repeated or supported by several members of the focus group. mckinney, k. journal of scholarship of teaching and learning, vol. 4, no. 1, may, 2004. 16 iii. results. in response to the question about what drew them to the field of sociology and how they became a sociology major, students often told brief life histories about what led them to the major of sociology. as part of this history, they sometimes noted their place in the social structure (e.g., from a single parent family or parents with low education levels or older nontraditional status) and/or critical experiences (e.g., a family member with a social problem, no criminal justice major at their school, taking a great sociology course). from this discussion, three themes emerged. several of the students stressed that the discipline “fit them,” it was “who they were,” it contained the material or areas in which they, personally, were interested, and it was relevant to who they were and to their lives. second, the students frequently mentioned the critical influence of a particular person such as an “inspiring teacher,” “good professor,” and involvement with a particular faculty member. finally, key positive out-or-class experiences were also noted as pathways into the major, including study abroad, internships, and research opportunities. these themes related to selecting the major, then, included characteristics about the students themselves, interactions or relationships with important others, or positive out-ofclass learning experiences. many ideas surfaced when i asked about their study and learning strategies. the themes i pulled from the data were making connections, the role of special others, talking with others about the material, diversity in pedagogy, and active construction of knowledge. students repeatedly talked about the need to make connections-connections between class and text, between the abstract and the concrete, between the material and their lives, between the teacher’s style and their own, and between the teacher’s work and their own work. connections involving relevance and application were seen as very important. a particular type of connection made up a second theme, the role of or connections to others. as in response to the first question, students mentioned the importance of faculty members, and of caring and enthusiastic faculty and staff for their learning. a third theme was also interpersonal. several students stressed that the best way to learn was to talk with others about the material or to try to explain the ideas to another person, verbally or in writing. they were clear that this “other” did not have to be a faculty member but could also be a friend, a classmate, a roommate, or a relative. students also noted, in a fourth theme, the helpfulness of being exposed to diverse pedagogies including discussions, good lectures, seminar formats, collaborative work, reading assignments, writing assignments, and multiple teaching styles. they stated this was important to keep their interest and to appeal to students’ different ways of learning. they argued that there was probably not one best way to learn sociology. thus, they had some explicit knowledge and understanding of the concept of learning styles. finally, there were responses that might best be labeled as active and constructivist, as students discussed the need to think critically, reflect, dialogue, question, write, summarize, and create their own knowledge. these themes indicate to me that the meaning of learning, for these students, goes well beyond memorization of concepts or surface learning. rather, it is deep learning, learning that is integrated, applied, and long-term that they were discussing. the importance of out-of-class learning was also noted. students acknowledged that much learning takes place out of class. they pointed to the importance of connections between class and books, class and internships, and class and watching the news. they also acknowledged the benefits of on-campus speakers and sociology club or alpha kappa delta but argued that these latter types of experiences may only help students who “already get it.” they mckinney, k. journal of scholarship of teaching and learning, vol. 4, no. 1, may, 2004. 17 seemed to believe that students with either a predisposition for, or hard work leading to, the understanding of sociology would seek out and benefit from these experiences but that other students would not—a person by situation—explanation. finally, they noted the value of independent work with faculty members, especially on research projects. in response to the question, “think about an example of a difficult moment in learning sociology. how did you manage to get beyond this moment of difficulty?” the students raised three ideas. first, they talked about persistence and “stepping away” from something tough then coming back to it later. in addition, they mentioned asking questions, getting feedback, and talking with someone else about the difficult material. finally, they noted that the level of cognitive and emotional development of the individual learner might also be a factor in understanding difficult ideas or skills. thus, their three strategies included two they could manage—persistence and getting assistance—and one they had limited control over, level of development. the last question i asked was whether there was anything else they wanted to share about how they learn sociology. they repeated some ideas mentioned earlier in the group conversation including making connections, critical thinking, and personal fit to the discipline. two new ideas also emerged. these were, first, the importance of attending class. second, they discussed the need to sometimes “play the game” with faculty, to occasionally adjust to faculty styles and demands. though several of the students agreed with this, one student was adamant that she never played such games. i found it striking that, even in this discussion, these students never raised the issue of grades. for them, this discussion was about learning, not necessarily about grades. this may be an artifact of the fact that these were all honors students earning high grades. iv. discussion. these honors students pointed to ideas about learning that were remarkably similar to the main components of models or theories about learning espoused in the higher education literature including the importance of experiential and active learning, the role of developmental factors, the constructivist nature of knowledge, the need to make connections or have integrated learning, and the importance of interpersonal relationships (e.g., light, 2001; baxter-magolda, 1999; pintrich, 1995). furthermore, many of the ideas expressed by the students support the widely cited seven principles of good practice in undergraduate education (chickering and gamson, 1987) including student-faculty contact, cooperation among students, active learning, prompt feedback from others, and respect for diverse talents and learning styles. with few exceptions, the responses of these students point to their ability to acknowledge their role in learning and to make internal attributions for their success in learning. similar to some of the previous studies, these students pointed to the importance of certain academic or study behaviors including attending class, writing, reading, and reflecting. though some previous research demonstrated relationships between demographic variables and success, these students did not explicitly discuss the role of their own background variables in their learning but, perhaps, their sense that they just “fit” the discipline or sociology “was who they were” is connected to background characteristics or past experiences. the strongest theme in this conversation was “connections.” this concept, and synonyms such as “relationships” and “links,” was frequently mentioned by most of the focus group members as key to their learning of sociology. clearly, learning opportunities that help students mckinney, k. journal of scholarship of teaching and learning, vol. 4, no. 1, may, 2004. 18 integrate their learning across courses, people, settings, and ideas is critical. perhaps connected and integrated learning experiences also increase time on task and level of challenge, two other best practices in undergraduate education (chickering and gamson, 1987). the connections noted between new material or course material and current/past experience fit with theory and empirical work on placing new learning in the context of students’ existing knowledge (e.g., baxter magolda, 1999; kegan, 1994; king and kitchener, 1994). one form of connections that came up repeatedly was connections to others. these students, though seemingly very secure, independent, and self-confident, strongly valued personally and academically meaningful relationships with faculty members and peers. i found it interesting that the students did not talk a great deal about the specific processes by which these connections, relationships, active construction, and so on were accomplished. students provided some examples of process including discussing course material with a mother or using their talent in the visual arts to understand the material. most of the time, however, there seemed to be an assumption that the specific processes were self-evident. perhaps, for these successful learners, doing these types and ways of learning is “easy” and, therefore, not as explicit or not something they make evident even to themselves. the questions i posed to the focus group asked them to reflect on their personal experiences. i was curious, however, to what extent they would utilize their sociological imagination in responding to these questions. they did so but to a limited degree. they noted the relationship between some social characteristics and their choice of sociology as a major. they recognized that context is important for learning. they did not, however, fit their learning into the broader institutional or societal framework. they rarely used language that might be categorized as reflecting any major sociological paradigms. v. conclusion. faculty, staff, and students can work with these themes in an effort to recruit strong majors and improve the learning of sociology majors. some may find the students’ ideas to be exactly what they expected, but it is important to confirm our expectations and to hear ideas in the students’ own voices. given the fit of these sociology students’ ideas with prior research and theory on learning in higher education, faculty and staff working in other disciplines should also consider ways to apply these findings with their majors. for example, to enhance various types of connections, strong emphasis on high quality teaching and student oriented instructors in introductory and lower-level classes is important. additional ways, live and virtual, must be created and supported to increase the quantity and quality of student-faculty and student-student contact. more and better faculty-student mentoring programs should be established. departments can also make an effort to provide, encourage participation in, and reward meaningful out-of-class learning experiences (e.g., mckinney, et. al, 2004) as well as to help students make connections between these experiences and material or skills from the discipline. in course design, instructors must think about the nature of the readings, assignments, and evaluation tasks both in and out of class, and how these can be altered to help students make the types of connections these students found so important. faculty members should also provide additional opportunities and encouragement for self-reflection, analysis, and collaborative work. appropriate faculty development to assist faculty in creating and implementing diverse and active teaching-learning activities, assignments, and contexts as well as in supporting increased student choice and responsibility is also mckinney, k. journal of scholarship of teaching and learning, vol. 4, no. 1, may, 2004. 19 important. faculty will need to be recognized for these efforts, especially in disciplines or institutions where such efforts are less common or not highly valued. on the other hand, one factor related to becoming a major and “getting it,” and mentioned by many of the students, was the “good fit” between who they are and what they believe as an individual, and the content and ideas of the discipline. this person-discipline fit will be a factor much more difficult to control or affect in order to recruit strong majors or enhance learning. perhaps convincing academically strong, creative students that the discipline of sociology is a good match to who they are, their life, and their goals is a strategy to consider and develop. faculty members in other disciplines may wish to consider the extent to which there is a similar fit between some characteristic of students and the choice of their discipline as a major, and the implications of such a fit for curriculum and pedagogy. those in other disciplines should also consider replicating this or similar work, listening to what their majors have to say about learning in their discipline, and attempting to assess the usefulness of these ideas for learning in their discipline. a critical question for those in other disciplines is whether or to what extent the learning strategies used by the successful sociology students are discipline-specific, general, or a bit of both. the students in this study indicated that there was no one best way to learn sociology, yet they expressed a great deal of consensus on learning strategies that worked for them in their major. i failed to ask these students, however, about how they learn in their classes outside of sociology. what do students in other fields say? this study includes the data from only one focus group or group interview. in addition, these students were exceptional in a number of ways. success was measured only by the fact that these were honors students. this must be kept in mind when considering what they had to say about their learning. we have only just begun to answer the question, how do sociology majors learn sociology. there is little knowledge available and much knowledge needed. thus, there are many areas for future research for sociologists that would also be transferable to those in other disciplines. questions include, to what extent do these results generalize to other students or to research using other methods? do unsuccessful or less successful learners of sociology report similar ideas about how they learn the discipline? explicit comparisons between more and less successful learners of sociology should be made using a variety of measures of success. furthermore, do successful learners in sociology share some common predisposition such as personality, interest, or learning style? are there interactions between learning style or motivation level and the learning behaviors that contribute to successful learning? how do sociology students conceptualize learning in our discipline? this work focused on the views of juniors and seniors. an important line of research would be to look at the development of learning and learning strategies over the course of the major as has been done in some other fields. do students earlier in the major rely more on surface learning strategies relative to those later in the major, for example? we need to consider the notion of connections in much greater detail. is the importance of this factor shared by students in other disciplines? what, more specifically, is important about these connections? further research is being conducted on my campus to triangulate and extend the results presented here. i urge others in sociology and in related disciplines to gather and share additional, relevant data. mckinney, k. journal of scholarship of teaching and learning, vol. 4, no. 1, may, 2004. 20 references albaili, mohamed a. 1997. “differences among low-, averageand high-achieving college students on learning and study strategies.” educational psychology 17(1-2 march june):171177 baxter-magolda, marcia. 1999. creating contexts for learning and self-authorship. nashville: vanderbilt press. calder, lendol. 2002. “looking for learning in the history survey.” perspectives. march. chi, michelene t.h., paul j. feltovich and robert glaser. 1981. “categorization and representation of physics problems by experts and novices.” cognitive science 5:121-152. chickering, a. w. and gamson, z.f. 1987. “principles for good practice in undergraduate education.” the wingspread journal, june. dietz, t. 2002. “predictors of success in large enrollment introductory courses…” teaching sociology 30:80-88. eckstein, rick, rebecca schoenike and kevin delaney. 1995. “the voice of sociology: obstacles to teaching and learning the sociological imagination.” teaching sociology 23:353363. entwistle, noel and hilary tait. 1990. “approaches to learning, evaluations of teaching, and preferences for contrasting academic environments.” higher education 19:169-194. hartlep, karen l. and g. alfred forsyth. 2000. “the effect of self-reference on learning and retention.” teaching of psychology 27:269-271. jacobs, dennis. 2002. “an alternative approach to general chemistry.” http://www.carnegiefoundation.org/castl/djacobs/index2.htm. johnson, genevieve. m. 1994. “undergraduate student attrition: a comparison of the characteristics of students who withdraw and students who persist.” the alberta journal of educational research xl:337-353. kegan, robert. 1994. in over our heads: the mental demands of modern life. cambridge, mass.: harvard university press. king, patricia m. and karen s. kitchener. 1994. developing reflective judgement: understanding and promoting intellectual growth and critical thinking in adolescents and adults. san francisco: jossey-bass. kreiner, davis s. 1997. “guided notes and interactive methods for teaching with videotapes.” teaching of psychology 24:183-185. mckinney, k. journal of scholarship of teaching and learning, vol. 4, no. 1, may, 2004. 21 light, richard. j. 2001. making the most of college: students speak their minds. cambridge, ma: harvard. mckinney, kathleen, tchernykh, maria, fite, kerry, and malak, janice. 2004. “beyond the classroom: out-of-class learning in sociology.” teaching sociology 32: 43-60. meeker, frank, daniel fox, and bernard e. whitley, jr. 1994. “predictors of academic success in the undergraduate psychology major.” teaching of psychology 21(4 december): 238-241. nelson, robert. 1998. “using a student performance framework to analyze success and failure.” journal of college reading and learning 29:82-89. neuman, w. lawrence. 1989. “which students learn the most, and why?: a replication and extension of the szafran pretest study.” teaching sociology 17:19-27. nist, sherrie l., michelle l. simpson, stephen olejnik, and donna l. mealey. 1991. “the relation between self-selected study processes and test performance.” american educational research journal 28:849-874. paulsen, michael b. and kenneth a. feldman. 1999. “epistemological beliefs and selfregulated learning. journal of staff, program, and organizational development 16:83-91. paulsen, michael b. and james a. gentry. 1995. “motivation, learning strategies, and academic performance: a study of the college finance classroom.” financial practice and education spring/summer:78-89. pintrich, paul r. 1995. (ed.) understanding self-regulated learning 63:fall. san francisco: jossey-bass. prosser, michael, paul walker and rosemary millar. 1996. “differences in students’ perceptions of learning physics.” physics education 31:43-48. szafran, robert f. 1986. “what do introductory sociology students know and when do they know it?: the results of pretesting students.” teaching sociology 14:217-223. van etten, shawn, geoffrey freebern, and michael pressley. 1997. “college students’ beliefs about exam preparation.” contemporary educational psychology 22:192-212. vermetten, yvonne j., hans g. lodewijks, and jan d. vermunt. 1999. “consistency and variability of learning strategies in different university courses.” higher education 37:1-21. watson, david l., deborah k. hagihara, and alyssa l. tenney. 1999. “skill-building exercises and generalizing psychological concepts to daily life.” teaching of psychology 26:193-195. mckinney, k. journal of scholarship of teaching and learning, vol. 4, no. 1, may, 2004. 22 yaworski, joann, rose-marie weber, and nabil ibrahim. 2000. “what makes students succeed or fail?: the voices of developmental college students.” journal of college reading and learning 30:195-221. journal of the scholarship of teaching and learning, vol. 8, no.1, february 2008, pp. 61 71. investigating whether contacting absent students increases course success thomas d. stucky1 abstract: studies suggest that student attendance in college classes increases course success. yet, surprisingly few studies have examined strategies to increase student attendance. the goal of the current study is to consider whether contacting consistently absent students increases success in an undergraduate research methods course. results of this classroom action study suggest that students view contacts positively and a majority stated that they were more likely to attend class following the contact. in regression analyses, however, net of other factors such as prior grade point average, contacts did not predict final grade percentage and d/f rate comparisons to a prior semester without contacts showed modest but not statistically significant improvements. implications are discussed. keywords: attendance patterns; instructor contact; course success it is commonly assumed that higher student attendance increases the likelihood that students will succeed in the course, and some research supports this notion. one common strategy for increasing attendance is a mandatory attendance policy and a few studies suggest that such policies increase attendance. another strategy is for the instructor to systematically contact absent students in an effort to bolster students’ feelings of connectedness and increase motivation to attend class. the goal of the current classroom action study is to examine whether contacting consistently absent students increases success in an undergraduate research methods course.2 results suggest mixed support for the effectiveness of contacting absent students. survey results suggest that students generally view the instructor contacts positively and a majority of contacted students stated that the contact made them more likely to come to class. regression analyses, however, did not suggest any independent impact of the contacts on final course grade, net of other factors such as prior grade point average (gpa). a comparison of d/f rates with a prior section without systematic contacts showed small but not statistically significant improvements in course success compared to a prior semester. the implications of the current study for future research and classroom attendance policies are discussed. i. a classroom challenge: student attendance. conventional wisdom among college instructors and administrators is that student attendance and course success are related, and some research supports this notion. for example, in a study of 300 undergraduates in a large midwestern university general education biology course, gump (2005) found that student absences were negatively related to course grades. 1 school of public and environmental affairs, indiana university purdue university at indianapolis, 801 west michigan street, bs 4069, indianapolis, indiana 46202. (317) 274-3462, tstucky@iupui.edu. 2 this study was undertaken as part of a statewide initiative called the indiana project on academic success to boost student retention rates. i focus in this study on course success as a means to support college retention. stucky journal of the scholarship of teaching and learning, vol. 8, no.1, february 2008. 62 similarly, brown, graham, money, and rakoczy (1999) found that higher numbers of absences were associated with lower grades in a study of nine nursing courses. such findings have been reported across a variety of undergraduate curriculum areas such as sociology (e.g. day, 1994), psychology (levine, 1992), business law (e.g. davenport, 1990), and physiology (hammen and kelland, 1994), but are not unequivocal (see for example st. clair, 1999 who views the evidence as mixed). if increasing student attendance boosts course success, then it seems reasonable to examine strategies to increase attendance rates. one strategy to increase student attendance is a mandatory attendance policy. although not all agree that mandatory attendance policies are a good idea (see hyde and flournoy, 1986; st.clair, 1999), some evidence suggests that such policies increase student attendance. for example, levine (1992) reports that student attendance was greater in courses where attendance was explicitly required compared to those where it was not. in paraphrasing the old “carrot and stick” approach to generating desirable behaviors (or deterring undesirable ones), one might think of a mandatory attendance policy as “the stick” because it typically involves (explicitly or implicitly) penalizing students for lack of attendance. one might also focus on “carrot” strategies to increase attendance such as extra credit points. yet, mandatory attendance or extra credit policies are blanket policies that apply to the entire class. a more targeted approach that zeroes in on specific students might be to contact students exhibiting frequent absenteeism. to date, only one study has considered whether contacting absent students can increase retention and success. richie and hargrove (2005) found that telephone contacts of absent students in freshman english classes were associated with fewer absences, higher course grades, and ultimately higher college retention in the following year. a number of studies suggest that student motivation enhances course success (see pintrich, 1994 for a discussion). although a number of factors influence student motivation, one factor may be the degree to which the student perceives that the instructor is concerned about them. i argue that contacting absent students could increase student motivation to attend class by making the student cognizant of the fact that the instructor cares enough about the student as an individual to reach out to them in cases of consistent absence. this is especially likely for a subset of the student population in many undergraduate classes, who exhibit spotty attendance and appear to have minimal motivation to attend class or connectedness to the class. typically these students miss the first day of class, attend infrequently, or never attend class prior to the first examination. this group often does poorly on the first exam and later withdraws or failed the course, usually due to even less consistent attendance after the first exam. i refer to these students as “half in/half outs” because they seem to be somewhat interested in taking part in the course and/or college more generally but also have their feet halfway out the door for a variety of reasons (often the pressures of balancing work, family and school demands). frequently poor initial examination scores push them the rest of the way out through failure or withdrawal. over the course of several semesters, anecdotal evidence from students suggested that contacting consistently absent students seemed to boost course attendance. for example, one student, whom i contacted after several absences, expressed to me that i was the first professor that had ever noticed her absence from class and cared enough to follow up. other students expressed similar positive responses to my informal attempts to contact them. this anecdotal evidence led me to hypothesize that a formal policy of contacting absent students would boost attendance and course success. thus, the research question in the current study is: does contacting consistently absent students increase student success? stucky journal of the scholarship of teaching and learning, vol. 8, no.1, february 2008. 63 ii. data and methods. although the ideal research design for determining cause and effect incorporates random assignment to treatment and control groups, such a design was impossible in the current study because of the nature of the courses i teach within the curriculum. specifically, only one section is typically offered per semester and students can take courses in any sequence. it was also not possible to consider longitudinal designs because students are only in the course for a single semester. randomization within the class was also not possible because any effects of increased attention to attendance could not be assumed to be restricted to the experimental portion of the class. therefore, i chose a two-pronged approach to examining the research question. first, i chose a quasi-experimental design that compared the overall grade information in the treatment class to a comparison class from a prior semester’s section of the same course (both sections were 200-level undergraduate research methods courses). in both semesters student attendance was explicitly mandatory and tracked through attendance sheets. in addition, to provide a positive incentive for students to attend class, in both semesters, i awarded 10 points of extra credit for students who missed 0 or 1 class periods, and 5 points of extra credit to students who missed 2 or 3 class periods. thus, the only variation regarding attendance from the previous semester was the instructor systematically contacting absent students. for students that agreed to participate in the study (47 of the 56 who began the semester), i attempted to contact them via email, and positively reinforce the desirability of class attendance if they were absent for two or more consecutive days, or exhibited a pattern of inconsistent attendance such as missing every other class or two or three out of five. if participating students continued to be absent or did not respond to email, phone contacts were attempted. records of all contacts and attempts to contact students were kept. the second prong of the evaluation process was to examine students’ subjective views of the contacts. at the end of the semester a confidential survey was distributed to all study participants (see appendix 1). students were informed that their participation was voluntary and would not affect their course grade. the survey examined reasons for absences, and for those students that were contacted via email or phone, how they viewed instructor contacts, including whether the contacts influenced future attendance. the survey also asked general questions regarding student views of the relationship between course attendance and course success. survey responses were linked to student attendance records and course grades. iii. results. of the 56 students enrolled at the beginning of the semester, 43 students received letter grades, 9 withdrew, and 4 received failing grades for non-attendance. in all, 19 contacts for absences were attempted or completed regarding 15 students. a few did not respond via email and therefore required follow-up phone contacts. three students received more than one contact by the instructor for consistent absences and two contacted students ultimately withdrew from the course and another failed because they stopped attending. of the 43 students that remained in the course when the survey was distributed, 33 completed surveys, representing a 77% completion rate.3 four students failed to report their names and five students failed to report their prior gpa. for those cases, mean substitution was 3 the response rate for the survey appears to have been mainly a function of the number of students in class on the day the survey was administered. stucky journal of the scholarship of teaching and learning, vol. 8, no.1, february 2008. 64 employed to eliminate missing values. as shown in table 1, students in the sample were absent approximately 2.9 days on average, were taking about 12.9 hours of classes, on average, and reported a mean prior gpa of 2.95. table 1. descriptive statistics. variable n mean std. dev minimum maximum grade pct. 29 78.7 6.35 67.4 91.3 days missed 29 2.93 3.16 0 10 contacts 29 0.15 0.36 0 1 hours 33 12.9 3.2 6 18 gpa 28 2.95 0.52 1.94 3.89 get points 29 0.62 0.49 0 1 looking at the basic question of whether attendance influenced course grade, table 2 compares the final grade percentage of those missing 2 or fewer classes compared to those missing 3 or more. the final grade percentage for those missing 2 or fewer classes was significantly higher (p < 0.005 in a two-tailed t-test) at 79.2%, compared to 73.4% for those missing 3 or more classes. thus, it appears that consistent attendance was associated with approximately a one-half letter grade higher course grade.4 thus, the evidence in the current study supports prior findings that consistent attendance is associated with greater course success. table 2. course final grade by days missed. classes missed grade % 0-2 79.62* 3+ 73.36 * p < 0.005 in a two-tailed t-test a. regression analyses predicting course success to examine whether contacting students influenced course success, i conducted regression analyses to isolate the impact of contacts, net of other factors likely to influence course success. table 3 shows the results of four regression equations predicting course grade percentage for students completing the survey (0-100%).5 equation 1 shows that, controlling for prior grade point average (gpa), the number of hours currently being taken by the student, and instructor contacts, the number of course sessions missed was significantly negatively associated with the course grade (p < 0.05). equation 2 substitutes whether the student attended frequently enough to earn extra credit or not. missing 2 or fewer classes was associated with significantly higher final grades in the course. this further confirms prior research that higher levels of attendance are associated with greater levels of course success. 4 some of this difference in the final grade was no doubt due to earning extra credit points for frequent attendance. yet, this cannot explain all of the more than one-half letter grade difference (6.26%) between the groups because the maximum extra credit points that could be awarded only constituted 2% of the final course grade. 5 in all 4 equations presented in table 3, an overall f-test was significant (p < .01). stucky journal of the scholarship of teaching and learning, vol. 8, no.1, february 2008. 65 table 3. regression results of course grade percent on attendance, instructor contact and prior gpa (n = 33). variable eq. (1) eq.(2) eq.(3) eq.(4) constant 57.579*** (5.784) 54.411*** (5.644) 55.004*** (5.796) 53.881*** (5.511) prior gpa 7.334*** (1.700) 6.931*** (1.684) 7.745*** (1.663) 7.438*** (1.672) current hours enrolled 0.089 (0.257) 0.094 (0.251) 0.164 (0.253) 0.165 (0.249) absences -0.574* (0.277) -0.181 (0.352) extra credit for attendance 4.206* (1.752) 2.226 (2.114) contacted by instructor -4.990 (2.907) -4.288 (2.697) adjusted r2 0.407 0.485 0.445 0.461 * p < 0.05, ** p < 0.01, *** p < 0.001, two tailed significance test turning to the central question of the current study, table 3 does not suggest that systematically contacting students was associated with higher student final grades net of the other factors in equations 3 and 4. there were no significant differences between the final grades of contacted students versus those that were not contacted.6 in fact, the most important predictor of final grade percentage was prior gpa. this perhaps should not come as a surprise. presumably, when it comes to educational outcomes, past performance is indicative of future performance. yet, it is important to keep the small n of this study in mind when interpreting these results.7 b. comparing attendance and course retention and success. in terms of course retention and success, explicitly having a policy of systematically contacting absent students, does not appear to have had a substantial impact. table 4 shows the d/f/ withdrawal (d/f/w) rates for two sections of the course—the treatment section (fall 2005) and the comparison section from the previous semester (spring 2005).8 overall the d/f rate was 5% lower (17% v. 22%) in the current section compared to the previous semester. yet, this was not a statistically significant difference (perhaps due to the relatively small class sizes involved). 6 the coefficients for attendance drop to non-significance with the inclusion of the contact variable in equations 3 and 4. variance inflation factors (vif) were examined to determine whether the inclusion of attendance and contact variables in the same equation created multi-collinearity problems. vifs were below 2.0 in all equations suggesting that multi-collinearity was not a major concern. 7 alternative specifications without mean substitution produced substantively similar results to those presented in table 3. 8 ideally, it would have been beneficial to compare attendance rates between the two classes but attendance records for the prior semester were unavailable at the time of the study. stucky journal of the scholarship of teaching and learning, vol. 8, no.1, february 2008. 66 table 4. dfw rate comparison for j202 sections. n % d % f % w % dfw % df a spring 2005 48 10.40% 8.30% 14.60% 33.30% 22.0% fall 2005 56 7.10% 7.10% 16.10% 30.40% 17.0%* a df rate excludes those who withdrew and although the d/f/w rates was 2.9% lower in the treatment semester (30.4% v.33.3%), this difference was also not statistically significant. ironically, there was a slight (though not statistically significant) increase (1.5%), in the withdrawal rate. this might actually indicate that student contact policy was working because the instructor counseled a few students who had never attended or only attended the course a few times to withdraw from the class or they would likely receive a failing grade. in these cases, it was determined that discretion might be the better part of valor and that students would be better served by withdrawal. thus, the results of the current study do not provide strong evidence of any impact of the contacts on course success either in terms of contacted students’ gpa or in terms of d/f/w rates compared to a semester without such contacts. c. student perceptions of contact. the student survey offers somewhat more encouraging evidence on the value of instructor contacts. appendix a lists the attendance survey questions and student response patterns. based on student responses, it is clear that students were aware of the attendance policies and the emphasis the instructor placed on the importance of attendance. for example, all 33 survey respondents acknowledged that the instructor had discussed the attendance policy early in the course. in terms of patterns of attendance, 22 of 33 respondents self-reported missing 2-5 days of class (out of approximately 30 course meetings). students’ stated reasons for missing class varied widely. most respondents reported missing class due to illness, emergency (self or family), or work obligations.9 the majority of respondents (70%) agreed that their patterns of attendance influenced their grade in the course. in addition, 30% reported attending the class more than others on campus, although it is not clear whether the increased attendance was due to the course content (perhaps unlikely given that the course is a required research methods course), the increased contact by the instructor, or some other combination of factors.10 in terms of student views of instructor contacts regarding absences, nine reported being contacted by the instructor, which was somewhat lower than the actual number of students contacted (n=15), (likely because those contacted by the instructor missed classes frequently and so were less likely to be in class on the day the survey was administered). five of the nine reported viewing the contact positively, while the remaining four reported being neutral regarding it. similarly, five of the nine reported being more likely to attend class following the contact than before, whereas the remaining 4 were about as likely to attend as before. additional 9 students may have had a disincentive to accurately report reasons for absences that they believed would be seen as less justified because the surveys were confidential rather than anonymous. still, the stated reasons for missing class match the statements made by many students outside of the survey context, in informal interactions with the instructor. 10 in retrospect, exploring students’ reasons for relative attendance levels in comparison to other courses would have been a valuable follow-up question to ask. stucky journal of the scholarship of teaching and learning, vol. 8, no.1, february 2008. 67 evidence from student email responses to instructor contacts suggests that, at least some of those contacted viewed it as an expression of the instructor’s concern for their well-being. thus, the evidence suggests that the contacts by the instructor are viewed positively by the students (or at least not negatively), and that the contacts caused at least some of the students to increase their attendance.11 students also reported believing that course attendance and success are related. overall, the results suggest that students do not resent being contacted, and in some cases the contacts increased the students’ reported likelihood of coming to class. thus, the survey results paint a somewhat more positive view of the value of contacting students than the quantitative analyses of final grade percentage and comparisons of d/f/w rates with a prior semester section of the same course. iv. discussion and conclusions. this classroom action research was undertaken to assess whether contacting consistently absent students (in the presence of mandatory attendance policies and extra credit for attendance) would increase course success. the current study confirmed the results of several prior studies that higher levels of attendance were associated with greater course success, especially for those with highly consistent attendance (2 or fewer absences), who scored more than one-half letter grade higher on average than those who attended less frequently. regression results, however, did not suggest that contacting students had an independent impact on course final grade, net of other factors such as prior gpa. nor was there much evidence of statistically significant reductions in d/f or d/f/w rates compared to a prior semester without systematic instructor contacts of absent students. in fact, the withdrawal rate actually was slightly higher (16.1% v. 14.6%) in the treatment class relative to the prior section. yet, increases in the withdrawal rate could indicate that the contact policy was working because the instructor was able to persuade some consistently absent students that withdrawal was better than failure. from an institutional standpoint withdrawal may not be a desirable outcome but from the student’s perspective it is likely to be a much more desirable outcome than a failing grade on the transcript. survey results were more supportive. respondents generally did not seem to resent the contacts and a majority (5 of 9) reported being more likely to attend class following the contact (the remaining 4 were about as likely). thus, students’ appeared to perceive the contacts positively and a majority stated that the contacts influenced their attendance. several limitations of the current study must be considered. first, the small number of students and the corresponding small number of instructor contacts in the study made isolating the independent effects of contacts on final grade difficult. this is a serious limitation that precludes firm conclusions regarding the effects (or lack of effects) of the contacts and suggests that additional studies are necessary. it is also possible that the addition of a policy of contacting absent students to other attendance policies limited the overall impact on student outcomes compared to alternative specifications with no mandatory attendance policy or extra credit points for attendance. future research could compare which strategy (mandatory attendance, systematic instructor contact, extra credit) is most effective at increasing attendance, course retention, and success. given that only one other published study has specifically examined the effects of instructor contacts on course success, it seems critical to explore these issues further with much 11 it should be noted, however, that these conclusions are based on a small number of responses. a larger number of responses might have produced more negative responses. stucky journal of the scholarship of teaching and learning, vol. 8, no.1, february 2008. 68 larger sections or in subject areas with multiple sections taught by a single instructor in the same semester to minimize other potential influences on course outcomes. another limitation is that students responded to the survey at the end of the course. this has several potential implications. first, students’ perceptions of the contact may have been inaccurate due to the time between the contact and the time the survey was completed (several weeks in some cases). second, the survey was only completed by those who remained in the course. the students who withdrew or failed for non-attendance might have had different reactions to the contacts had they remained to complete the survey. in methodological terms, this is an issue of sample selection. it could be that those who remained to take the survey viewed attendance and contacts differently than those who did not remain in the course. this is an issue that is not easily resolved given the constraints of the current research setting. one possibility would be for universities to routinely conduct exit surveys of withdrawing students or those who fail for non-attendance to determine students’ reasons for withdrawal or non-attendance and their perceptions of the course and college environment. the current research suggests that contacting absent students is no magic bullet. the reasons for student absences vary tremendously and often reflect the conflicting demands in students’ lives. some have children who become ill and cannot find child care. others cannot control their work schedules. telling students that attendance is important does not make their child any less sick or their work schedule any more flexible. thus, the likely impact of attendance policies or increased contact from instructors on student attendance must be placed in this context. students often have a number of competing demands on their time, and regardless of the course policies or content, this reality is unlikely to change. it also appears that good students think attendance is important and consistently do so. prior gpa emerged as the strongest predictor of the final grade percentage. perhaps this is to be expected. students who do well in prior courses can be reasonably assumed to possess the skill and motivation to do well in the current course. presumably part of the skill and motivation that leads to past and present student success is reflected in higher attendance in the course. thus, it could be that students attend because they are conscientious, motivated students. still, contacting absent students serves worthwhile purposes. by contacting absent students, especially early in the course, the instructor may be able to head off any problems before they become large enough to preclude course success, or allow the student to withdraw before failure becomes inevitable. students in the current study (at least those remaining in the course to be surveyed at the end) appeared to view these contacts as an expression of concern from the instructor. the additional work is minimal and, regardless of whether there are substantial effects on attendance or success, the instructor has attempted to maintain a relationship with students who are not consistently in the classroom. the policy of contacting absent students may also represent an acceptable middle-ground for instructors who are uncomfortable with mandatory attendance policies. the larger point, however, may be that when dealing with attendance or other classroom issues—try something. this study highlighted for me that students often miss class for very legitimate reasons that they perceive to be beyond their control. yet, the attention that instructors pay to attendance creates incentives that maximize the likelihood that students will attend, and contacting absent students enhances student beliefs that the instructor is concerned about their well-being and success. in a broader sense, more important than the success or failure of a particular classroom strategy, is that identification of classroom problems and initiating proactive strategies to improve them is critical to improving teaching. stucky journal of the scholarship of teaching and learning, vol. 8, no.1, february 2008. 69 v. acknowledgements. this research project was undertaken as part of the indiana project on academic success (ipas), a statewide project funded by the lumina foundation. an earlier version of this paper was presented at the e.c. moore symposium at indiana university purdue university at indianapolis, february 2006. i wish to acknowledge the tremendous intellectual support of elizabeth rubens, nancy evans, and scott weeden. all analyses reported and conclusions drawn are the sole responsibility of the author. stucky journal of the scholarship of teaching and learning, vol. 8, no.1, february 2008. 70 appendix 1. attendance survey fall 2005 with results. is this your first class with this instructor? yes ____31__ no __2______ did you attend the first day of class? yes ____32__ no __1______ did the instructor discuss attendance or an attendance policy early in the course? yes ____33__ no __0______ were you aware that extra credit points were available for course attendance? yes ____32__ no __1______ did the extra credit make it more likely that you would come to class? __1__ much more likely __8__ somewhat more likely __5__ slightly more likely __10__no more or less likely __8__ less likely how many class days have you missed this semester? ___2__ 0-1 ___10__2-3 ___12__4-5 __2___ 6-8 __2___ 9+ ___2__ not sure for the days that you missed class, could you describe the reason (check all that apply): ___8__ not feeling well ___2__ other schedule conflict ___7__ family illness ___4__ difficulty with transportation ___4__ family emergency ___12__ other ___6__ work schedule did the instructor contact you regarding your attendance? __8__ yes __25__ no if yes, how many times? 6 people once; 2 people twice how did you view this contact from the instructor? ___1_ very positively ___4_ positively ___4_ neutral ___0_ negatively ___0_ very negatively after the instructor contacted you how likely were you to attend class? ___3_ much more likely ___2_ more likely ___4_ about as likely as before ___0_ somewhat less likely ___0_ much less likely did your class attendance affect your grade in the course? ___7_ strongly agree ___14_agree ___8_ neutral ___1_ disagree ___1_ strongly disagree how did your attendance in this class compare to others you are taking / have taken at iupui? __4__ much more often than others __6__ somewhat more often than others __22__about as often as others __1__ somewhat less often than others __0__ much less often than others did the instructor create a classroom environment that was conducive to learning? __21__strongly agree __9__agree __3__neutral __0__disagree __0__strongly disagree stucky journal of the scholarship of teaching and learning, vol. 8, no.1, february 2008. 71 references brown, b., graham, c., money, s., & rakoczy, m. (1999). absenteeism and grades in a nursing curriculum. michigan community college journal: research & practice, 5(81), 81-84. day, s. (1994). learning in large sociology classes: journals and attendance. teaching sociology, 22(2), 151-165. davenport, w. s. (1990). a study of the relationship between attendance and grades in three business law classes at broome community college (doctoral dissertation, nova university, 2001). gump, s. e. (2005). the cost of cutting class: attendance as predictor of student success. college teaching, 53(1), 21-26. hammen, c. s., & kelland, j. l. (1994). attendance and grades in a human physiology course. advances in physiology education, 267(6), s105-s108. hyde, r. m., & flournoy, d. j. (1986). a case against mandatory lecture attendance. journal of medical education, 61(3), 175-176. levine, j. r. (1992). the effect of different attendance policies on student attendance and achievement. paper presented at the annual meeting of the eastern psychological association, boston, ma. pintrich, p. r. (1994). student motivation in the classroom. in k. w. prichard & r. m. sawyer (eds.), handbook of college teaching: theory and application (pp. 23-34). westport, ct: greenwood press. richie, s. d., & hargrove, d. s. (2005). an analysis of the effectiveness of telephone intervention in reducing absences and improving grades of college freshman. journal of college student retention: research, theory & practice, 6, 395-412. st. clair, k. l. (1999). a case against compulsory class attendance policies in higher education. innovative higher education, 23(3), 171-180. 503 service temporarily unavailable 503 service temporarily unavailable nginx/1.14.1 503 service temporarily unavailable 503 service temporarily unavailable nginx/1.14.1 microsoft word kreberfinal.doc journal of scholarship of teaching and learning, vol. 6, no. 1, august, 2006, pp. 88 – 109. developing the scholarship of teaching through transformative learning carolin kreber1 abstract: following a cognitive-developmental perspective, the scholarship of teaching & learning is understood as a process of knowledge construction whereby knowledge claims are validated through reflection on teaching experience and educational theory. these reflective processes can be documented and peer reviewed. teaching portfolios allow for the documentation of indicators of reflection. indicators can be developed for each of three domains of teaching knowledge: (1) what we consider to be meaningful goals and purposes of higher education; (2) what we know about student learning and development in relation to these goals; and (3) what we know about the teaching and instructional design processes needed to bring about academic learning and development. keywords: learning about teaching, transformative learning, reflection, professionalism and citizenship, scholarship, documentation and peer review i. introduction: the scholarship of teaching as professionalism guided by citizenship “we develop a scholarship of teaching when our work as teachers becomes public, peerreviewed and critiqued. and exchanged with members of our professional communities so they, in turn, can build on our work” (shulman, 2000). far from having remained “an amorphous term, equated more with commitment to teaching than with any concrete, substantive sense of definition or consensus as to how this scholarship can be recognized” (menges & weimer, 1996, p.xii), the scholarship of teaching and learning has gained much clearer contours and recognition in recent years (kreber, 2003). often linked to the notion of professionalism in university teaching, the scholarship of teaching and learning is progressively associated with a form of knowledge about teaching and student learning that can be rationally verified through disciplined inquiry. “professional knowledge” thus construed is knowledge oriented towards “best practices”. while the question of what constitutes “best practices”, ultimately, is a philosophical one, there remains little doubt that we can observe a trend in the educational policy arena to equate the idea of “best practices” increasingly with notions of effectiveness and efficiency. applied to the scholarship of teaching and learning, professional practices (or “best practices”), then, are identified by exploring the question “which teaching innovations produce the best results (i.e., more learning, better/deeper learning, or a closer fit of learning outcomes with those required by the job market, etc)?”. no one would dispute that this is a significant question to delve into; however, it is just one question that the scholarship of teaching is (or should be) concerned with. a second way of exploring university teaching in a scholarly (or if you will professional) way is to turn to its moral and civic purposes. thus construed, the scholarship of teaching (and 1 director of the centre for teaching, learning and assessment, moray house school of education, paterson’s land, university of edinburgh, uk. questions and comments may be emailed to carolin.kreber@ed.ac.uk. kreber, c. journal of scholarship of teaching and learning, vol. 6, no. 1, august, 2006. 89 learning), or professionalism in teaching, is more appropriately associated with the notion of citizenship rather than with “effectiveness or efficiency”. while understanding how students learn and finding ways to optimize their learning, preferably through replicable and publishable forms of scientific inquiry, is clearly important, this alone cannot be the essence of the scholarship of , or professionalism in, teaching. “best (teaching) practices”, therefore, are no more than the means by which to bring about desired educational outcomes. and so a question to be addressed early on in any deliberations on university teaching ought to be “what are the ends that the contemporary university serves through its teaching practices and curriculum?” and, more importantly, “are these the same that it should serve”? are we concerned with training and socializing researchers into our discipline, preparing students for specific jobs or for lifelong learning, facilitating their personal development, promoting their successful participation in a democratic society, or perhaps none, or all, of these? analyses of these latter questions are just as essential aspects of practicing the scholarship of teaching as are explorations of how well certain teaching methods work and how, or how well, students learn (hutchings & shulman, 1999). the scholarship of teaching and learning, or professionalism in teaching, therefore, needs to be conceptualized broadly and integrate the notion of professionalism with the notion of citizenship (see also walker, 2001). by asking (1) what do we consider to be meaningful goals and purposes of higher education, (2) what do we know about student learning and development in relation to these goals, and (3) how can we promote such learning and development (kreber & cranton, 2000; kreber, 1999), the scholarship of teaching and learning could lead to changes that go beyond the development and implementation of instructional innovations but are expressed also in the larger curriculum and co-curriculum (kreber, 2005a) i am stating what is obvious to everyone, of course. certainly goals are important and no one would dispute this. in recent years, many scholars have highlighted the university’s role in promoting moral and civic education (e.g., colby, ehrlich, beaumont, & stephens, 2003; lempert, 1996; piper, 2002; orr, 1993; rhoads, 2000) and numerous higher education policy documents directly speak to the importance of these goals (e.g., dfes, 2003; world conference of higher education, 1998; national panel report, 2002). more over, the focus for the 20032004 cohort of higher education carnegie scholars organized by the carnegie academy for the scholarship of teaching and learning, was on liberal learning. the carnegie foundation also recently initiated the “political engagement project” to address the problem of young people increasingly disengaging from politics. this initiative builds on the earlier work by colby et al. (2003) on moral and civic responsibility. clearly, all these projects demonstrate careful consideration of the goals and outcomes of higher education and a concern with student learning that goes beyond the development of discipline-experts, or training for specific jobs. however, the links between these moral and civic goals and the scholarship of teaching and learning, though possibly assumed by many, has not always been made explicit. even though many have discussed the importance of educational goals and purposes over the years, until recently these discussions occurred largely outside rather than within the discourse on the scholarship of teaching and learning (by which i mean a discourse found in softl specific journals and conferences). i suggest that it is perhaps for this reason that many faculty and administrators associate the scholarship of teaching and learning still primarily with the notion of “best (teaching) practices” rather than a broader notion of professionalism that would integrate the idea of “citizenship”. though the latter is possibly taken for granted by some, it still occupies kreber, c. journal of scholarship of teaching and learning, vol. 6, no. 1, august, 2006. 90 somewhat of a secondary role in conceptions of what the scholarship of teaching and learning is, should, or could be. in this article, then, i suggest that the scholarship of teaching and learning involve (1) careful consideration of educational goals and purposes suitable for addressing the various political, social, cultural, environmental and economic challenges of our times, (2) understanding how students learn and develop toward these and other academic goals, and (3) identifying ways to best facilitate this learning and developmental process. i further contend that the notion of the scholarship of teaching and learning implies that we approach our teaching practice with the same sense of skepticism that guides our research. as researchers, we habitually provide arguments or reasons for our assertions. depending on our discipline or subject area, we engage in the process of hypothesis testing, interpretation or critical analysis routinely. moreover, we recognize that it is important to share with colleagues the evidence we generated for our point of view and invite them to follow, and possibly critique, our lines of argumentation. it has been proposed that we engage in similar processes with respect to our teaching; however, traditional ways of sharing such as conference papers and journal publications are but two of several possibilities. later in this paper i will discuss how and why teaching portfolios are particularly suitable for this purpose. in the remainder of this article i will build on these observations and discuss how the scholarship of teaching and learning may be developed through transformative learning (mezirow, 1991), a process by which faculty construct knowledge about teaching and learning through reflection. in line with the earlier arguments, i suggest that faculty construct knowledge in three different domains. the first domain of knowledge relates to what we consider to be meaningful goals and purposes of higher education (curricular knowledge). the second refers to what we know about student learning and development in relation to these goals (pedagogical knowledge, or perhaps more appropriately referred to as psychological knowledge). the third pertains to what we know about the teaching and instructional design processes needed to bring about student learning and development (instructional knowledge). questions that will be examined in this article include: • how is reflection on teaching and learning valuable? • what role do experience and theory play in reflection on teaching? • what is transformative learning? • are there different kinds or levels of reflection, and if so, are all levels of reflection equally conducive to fostering change and development in higher education teachers? • how is reflection, and transformative learning, linked to the scholarship of teaching? • how can transformative learning on teaching and student learning be demonstrated and reviewed? ii. how is reflection on teaching and learning valuable? for more than two decades researchers have explored the role of reflection in teacher training (e.g., hatton & smith, 1995; calderhead, 1989; zeichner et al., 1987) and more recently, it has become a buzzword also in higher education. time and again faculty are reminded of the importance of reflecting on their teaching (e.g., brookfield, 1991, 1995; cranton, 1998; ramsden, 1992; schön, 1995), though the process of reflection itself remains poorly understood kreber, c. journal of scholarship of teaching and learning, vol. 6, no. 1, august, 2006. 91 (e.g., moon, 2000). reflection also has been identified as a key process in the scholarship of teaching and learning. andresen (2000), for example, argued that the scholarship of teaching, should be inquirydriven, involve critical reflectivity, and scrutiny by peers. two other studies, one with “experts” in the scholarship of teaching (kreber, 2002b) and one with “regular faculty” (kreber, 2003) showed that both “experts” and regular academic staff consider the scholarship of teaching to be inquiry driven and to involve critical reflection. trigwell et al. (2000) identified five qualitatively different conceptions of the scholarship of teaching in a study with faculty at an australian university following the research tradition of phenomenography (marton 1981). these five conceptions were shown to differ in terms of four dimensions, one of them being the focus that reflection on teaching can take. finally, glassick et al (1997) proposed reflective critique as one of six criteria or standards by which to judge the scholarship of teaching. clearly, reflection is recognized as an important aspect of the scholarship of teaching. however, just what precisely it is that we hope reflection will accomplish is not always made clear. at the same time there perseveres a deeply-held belief that through reflection we can enhance our teaching practice, and by extension, the learning experiences of our students. such conclusions presuppose that reflection will lead to valid and valuable forms of knowing. for if the outcomes of reflection on teaching were not assumed to be valid, how would such reflection be meaningful? it is this idea of checking whether what we think actually makes sense, or is “valid”, given the context we find ourselves in, that is at the core of mezirow’s (1991) theory of transformative learning. in emphasizing the importance of validity testing, mezirow is inspired by the work of german sociologist jürgen habermas. borrowing heavily from habermas’s (1971) theory of knowledge-constitutive interests, mezirow distinguishes three different forms of learning: instrumental, communicative and emancipatory. it is through these three forms of learning that we can test the validity of our assumptions. a.three different forms of learning when engaged in instrumental learning we verify a belief or assertion by subjecting it to the empirical-analytical method; that is by posing it as a hypothesis that then can be tested by gathering data that will either support or contradict it. communicative learning, on the other hand, relies on the notion that through communication with others we can reach a common understanding on what is true. an assertion or belief is valid, therefore, if agreement on it can be reached within a community. such a consensus then is based on what the community at some point has accepted as the norm. while communicative learning is very valuable if the goal is to reach greater understanding within a framework of given norms, this form of learning does not concern itself with the question of how these norms have come about. put differently, through communicative learning we do not ask “why did we ever conclude that things should be this way?” or “why does it matter that we do them this way?” in response to the limits of communicative learning, mezirow (1991), leaning on habermas (1971), suggests that important aspects of learning do not occur on the basis of subjective understanding and consensus within a given social context. instead the most significant forms of learning involve a critical analysis of the processes and conditions by which certain norms we have come to take for granted have evolved and how “consensus” was reached. this is the nature of emancipatory learning. kreber, c. journal of scholarship of teaching and learning, vol. 6, no. 1, august, 2006. 92 whether or not the assumptions or conceptions we hold about university teaching are valid, therefore, can be determined through instrumental, communicative, or emancipatory learning or any combination of these. iii. what role do experience and theory play in reflection on teaching? when we think of reflection on teaching, we typically have in mind teachers reflecting on their personal teaching experiences rather than on research findings (see also hiebert, gallimore & stigler, 2002; huberman, 1985). this notion is supported by an extensive body of literature which argues that instructors who reflect on their teaching experiences acquire knowledge that is useful to them in the contexts in which they teach (e.g., cochran-smith & lytle, 1990; munby & russell, 1994; schön, 1983). moreover, these scholars suggest that the teachers’ personal knowledge, constructed on the basis of teaching experience, is more valuable than theoretical or research-based knowledge on teaching. theoretical knowledge about teaching, such as that found in books and academic journals, is, according to these scholars, more or less irrelevant or worthless as it cannot be directly applied to practice. an intriguing question, however, is whether the reflective process might also be directed to theoretical knowledge about teaching, and if so, under what circumstances would theoretical knowledge be of value to teachers? norris (2001) suggests that the value of educational theory for the practice of teaching depends on how teachers engage with theory. educational theories, he argues, surely will not seem particularly useful to teachers if they are wrongly expected to serve as situational or context-specific problem-solving strategies. this cannot be the purpose of any theory. instead, teachers need to understand the value of theories as “general models” which they need to adapt to their specific context. whether and, if so, how such research-based knowledge applies to a teacher’s given situation is a question that only those who know the particulars of the situation can answer. “when the situation is the classroom, teachers know the most about them” (norris, 2001). hiebert et al. (2002) also emphasize the importance of local hypotheses that teachers develop and test across specific contexts thereby working in collaboration with researchers “to digest and transform their general findings into professional knowledge for teaching “ (p.13). it follows that while reflection is certainly associated with experience (see also boud, keoch & walker, 1985; dewey, 1991; kelly, 1955; kolb, 1984; mezirow, 1991; moon, 2000) it also plays a significant role in determining the usefulness of theoretical or research-based knowledge. jarvis (1999) summed it up most succinctly when he argued that theories serve as information that practitioners need to transform into situationspecific knowledge as they try them out in practice. in doing so they create valid knowledge. the view that there are at least two equally important sources of reflection on teaching – educational theory and teaching experience--, has been repeatedly articulated also in the higher education teaching and learning literature (e.g., kreber, 2002; kreber & cranton, 2000; menges & weimer, 1996; paulsen, 2001; rando & menges, 1991). as rando and menges (1991) suggested more than a decade ago: “articulating a rationale for one’s instructional world…requires reflection about personal theories, knowledge of formal theories, and blending of the personal and formal” (pp. 13-14). while it is indeed important to consider both experience and theory in the discussion of reflection on teaching, doing so does not in and of itself address the question of how reflection enhances the practice of teaching. as we have just seen, jarvis proposed that theories are validated through practical experience but the idea of validation would benefit from a more thorough analysis. as discussed above, mezirow (1991) suggests that we come to know things as being either “true” or “false” through instrumental, kreber, c. journal of scholarship of teaching and learning, vol. 6, no. 1, august, 2006. 93 communicative, and/or emancipatory learning. these forms of learning are linked to different kinds of reflection. we will turn to these different forms of learning and kinds of reflection next. iv. what is transformative learning? mezirow’s theory of transformative learning, to a large extent, is informed by the cognitive-developmental tradition (e.g., kelly, 1955; piaget, 1964). this particular strand of psychology assumes that individuals develop intellectually as they encounter events that cannot be interpreted through their existing mental frames of reference. intellectual development occurs as frames of reference get revised as a result of reflection. a frame of reference, often called a conceptual structure, is interpreted as an interrelated set of assumptions, constructs, or conceptions, individuals actively form through experience. as specific assumptions are questioned and possibly revised (or transformed) in light of contradicting evidence, this can lead to a more substantial change in frame of reference (or “conceptual structure”). since individual assumptions are understood to be hierarchically organized, it would depend on their position or relative importance within the larger frame of reference, whether a revised assumption leads to a transformation in the frame of reference itself. to illustrate this point by means of an example, think of a new faculty member whose assumption that all students would prepare the readings assigned for class is challenged early in the semester when she notices that this holds true for only about 50 % of students. clearly, she now realizes that undergraduate students are not equally ready to assume responsibility and control over their learning. whether or not the transformation of this one assumption (or conception) will actually promote a more substantial transformation in frame of reference (or conceptual structure) will depend on the importance she attributes to this new knowledge. it would be possible, for instance, that she begins to question related assumptions and reflects on how she could better facilitate the process of self-regulated learning for different students, and whether, and if so, why, it matters that students learn to take on more responsibility for their learning. since, as was noted, the assumptions (or conceptions) we hold about teaching and learning are in some ways interrelated, a transformation of one assumption may promote reflection on other assumptions. not in all cases, however, will reflection lead to a drastic change in frame of reference for, through reflection, we may also find our assumptions to be confirmed or validated. let us now look at the different forms “reflection” (as conceived by mezirow) can take. v. are there different kinds or levels of reflection, and if so, are all levels of reflection equally conducive to fostering change and development in higher education teachers? in distinguishing between different kinds of reflection, mezirow put clearer parameters on the rather vague term “reflection” and, hence, made a significant contribution to our understanding of the reflective process and teacher thinking. the three kinds of reflection mezirow identified are content, process, and premise reflection. he describes the differences between the three forms of reflection as such: the critique of premises or presuppositions pertains to problem posing as distinct from problem solving. problem posing involves making a taken-for-granted situation problematic, raising questions regarding its validity. … the term “critical reflection” often has been used as a synonym for reflection on premises as distinct from reflection on kreber, c. journal of scholarship of teaching and learning, vol. 6, no. 1, august, 2006. 94 assumptions pertaining to the content or process of problem solving (mezirow, 1991, p.105). it follows that content, process and premise reflection are very different in terms of their nature and what they can achieve; indeed, one could say that they represent different levels of reflection. the term content reflection is at times confusing to people, particularly when discussed in the context of teaching and learning. contrary to our intuitive understanding, the term “content reflection” does not refer to reflection on the content of the courses we teach. what mezirow means by content reflection is having a clear sense of, and providing a description of, the content of the problem that we need to solve. in short, content reflection asks “what’s really the problem here and what do i need to do?”. in content reflection, we do not question the presuppositions underlying our argument but simply use our present knowledge, that is the assumptions or conceptions we presently hold, to describe a problem and how it is habitually solved by us. according to mezirow, content reflection, is a process in which we “are not attending to the grounds or justification for our beliefs but are simply using our beliefs to make an interpretation” (mezirow, 1991, p.107). to be clear, the question of whether our knowledge is valid is not one posed by content reflection. all we ask through content reflection is “what do i presently know about how to solve this problem?” process reflection, on the other hand, is focused on the effectiveness of the problemsolving strategy itself. here we ask “how do i know that i am effective (or was conscientious) with what i do?”. finally, in premise reflection, we call into question the presuppositions on which our present knowledge is based and ask “why is it that i choose to attend to this problem— is there an alternative?” how these forms of reflection are linked to the three forms of learning discussed earlier-instrumental, communicative and emancipatory-is illustrated next. it has become evident that content reflection does not address the question of validity of the outcomes of reflection. through process and premise reflection, however, we test the validity of our assumptions or conceptions. in process reflection we find out whether what we do works by seeking some form of evidence for its “effectiveness” (which in some case is better interpreted as “meaningfulness” or “conscientiousness”). this evidence might be found through published research we read about, research we conduct ourselves, or through experience such as talking to others. reflection then can be informed by the two sources of knowledge construction discussed earlier: personal teaching experience and educational theory. process reflection then occurs through either instrumental or communicative forms of learning, or both. in case of instrumental learning we might validate our knowledge by posing it as a hypothesis that we then test (for example, “students achieve better test scores if i give them the opportunity to choose between two assignments”). in case of communicative learning, we might validate our knowledge as we discuss what we assume to be true with a community of peers to achieve consensus (for example, as we discuss the meaningfulness and relevance of certain goals or values which guide our curriculum planning). obviously, it is only when we are engaged in premise reflection, that is the questioning of presuppositions of what we believed to be true, that our learning would become also emancipatory. we may question, for example, why we ever decided on certain goals and values and examine the processes and conditions by which these came about. kreber, c. journal of scholarship of teaching and learning, vol. 6, no. 1, august, 2006. 95 if our goal is to enhance university teaching, it is vital that we encourage also process and premise reflection rather than just content reflection on teaching. let us now explore what it might look like when people engage in content, process or premise reflection on teaching. v. how is reflection, and transformative learning, linked to the scholarship of teaching? kreber and cranton (1997, 2000) suggest that the scholarship of teaching and learning involves learning about three equally important domains of teaching knowledge: (1) knowledge about the goals and purposes of our teaching (labelled curricular knowledge), (2) knowledge about how students learn (back then was labelled pedagogical knowledge but should perhaps have been labelled psychological knowledge), and (3) knowledge about instructional design and the instructional process (labelled instructional knowledge). this taxonomy is not unrelated to other models describing the knowledge base of teaching (e.g., rice, 1991; shulman, 1987). the main difference is that the soft model is not limited to identifying knowledge domains but is concerned with the construction of knowledge, through reflection, in each domain. another important aspect of the model is that it stresses the critical examination of goals and purposes of higher education as an integral part of the scholarship of teaching and learning. mezirow’s three levels of reflection (on content, process and premise) serve to explain how faculty might construct knowledge in the domains of curriculum (what are the goals and purposes of our teaching?), pedagogy (how do students learn and develop toward these goals?), and instruction (what can be done to optimize this learning or developmental process?). it follows that individuals may actually be involved in as many as nine different kinds of reflection (namely content, process and premise reflection in the three domains of instructional, pedagogical and curricular knowledge) with each of the nine kinds of reflection generating a slightly different form of teaching knowledge. in summary, as higher education teachers are involved in any of these nine forms of reflection, they draw on their teaching experience or educational research, or both. clearly then, reflection, whether informed by experience or theory, leads to knowing, indeed, as many argue, is a process of active knowledge construction. the knowledge higher education teachers construct through these forms of reflection can be tested for its validity through instrumental, communicative or emancipatory learning processes. when our assumptions become validated as a result of process or premise reflection, we can present some sound arguments by which to justify our practice. to borrow dewey’s (1933) words, through reflection we carefully considered “any belief or supposed form of knowledge in light of the grounds that support it and the further conclusions to which it tends” (p.9). when individual assumptions were not validated as a result of reflection, transformative learning (a revision of assumptions or conceptions) might occur as a result. premise or critical reflection on a single assumption may or may not lead to a transformation of one’s larger frame of reference (or conceptual structure) on teaching2. table 1 summarizes what has been argued and provides some concrete examples of possible responses to the reflective questions posed by the scholarship of teaching (soft) model. tables 2 to 4 illustrate the process of content, process 2 it might be useful to think of the conceptual structure as something similar to prosser and trigwell’s (1999) “conceptions of teaching” (see also martin & balla, 1991; ramsden, 1992; kember & gow, 1994), which can be predominantly teacher-focussed or increasingly more student-focussed and oriented towards promoting students’ conceptual change in the understanding of subject matter. for instructors to change their conceptions, transforming one single assumption may or may not trigger a more drastic change or transformation in “conceptions of teaching”. kreber, c. journal of scholarship of teaching and learning, vol. 6, no. 1, august, 2006. 96 and premise reflection for the domains of curricular knowledge, pedagogical (or rather psychological) knowledge and instructional knowledge respectively. clearly, the goals we identify as a result of reflection within the domain of curricular knowledge influence the reflective processes in the other two domains. the three goals discussed in the tables --selfmanagement, autonomy and social responsibility-are examples only (though they are, of course, consistent with the understanding of the scholarship of teaching and learning as a “professionalism” that is inclusive of the notion of “citizenship”). obviously there are other important goals of higher education including those that are more subject, discipline or program specific. an essential aspect of the soft model is its emphasis on justifying educational goals through process and premise reflection. following this model, faculty can provide evidence of engagement in the scholarship of teaching. this involves demonstrating that we made efforts to validate our knowledge of teaching in learning in the three knowledge domains, and acted on the results of our reflection. given appropriate criteria, it could also be assessed through a process of peer review (kreber, 1998; 2001). how the various reflective processes could be demonstrated in a teaching portfolio i will discuss below. vii. how can transformative learning on teaching and student learning be demonstrated and reviewed? the idea of the teaching portfolio originated in canada in the early 1970s (knapper, mcfarlane, & scanlon, 1972) and later resulted in a publication sponsored by the canadian association of university teachers (caut), entitled "the teaching dossier: a guide to its preparation and use" (shore and others, 1980, 1986). teaching portfolios are meant to have both formative and summative evaluative purposes (edgerton, hutchings, & quinlan, 1991; knapper, 1995; smith, 1995). this is to say that they are meant to promote teacher growth and provide a basis for judging teaching effectiveness. typically, teaching portfolios include various documents among them a philosophy statement, outlines of courses taught, unsolicited comments from students, written feedback from colleagues, examples of course work completed by students, summary of teaching evaluation from students, and so forth. evidently, sources such as these, particularly if compiled together, provide a broader and more objective picture of teaching effectiveness than, for example, student ratings of instruction alone. clearly, teaching portfolios can be very useful in demonstrating teaching effectiveness to an evaluation committee. however, it is less obvious how teaching portfolios thus construed serve their formative purpose. most teaching philosophy statements that i have read over the years provide “thick” descriptions (e.g., lincoln & guba, 1985) of present practices but demonstrate little evidence of critical reflection on the underlying assumptions themselves. these descriptions are good examples of what is meant by “content reflection”. surely, this does not necessarily mean that faculty are not critically reflective; it is equally possible that many simply do not know how to demonstrate their reflection. my purpose in this essay is to show how teaching portfolios could be used to document engagement in the various reflective processes associated with the soft model. the portfolio then could be a means not only for stating the assumptions we hold about instructional design issue, student learning and development and goals and purposes but also for documenting the processes by which assumptions were constructed. the basic idea behind such a portfolio is that both authors and reviewers can form judgements regarding the validity of the outcomes of reflection by exploring and assessing the extent to which stated assumptions are the result of kreber, c. journal of scholarship of teaching and learning, vol. 6, no. 1, august, 2006. 97 table 1 the scholarship of teaching (soft) model (adapted by kreber & cranton, 2000): content, process and premise reflection in the three knowledge domains (examples of possible responses) curricular knowledge pedogogical/psycholog ical knowledge instructional knowledge validity testing involved? form of learning sources of knowledge used in reflection content reflection “what are the goals and purposes of my teaching?” “what do i know about how students learn and develop?” “what instructional strategies should i use?” the problem is described “the problem i need to solve here is clarifying my goals.” “the problem i need to solve here is how to promote moral development and social responsibility.” “the problem i need to solve here is (for example) how to provide students with real-life problems.” i make explicit what i already “know” (what i believe to be true) no challenging of assumptions is taking place at this point. largely experience-based but could also be research-based possible habitual response “my main goal is to promote in students a greater sense of moral development and social responsibility.” “i can do this by providing them with opportunities to solve real-life ill-structured problems.” “i can do this by incorporating a service learning component.” no process reflection “how conscientious have i been in identifying this goal?” “how effective am i in learning how to promote moral development and social responsibility?” “how effective has my use of service learning been in providing students with opportunity to solve real-life problems?” yes instrumental and/or communicative experience-based and/or researchbased premise reflection “why does my goal of promoting social responsibility matter – what possible alternatives are there?” “why does it matter that i offer opportunities to solve real-life and illstructured problems – what possible alternatives are there?” “why does it matter that i use this approach (here service learning). is there an alternative? yes emancipatory (possibly preceded by instrumental and/or communicative learning) experience-based and/or researchbased kreber, c. journal of scholarship of teaching and learning, vol. 6, no. 1, august, 2006. 98 table 2 examples of possible responses to content, process and premise reflection questions in the domain of curricular knowledge. first goal second goal third goal content reflection “what are my goals” process reflection “how effective or conscientious have i been in identifying these goals” or “how did these goals come about”? premise reflection “why does it matter that i promote these goals?” “my goal is to promote selfmanagement.” “the need for students to develop the skills, abilities and attitudes needed for continuous learning has been repeatedly emphasized in the educational literature.” “while there are other important goals, students need to acquire the capacity to engage in continuous adaptive learning because such learning has become a reality in our rapidly changing world. without this capacity individuals will be seriously limited in their opportunity to make changes necessary to improve their lives.” “my goal is to promote personal autonomy” “philosophers, educators and social critics have identified the cultivation of autonomous individuals as an important purpose of higher education.” “while there are other important goals, students need to be able to distinguish mere habit and convention from what they can defend by argument in order to solve the most pressing problems of our times. without this ability individuals easily become puppets and automatons of hegemonic forces that take control of their lives.” “my goal is to promote social responsibility” “people arguing that higher education should bring about autonomous thinkers also emphasize the need for responsible citizens. carnegie identified encouraging civic engagement as an important educational purpose.” “while there are other important goals, students need to develop a sense of responsibility towards the community and the environment because our most pressing problems can be solved only by recognizing that we are in this together. without this sense of stewardship and citizenship we open the door to conflict as marginalized groups and social and environmental issues easily get ignored.” kreber, c. journal of scholarship of teaching and learning, vol. 6, no. 1, august, 2006. 99 table 3 general questions asked in content, process and premise reflection in the domain of pedagogical (or “psychological”) knowledge. self-management autonomy social responsibility content reflection process reflection premise reflection “what do i know about how students develop a sense of self-management?” “how do i know that what i believe about how to promote self-management (for example, promote selfregulated learning, deep level approaches, self-evaluation, information finding, etc), is actually true?” and “how effective have i been in identifying, or learning, about how students develop selfmanagement?” “why does it matter that i focus on these constructs, ideas or theories to help students develop selfmanagement – what possible alternatives are there?” “what do i know about how students develop a sense of autonomy?” “how do i know that what i believe about how to promote autonomy (for example, promoting intellectual development, critical thinking, exercising freedom of choice, etc) is actually true?” “why does it matter that i focus on these constructs, ideas or theories to help students develop autonomy - what possible alternatives are there?” “what do i know about how students develop a sense of social responsibility?” “how do i know that what i believe about how to promote social responsibility (for example, promoting moral development, social commitments, cultural and environmental sensitivity, etc) is actually true?” “why does it matter that i focus on these constructs, ideas or theories to help students develop social responsibility -what possible alternatives are there?” kreber, c. journal of scholarship of teaching and learning, vol. 6, no. 1, august, 2006. 100 table 4 general questions asked in content, process and premise reflection in the domain of instructional knowledge. self-management for example, linked to self-regulated learning deep level learning self-evaluation finding information autonomy for example, linked to critical thinking deep level learning intellectual development exercising freedom of choice social responsibility for example, linked to moral development cultural and environmental sensitivity content reflection process reflection premise reflection “what do i know about the strategies needed to help students develop a sense of self-management?” “how do i know that these strategies are effective?” “why does it matter that i use these strategies -what possible alternatives are there?” “what do i know about the strategies needed to help students develop a sense of autonomy?” “how do i know that these strategies are effective?” why does it matter that i use these strategies -what possible alternatives are there?” “what do i know about the strategies needed to help students develop a sense of social responsibility?” “how do i know that these strategies are effective?” why does it matter that i use these strategies -what possible alternatives are there?” kreber, c. journal of scholarship of teaching and learning, vol. 6, no. 1, august, 2006. 101 engagement in process and premise reflection, or instrumental, communicative and emancipatory learning. in order for us to demonstrate our reflection on the various aspects of the scholarship of teaching model, it is not enough to tell others that we reflected. demonstrating reflection more convincingly, would involve providing indicators of engagement in the reflective processes underlying the scholarship of teaching and learning (here the soft model). indicators of engagement in the scholarship of teaching are concrete actions we took from which engagement in the various reflective learning processes can be inferred. while kreber and cranton (2000) proposed a list of indicators based on a conceptual analysis, a recent study with thirty-six science faculty (kreber, 2005) identified indicators empirically. the list below draws on both records but makes some additional suggestions. the items in the list are concrete actions faculty can take and also describe and document in the portfolio. 1. describing the instructional strategies one uses (content reflection/instructional knowledge--experience-based); 2. asking for peer review of course outline (process reflection/instructional knowledge--experience-based); 3. collecting data on students' perceptions of methods and materials (process reflection/instructional knowledge—experience-based); 4. experimenting with alternative teaching approaches and checking out results (premise reflection/instructional knowledge—experience-based); 5. comparing different instructional strategies for their suitability in a given context (premise reflection/instructional knowledge—experience-based) 6. paying attention to end of term teaching evaluations (process reflection/instructional knowledge—experience-based) 7. writing critiques on “how-to teaching books” (premise reflection/instructional knowledge—research-based); 8. administering learning styles or other inventories to students (process reflection/pedagogical knowledge—research-based/experience-based); 9. writing an article on how to facilitate learning in the discipline and submit it to a scholarly journal (content/process reflection/pedagogical knowledge—researchbased); 10. gathering feedback from students on their learning of discipline-specific concepts (process reflection/pedagogical knowledge—experience-based); 11. comparing research-based insights gained from courses on teaching and learning to one’s knowledge of how students learn (process reflection/pedagogical knowledge—research-based) 12. listening to others, observing how others learn, and discussing or writing about it (process reflection/pedagogical knowledge—experience/research-based) 13. reading articles or books on learning and developmental theory (content reflection/pedagogical knowledge—research-based); 14. observing others teach and observing the reactions of their learners (process reflection/pedagogical knowledge—experience-based) 15. conducting an action research project on student learning (process reflection/pedagogical knowledge—research-based); kreber, c. journal of scholarship of teaching and learning, vol. 6, no. 1, august, 2006. 102 16. presenting findings from classroom teaching experiments at teaching-related sessions at conferences (process reflection/instructional knowledge—researchbased); 17. showing how goals of one’s teaching relate to what students need to live successful lives (process reflection/curricular knowledge—experience-based) 18. consulting with an educational development specialist (process reflection/pedagogical knowledge—research-based); 19. comparing classroom experience to formal research results on student learning (process reflection/pedagogical knowledge—research-based); 20. explaining how and why goals have changed over time (premise reflection/curricular knowledge—experience-based); 21. consulting with employers to see what goals they have in mind (premise reflection/curricular knowledge—experience-based). 22. participating in a curriculum review committee (premise reflection/curricular knowledge—experience-based). 23. participating in philosophical discussions on student learning, for example through a listserv or with colleagues (premise reflection/pedagogical knowledge—experience-based) 24. reading books on the goals of higher education and comparing goals to those underlying the programs offered in the department (process reflection/curricular knowledge—research-based) 25. writing articles that compare the usefulness of textbooks in one’s field and compare outcomes of analysis to own text and course content (process reflection/curricular knowledge—research-based) prior to discussing this list it might be useful to make one further observation. it has been suggested that there is a difference between scholarly teaching and the scholarship of teaching (e.g., richlin, 2001) and some feel that this distinction is one of critical importance. to the extent that we adopt this view, we will conclude that most of the above indicators reflect scholarly teaching but not the scholarship of teaching. however, does interpreting the scholarship of teaching exclusively as “publication of research on teaching and learning” (and considering any practices that do not meet this criterion as scholarly teaching) not unnecessarily reduce the scholarship of teaching to the scholarship of discovery on teaching3? the latter, i propose, is just one way by which to engage in and demonstrate the scholarship of teaching. if the scholarship of teaching is aimed at enhancing the quality (and recognition) of teaching and supporting student learning, is informed by knowledge of the field, is inquiry-driven, involves critical reflectivity and scrutiny by peers, as many have argued over the years (e.g., andresen, 2000), does it then not follow that the above indicators are indeed indicators of the scholarship of teaching, particularly if they themselves are shared with peers? essentially, the idea of using indicators is that they allow us to make inferences about the kind of learning or reflection the faculty member has engaged in. to the extent that the indicators suggest engagement in process and premise reflection (through instrumental, 3 i assume that advocates of this view would suggest that the difference between what educational researchers do and what scholars of teaching do, is that for scholars of teaching the research problem originates in their own teaching practice, whereas for educational researchers the problem originates on the basis of theory. this distinction, however, would not hold true for all educational researchers either. kreber, c. journal of scholarship of teaching and learning, vol. 6, no. 1, august, 2006. 103 communicative or emancipatory learning), we can infer that assumptions about teaching and learning were tested for their validity. to be clear, not every indicator on this (or any other) list of suggestions needs to be addressed. the idea is that one would want to see some evidence of process and premise reflection in the domains of curricular, pedagogical and instructional knowledge. such evidence might be produced through traditional forms of inquiry and sharing such as studies of how students learn that are then published, but many of the above indicators do not involve publication. without a doubt, the scholarship of teaching can be demonstrated in many different ways (see for example, theall & centra, 2001). other indicators than the ones suggested here are clearly possible and the development of further indicators by those who practice the scholarship of teaching is both necessary and encouraged. for an outsider it is not always easy to decide whether an indicator (i.e., a concrete action that a faculty member may take) gives evidence of content, process or premise reflection on the part of a faculty member. a higher degree of trustworthiness in the interpretation of indicators might be possible only by talking to the faculty member him or herself. when teaching portfolios are used as described in this essay, namely with the intent to record and demonstrate reflection of different kinds, their greatest value may lie in the formative purposes they serve. conversations based on the portfolio between educational developer and faculty member would hold great promise for further reflection and continued growth in teaching. as for summative purposes, the portfolio holds potential as well. as was noted, it is neither likely nor necessary that reviewers of portfolios will be able to decide for each recorded instance whether reflection was focussed on content, process or premise. nonetheless, reviewers can still arrive at an overall impression of whether the faculty member went beyond content reflection on goals, his or her understanding of student learning and development, and how to promote academic learning and development. results from a recent study suggest that faculty engage primarily in content reflection on teaching, followed by process reflection and that premise reflection across all three knowledge domains is not as common (kreber, 2005b). viii. concluding comments i argued in this essay that the scholarship of teaching and learning needs to be informed by a broader conceptualizations of professionalism, one that is not limited to “best teaching practices” but is inclusive of the notion of citizenship (contributing to the university’s moral and civic purpose). the scholarship of teaching (soft) model (kreber & cranton, 2000) suggests that faculty develop in the scholarship of teaching and learning as they actively construct and validate their knowledge through reflection in three domains of teaching knowledge: (1) knowledge about the goals and purposes of university teaching (curricular knowledge), (2) knowledge about student learning and development toward those goals (pedagogical knowledge, or rather psychological knowledge), and (3) knowledge about how to optimize this learning and developmental process (instructional knowledge). reflection is informed by knowledge gained through personal experience and/or through formal inquiry faculty conduct themselves or read about. content, process and premise reflection are three qualitatively different kinds of reflection. only in process and premise reflection are assumptions or conceptions questioned for their validity. in content reflection we merely make our present assumptions explicit (i.e., we state what we believe to be true); however, content reflection is not irrelevant as identifying kreber, c. journal of scholarship of teaching and learning, vol. 6, no. 1, august, 2006. 104 assumptions is a critical first step in reflection. both process and premise reflection go beyond this and involve “validity testing”. process reflection occurs through instrumental and/or communicative forms of learning and focuses on the problem-solving strategy (“how do i know that what i’m doing makes sense?”). premise reflection can lead to emancipatory learning and focuses on the presuppositions underlying our practices and how they came to be taken for granted (“why does it matter that i/we focus on this problem?”). reflection on assumptions (or conceptions) we hold does not always lead to a transformation of assumptions (or conceptions) to be meaningful (kelly, 1955; mezirow, 1991), as through reflection we may find justification for our practices. only if, through reflection, assumptions are not validated might transformative learning take place. however, even then it will depend on a combination of factors -including personal (for example, willingness to change), social (for example, support by others for change) and contextual ones (for example, external constraints to change) -whether reflection will lead to a transformation of assumptions and, ideally, changes in practice. as well, as was stated earlier, whether or not one transformed assumption leads to a transformation in the larger frame of reference (or conceptual structure) on teaching depends on the importance we attribute to our new insight or knowledge. the scholarship of teaching and learning thus construed involves both learning and knowing about teaching. teaching portfolios offer the opportunity to document or publicize our engagement in the scholarship of teaching and learning and to share or exchange the insights we gained through the various reflective processes with the larger academic teaching community so that they, in turn, can review and critique our practices. this exchange or sharing of indicators of reflection with members of the academic teaching communities may encourage others to build on our work. it might make most sense to start this sharing within our own departmental contexts where our insights can immediately be built upon to improve practice. in an ideal scenario there would be teaching environments in place in all departments across the university that encourage faculty to support each in other in the process of content, process and premise reflection on educational goals and purposes, learning and student development and instruction design. but small groups of faculty who start a weekly or monthly discussion group can make a difference. and if this group decides to go together to conferences on teaching and learning to share their own work more widely or to learn from that of colleagues, they have even greater insights to share with their own departmental colleagues when they return. i should not conclude without noting that the three forms of reflection described here (on content, process and premise) are also useful in the planning of educational development program initiatives. questions program planners could ask themselves include: ● to what extent are participants in our program encouraged to engage in content, process and premise reflection on the goals and purposes of the courses they teach specifically and on the goals and purposes of a university education more generally? ● to what extent are they encouraged to engage in content, process and premise reflection on learning and student development? ● to what extent are they encouraged to engage in content, process and premise reflection on instructional design (including teaching and assessment methods)? ● to what extent are they encouraged to reflect on their personal experience and on education theory? ● to what extent are they encouraged to contribute to educational theory? ● how do we evaluate these kinds of learning? kreber, c. journal of scholarship of teaching and learning, vol. 6, no. 1, august, 2006. 105 i suggested that faculty can record the indicators of their reflections (i.e., the concrete activities they engaged in that made them reflect) in the form of a teaching portfolio for formative and summative evaluation purposes. however, the list of indicators introduced earlier also serve a second purpose: these indicators are useful also for planning educational development assignments or activities for faculty and gtas as they are concrete examples of activities that can be planned that would involve course participants directly in the desired forms of reflection. references andresen, l.w. (2000). a usable, trans-disciplinary conception of scholarship. higher education research and development, 19(2), 137-153. boud, d., keoch, r., & walker, d. (1985). reflection: turning experience into learning. london: croom helm. brookfield, s. (1991). the skillful teacher. san francisco: jossey-bass. brookfield, s. (1995). becoming a critically reflective teacher. san francisco: jossey-bass. calderhead, j. (1989). reflective teaching and teacher education. teaching and teacher education, 5(1), 43-51. cochran-smith m. & lytle, s.l. (1990). research on teaching and teacher research: the issues that divide. educational researcher, 19(2), 2-11. colby, a., ehrlich, t., beaumont, e. & stephens, j. (2003). educating citizens. preparing america’s undergraduates for lives of moral and civic responsibility. the carnegie foundation for the advancement of teaching. san francisco: jossey-bass. cranton, p.a. (1998). no one way. toronto: wall and emerson. dewey, j. (1933). how we think. new york: heath. dfes (2003). the future of higher education (white paper). norwich, uk. edgerton, r., hutchings, p., & quinlan, k. (1991). the teaching portfolio: capturing the scholarship of teaching. washington, dc: american association for higher education. glassick, c.e., huber, m.t. & maeroff, g.i. (1997). scholarship assessed. evaluation of the professoriate. san francisco, ca: jossey-bass. habermas, j. (1971). knowledge and human interests. boston: beacon press. hatton, n., & smith, d. (1995). reflection in teacher education: towards definition and implementation. teaching and teacher education, 11, 33-49. kreber, c. journal of scholarship of teaching and learning, vol. 6, no. 1, august, 2006. 106 hiebert, j., gallimore, r., & stigler, j. (2002). a knowledge base for the teaching profession: whaat it would like it and how we can get there, educational researcher, 31(5), 3-16. huberman, m. (1985). what knowledge is of most worth to teachers? a knowledge-use persepctive. teaching and teacher education, 1, 251-162. hutchings, p., & shulman, l. s. (1999). http://www.carnegiefoundation.org/elibrary/docs/sotl1999.htm (also published in change magazine, september/october 1999. volume 31, number 5, pages 10-15) jarvis, p. (1999). the practitioner researcher: developing theory from practice. san francisco: jossey-bass. kelly, g.a. (1955). the psychology of personal constructs, vol. 1. new york, ny: norton and company. knapper, c.k., mcfarlane, b., & scanlon, j. (1972). student evaluation: an aspect of teaching effectiveness. caut bulletin, 21(2), 26-34. knapper, c.k. (1995). the origins of teaching portfolios. journal of excellence in college teaching, 6(1), 45-56. kolb, d. (1984). experiential learning. englewood cliffs, nj: prentice hall. kreber, c. (2005a). charting a critical course on the scholarship of university teaching movement. studies in higher education. volume 30, number 4, 389-407. kreber, c. (2005b). reflection on teaching and the scholarship of teaching. higher education. volume 50 (2), 323-359. kreber, c. (2003). the scholarship of teaching: conceptualizations of experts and regular academic staff. higher education,46 (1): 93-121. kreber, c. (2002a). teaching excellence, teaching expertise, and the scholarship of teaching. innovative higher education, 27(1), 5-23. kreber, c. (2002b). controversy and consensus on the scholarship of teaching: a delphi study. studies in higher education, 27 (2), 151-167. kreber, c. (2001). designing teaching portfolios based on a formal model of the scholarship of teaching. to improve the academy. the professional and organizational development network in higher education. bolton, ma: anker publishing com, vol 19, pp. 285-305. kreber, c. (1999). a course-based approach to the development of teaching-scholarship: a case study. teaching in higher education, 4(3),309-325. kreber, c. journal of scholarship of teaching and learning, vol. 6, no. 1, august, 2006. 107 kreber, c. (1998). perfection through reflection: the teaching portfolio to evaluate and professionalize university teaching (in german). in, handreichungen zur hochschuldidaktik (section 18, pp.1-24), handbuch hochschullehre. bonn, germany: raabe verlag. kreber, c. & cranton, p. a. (2000). exploring the scholarship of teaching. journal of higher education, 71(4), 476-496. kreber, c. & cranton, p.a. (1997). teaching as scholarship: a model for instructional development. issues and inquiry in college learning and teaching, 19(2), 4-13. kember, d., & gow, l. (1994). orientations to teaching and their effect on the quality of student learning. journal of higher education, 65, 59-74. lempert, d.h. (1996). escape from the ivory tower. student adventures in democratic experiential education. san francisco: jossey-bass. lincoln, y., & guba, e. (1985). naturalistic inquiry. beverly hills, ca: sage. martin, e., & balla, m. (1991). conceptions of teaching and implications for learning, research and development in higher education, 15, 148-55. marton, f. (1981). phenomenography-describing conceptions of the world around us. international science, 10, 177-200. menges, r. j. & weimer, m. (1996). teaching on solid ground: using scholarship to improve practice. san francisco: jossey-bass. mezirow, j. (1991). transformative dimensions of adult learning . san francisco: jossey-bass. munby, h. & russell, t. (1994). the authority of experience in learning to teach: messages from a physics methods class. journal of teacher education, 45(2), 86-95. moon, j.a. (2000). reflection in learning and professional development. london: kogan page. national panel report (2002). greater expectations: a new vision for learning as a nation goes to college. washington, dc: association of american colleges and universities. norris, s. p. (2001). the pale of consideration when seeking sources of teaching expertise. the american journal of education, 108 (3), 167-196. orr, d.w. (1990). the liberal arts, the campus, and the biosphere. harvard educational review, 60(2), 205-217. paulsen, m. b. (2001). the relation between research and the scholarship of teaching, in kreber, c. (ed.) revisiting scholarship: perspectives on the scholarship of teaching. new direction for teaching and learning, no. 86. san francisco: jossey-bass. kreber, c. journal of scholarship of teaching and learning, vol. 6, no. 1, august, 2006. 108 piaget, j. (1964). the child's conception of the world. london : routledge & kegan paul. piper, m. (2003). special feature: building a civil society. a new role for the human sciences. 2002 killam annual lecture. the canadian journal of higher education, 33(1), 113-130. prosser, m., & trigwell, k. (1999). understanding learning and teaching. the experience in higher education. society for research into higher education: open university press. ramsden, p. (1992). learning to teach in higher education. london: routledge. rando w.c. & menges, r.j. (1991). how practice is shaped by personal theories, in: menges, r.j. & svinicki, m.d. (eds) college teaching: from theory to practice. san francisco: josseybass. rhoads, r.a. (2000). democratic citizenship and service learning: advancing the caring self. in, m. baxter magolda (ed), teaching to promote intellectual and personal maturity: incorporating students’ worldviews and identities into the learning process (pp. 27-44). new directions for teaching and learning, no. 82. san francisco: jossey-bass. rice, r.e. (1991). the new american scholar: scholarship and the purposes of the university, metropolitan universities, 1, 7-18. richlin, l. (2001). ‘scholarly teaching and the scholarship of teaching’, in kreber, c. (ed), revisiting scholarship: perspectives on the scholarship of teaching, new directions for teaching and learning, no 86. san francisco, ca: jossey-bass. schön, d. (1983). the reflective practitioner. san francisco: jossey-bass. schön d (1995, november, december). the new scholarship requires a new epistemology. change, 27-34. shore, b. m., foster, s. f., knapper, c. k, nadeau, g. g., neill, n., & sim, v. (1980, 1986) the teaching dossier. (revised edition). montreal: canadian association of university teachers. shulman, l.s. (1987). knowledge and teaching, harvard educational review, 57, 1-22. smith. r.a. (1995). creating a culture of teaching through the teaching portfolio. journal of excellence in college teaching, 6(1), 75-100. theall, m.,& centra, j.a. (2001). assessing the scholarship of teaching: valid decisions from valid evidence. in c. kreber (ed.), revisiting scholarship: perspectives on the scholarship of teaching. new directions for teaching and learning, 86. san francisco: jossey-bass. trigwell, k., martin, e., benjamin, j, & prosser, m. (2000). scholarship of teaching: a model. higher education research and development, 19(2), 155-168. kreber, c. journal of scholarship of teaching and learning, vol. 6, no. 1, august, 2006. 109 walker, m. (ed) (2001). reconstructing professionalism in university teaching. society for research into higher education: open university press. world conference on higher education (1998). higher education in the twenty-first century. vision and action. volume i, final report. paris: unesco publishing. zeichner, k.m., tabachnik, b.r., & denmore, k. (1987). individual, institutional, and cultural influences on the development of teachers’ craft knowledge, in j. calderhead (ed) exploring teacher thinking. london, uk: cassell. microsoft word isaacsonfinal.doc journal of the scholarship of teaching and learning, vol. 6, no. 1, august 2006, pp. 39 55. metacognitive knowledge monitoring and self-regulated learning: academic success and reflections on learning randy m. isaacson and frank fujita1 abstract: during the past decade the relationship of self-regulated learning (srl) to academic success has been extensively explored but the impact of metacognition in this process has not been thoroughly examined. this study examined the relationship of metacognitive knowledge monitoring (mkm) to classroom performance. eighty-four undergraduate students in an introductory educational psychology class completed ten weekly in-class tests in which they were allowed to choose test questions. students were asked to identify the number of hours they studied, their level of confidence, and to predict their test results after completing the test but before it was graded. high achieving students were: more accurate at predicting their test results; more realistic in their goals; more likely to adjust their confidence in-line with their test results; and more effective in choosing test questions to which they knew the answers. the study supports the relationship of metacognitive knowledge monitoring to self-regulated learning and academic success. keywords: self-regulated learning, metacognitive knowledge monitoring i. introduction. the application of self-regulation to learning is a complicated process involving not only the awareness and application of learning strategies but also extensive reflection and selfawareness. pintrich (1995) describes self-regulation as the “active, goal-directed self-control of behavior, motivation, and cognition for academic tasks by an individual student.” (p. 5) students who are skillful at academic self-regulation understand their strengths and weaknesses as learners as well as the demands of the specific tasks. they approach learning with an assortment of strategies they might apply to achieve their goals and an understanding of when and how to implement their plan. but students who are expert learners have more than an arsenal of study strategies and the ability to regulate academic resources, they also know when they have mastered, or not mastered, the required academic tasks. that is, expert learners are also skillful at metacognitive knowledge monitoring (mkm). self-regulated learners are skillful at monitoring their learning and comprehension which has a direct effect on each step in the self-regulation process. pintrich et al. (2000) compares monitoring to the thermostat of a furnace. when the temperature falls below a specified level the thermostat tells the furnace to turn on the heat; when a learner is confused or does not comprehend what they are studying the monitor tells the learner to regulate their behavior, cognitive strategies, or motivation and affect to increase learning. to be effective learners, students must adjust their efforts based on their awareness of their own understanding and the level of difficulty of the upcoming task. one of the critical barriers to success for many students may be their inability to objectively assess their mastery of the academic tasks they are facing. accurate monitoring of learning can impact self-regulation throughout the learning 1 respectively, professor of education, indiana university south bend, risaacso@iusb.edu and associate professor of psychology, indiana university south bend, ffujita@iusb.edu. isaacson, r.m., and fujita, f. journal of the scholarship of teaching and learning, vol. 6, no. 1, august 2006. 40 process. zimmerman (1998) proposes three phases to self-regulation which incorporate metacognition. the first phase is forethought which includes goal setting, strategic planning, and self-efficacy; students identify their goals, their plans for achieving them, and consider how likely it is they will achieve their goals. the second phase is performance or volitional control which includes attention focusing, self-instruction, and self-monitoring; students attempt the learning tasks and monitor what they are learning. the third phase is self-reflection which focuses on comparing self-monitored information with a standard or goal and reactions to the results. during the reflection stage students assess their success or failure, modify their selfefficacy, make causal attribution, and adapt for future learning. in all three of these phases students are using academic goals as the yardstick against which they assess their learning; using the thermostat metaphor, goals are the set-temperature the thermostat uses to judge whether to turn-on the furnace. in each phase, self-reflection and selfmonitoring are critical to master the skills of self-regulation as well as the content being learned. in each phase students who are skillful at self-regulation are cognizant of their understanding and adjust their goals and self-efficacy based on internal as well as external feedback on their mastery of the tasks. students’ ability to monitor their learning is one of the key building blocks in selfregulated learning; students who are aware of the level of their mastery of material can adjust their study time and strategies. over the past decade a number of concepts have been used to describe students’ awareness of their learning. these concepts were originally investigated at a micro-level focusing on metamemory using an experimental format. ease-of-learning (eol) judgements refer to a student’s inferences about how easy or difficult a task will be to learn (nelson & narens, 1990; nelson & leonesio, 1988). eol judgments occur in advance of actual learning, but clearly are a stumbling block for students who underestimate the difficulty of the material or the level at which it must be mastered. judgments of learning (jol) occur during or after learning and are predictions of future test performance (kelemen, 2000; koriat, 1997; nelson & narens, 1990). students whose jol overestimate their actual learning are also likely to terminate their studying prior to mastering the material and fall short of their goals, especially when the demands of the tasks exceed the levels at which they have learned the required material. these assessments of learning have important implications for self-regulated learning as each influence the thermostat that adjusts how much time and effort a student devotes to studying. but the primary focus of the research on eol and jol has been on metamemory with very little attention to the higher level of concepts and problem solving which occurs in classrooms, particularly in postsecondary classrooms. in these studies metamemory is not a measure of understanding or comprehension, but rather the ability to retrieve information from long term memory. while obviously not irrelevant to classroom learning, these measures are primarily applicable to the most basic learning which occurs in school. the connection to the self-regulation of cognitive strategies is probably most suitable to rehearsal or the most rudimentary forms of elaboration. skillful self-regulators should be able to go beyond the assessment of their ability to recall facts; effective self-regulators should be able to estimate how well they have mastered a body of knowledge and how well they will be able to demonstrate their mastery. skillful self-regulators should be able to predict how well or poorly they will do, and have done, on a test while naive self-regulators should be less able to estimate their academic success. a series of studies by maki and colleagues has shown the significance of metacognition isaacson, r.m., and fujita, f. journal of the scholarship of teaching and learning, vol. 6, no. 1, august 2006. 41 on the comprehension of college students in experimental situations. in a study by maki and berry (1984) students who scored above the median on multiple choice items of text showed more accuracy in their test prediction than students who scored below the median. students who had done well on the test were also better at predicting their test scores: greater metacomprehension could be inferred to influence self-regulation. students who were asked to process information at high levels were also found to be more accurate in their test predictions (maki, et.al., 1990) as were students who had increasingly more information about the text (maki and serra, 1992). maki invited the connection between metamemory, metacomprehension, and classroom learning when she found an experimental relationship between higher order thinking questions and prediction accuracy. studies of metamemory had used simple recall questioning, but maki (1995) found that there was greater accuracy of predictions when students were asked higher order questions that did not require verbatim recall of information. this was in support of the finding of weaver (1990) who had proposed that multiple questions had increased the reliability and calibration of comprehension assessment. these finding have important implications for post-secondary education which places greater value on higher level thinking skills. do successful college students have better metacognitive awareness on tasks that require higher level thinking skills? can they adjust their study skills in response to the increasing demand of these tasks? the transition from high school to college puts many demands on young adults. in the classroom the greatest challenge may be the move from the declarative knowledge emphasized in high school to the higher level thinking skills typically required in college. when students do not recognize that these new demands require new approaches to learning and studying they may be unwilling and/or unable to make the necessary changes. at the core of this problem may be that students do not realize that their learning does not match the demands of the task. they assume they have learned the material if they can recall the important terms. they do not recognize that different academic tasks (e.g., different test formats) demand different levels of learning. they overestimate their understanding because they do not recognize the implication of different levels of learning and varying levels of task difficulty. to be successful in college, students may need to have a variety of srl and metacognitive skills that were not necessarily essential in high school. first, successful college students recognize that professors expect more than the memorization of declarative knowledge. second, successful college students use accurate mkm while studying to assess their mastery of the required material, particularly in relation to what will be require on the performance task (e.g., test). third, successful college students have an arsenal of srl strategies they can choose from to match their level of learning to the demands of the performance task. and finally, successful students are able to self-monitor their understanding and the demands of the performance task during evaluation to adjust their demonstration of the learning (e.g., to choose the right questions to answer during a test.) while it may be very difficult to assess metacognition before or during learning, it is possible to assess mkm during the assessment of their learning. while taking a test are expert students better at identifying what they know and what they do not know? for the past decade a program of research by tobias and everson (2000, 2002) has examined learners’ ability to differentiate between what they know and do not know. their findings indicate learners of all levels of ability and developmental stages are affected by their ability to monitor their learning. in dozens of studies with students of all ages and abilities, tobias and everson have found that students who are able to differentiate between when they know and when they do not know are more likely to excel than students who are not able to isaacson, r.m., and fujita, f. journal of the scholarship of teaching and learning, vol. 6, no. 1, august 2006. 42 distinguish their level of comprehension. the studies by tobias and everson have focused on the correlation between knowledge monitoring and student’s academic performance. but very little evidence exists which explores the relationship between knowledge monitoring and academic choices within classes: how do students who possess effective knowledge monitoring skills use these skills to make decisions which impact their academic success? these experimental studies of metamemory and metacomprehension are an open invitation to naturalistic studies performed in actual classrooms. studies on eol (nelson & narens, 1990) suggest that students who underestimate the difficulty of classroom content may abandon their learning efforts before they have mastered the material. experimental studies on jol (kelemen, 2000; koriat, 1997) indicate that students overestimate their understanding which can lead to discontinuing learning efforts prior to mastery. schommer & surber (1986) demonstrated an illusion of knowing (ik) when students believe that only shallow processing is necessary when the material is actually difficult, which could lead some students to underestimate the necessity of learning classroom content to the degree which will be required for success. each of these issues have clear implications for self-regulation in the classroom context. when students in post-secondary education are presented with learning tasks that require higher level thinking they must accurately judge the difficulty of material to be learned, accurately judge the level of their own understanding, and accurately judge the requirement of the performance task (e.g., test) they will be given in class. while laboratory predictions of test performance are revealing of student’s selfmonitoring there is little classroom research on this topic. in the first study of classroom confidence, shaughnessy (1979) reported high achieving students as being better able to distinguish between known and unknown information. in a study of self-efficacy, sinkavich (1995) reports a significant difference between high and low achieving students on their confidence on individual test items. hacker et al. (2000) report similar findings; high performing students were accurate in predicting their test results with their accuracy improving over multiple exams, while low performing students were poor at predicting test results. hacker et al. (2000) found significant differences between low performing students and high performing students in relation to their ability to predict (before taking a test) and postdict (after taking a test but before receiving their grade) their test results. the implication being that students who were doing poorly in a college course were unable to monitor their knowledge of the course material (i.e., they overestimate their test results in both prediction and postdiction) and therefore were unable to regulate their studying to assure mastery of the course material. the student sample in the hacker et al. study were college students in an undergraduate educational psychology class in which the students took three tests. on each test the highest achieving students on that test were more accurate in their predictions and postdictions, but the analysis did not focus on the overall achievement of students across the semester. would the highest achieving students for the semester be more accurate in their postdictions and would they be more likely to accurately adjust their test postdictions on a test-by-test basis? the present study examines students across ten tests during a one semester undergraduate educational psychology course comparing intra-individual differences for low and high preforming students. all three levels of zimmerman’s (1998) academic self-regulation learning cycle emphasize the importance of goals setting. in the forethought stage students set goals, in the performance and volitional control stage students monitor their learning in relation to their goals, and in the reflection stage students assess their success or failure in relation to their goals. extensive research has focused on goal setting (locke & latham, 1990) but no classroom isaacson, r.m., and fujita, f. journal of the scholarship of teaching and learning, vol. 6, no. 1, august 2006. 43 research to date has examined the impact of goal setting and goal achievement on self-regulation. the present study will explore the relationship among the variables of goal setting, selfmonitoring, performance, and self-regulation. in the performance and volitional control stage of zimmerman’s (1998) learning cycle students are attempting the learning task and monitoring their mastery of the task. in this cycle students who possess good mkm should be able to reflect on the application of their learning to the task and choose appropriate tasks. when given choices of tasks of varying difficulty, such as a selection of diverse test questions, students with good metacognitive awareness should be capable of choosing tasks in which they will succeed and avoid tasks in which they are less likely to succeed. the present study will examine the ability of students to use mkm to choose between tasks of varying difficulty. the self-reflective stage in zimmerman’s (1998) learning cycle examines the outcomes when students compare self-monitored information to their goals, especially in relation to the impact on self-evaluation, self-efficacy, and adaptation of strategies. the combination of goals with knowledge of performance impacts self-efficacy which heightens motivation (bandura, 1997). skillful self-regulators effectively monitor their progress in relation to their goals and then adjust their self-efficacy and future strategies. the present study will begin to explore how the feedback students receive from completing tasks of varying difficulty (e.g., tests) impacts their self-efficacy as they progress through a college course. this study will explore the metacognitive differences between high achieving students and low achieving students. high achieving students, who have good mkm, should be more reflective and thoughtful about decisions they make in their studying, test taking, and selfefficacy. are students who excel across the semester more likely to be more accurate in their estimation of their individual test grades and do they accurately adjust their estimations based on their mastery of the material? do students who excel have goals that are more consistent with their academic performance? do students who excel have realistic self-efficacy and how does that self-efficacy change over the course of a semester? are students who excel more likely to make accurate task choices based on their understanding of the required material. ii. method a. participants the participants were 84 undergraduate college students (59 females, 25 males) enrolled in an introductory educational psychology course on a commuter campus of a mid-western university. all students were university students enrolled as education majors. b. procedures participants took weekly objective tests (true-false and multiple choice) and completed a questionnaire for each test. part of the questionnaire was completed before the participants took the test and part immediately after taking the test but before scoring the test. prior to taking the test each student was asked to report the number of hours they had studied, how many points they would have to achieve to be satisfied with their performance (satisfaction goal), how many points they would have to achieve to be proud of their performance (pride goal), and how confident they were about achieving their satisfaction goal (pre-test self-efficacy). after completing the test, but before it was graded, each student was asked to identify isaacson, r.m., and fujita, f. journal of the scholarship of teaching and learning, vol. 6, no. 1, august 2006. 44 how many points they believed they would achieve on the test2 and how confident they now were about their achieving their satisfaction goal (post-test self-efficacy). then, tests were graded and were returned to the student for review before the student was dismissed from class. each of the weekly tests included 40 objective test questions: 18 lower level test questions which emphasized knowledge and comprehension and were worth 1 point each; 18 moderately difficult questions which emphasized application and were worth 2 points each; and 4 difficult test questions which emphasized analysis and synthesis and were worth 3 points each. for each weekly test, students were allowed to answer only 30 of the 40 test questions; their grade being dependant on both the accuracy of their answers (number of questions correct) and the type of test questions they chose and answered correctly. to earn an a in the class, students had to choose more difficult test questions (worth 2 or 3 points) and get them correct. to earn a lower grade, students could either take more difficult questions and get a lower percentage correct, or take less difficult test questions (worth 1 or 2 points) and get a higher percentage correct. therefore, the key to success in the course was not only correctly answering test questions, but also choosing the test questions you could answer correctly. the weekly tests were designed to reveal and substantiate student metacognitive awareness during testing. ten tests were administered during the semester (approximately one per week). iii. results this study examines the differences between high and low achieving students on a number of metacognitive variables: what are the long-term changes between their estimations of test grades early to late in the semester?; how are their satisfaction and pride goals different from their actual performance?; how does their self-efficacy change and how does that compare to their actual test score?; and, are students able to make academic choices based on their mkm? a. test scores postdiction following the format used by hacker (2000) we examined the hypothesis that high performing students would be more accurate in predicting their test scores than low achieving students. using hacker’s terminology, our participants made “postdictions” because they took the test before estimating their score on the test. for each of the ten tests, a correlation was computed between the test score and the squared error of the student’s postdiction [(test score – expected test score) 2]: let us call this type of correlation the matched-score format. students who have good mkm should be better at postdicting their test scores which would result in smaller squared error scores and a negative correlation between test scores and squared error scores in the matched-score format. the average (median) of these ten correlations was -.26 (all listed correlations are significant at p < .05 unless noted otherwise). for the three most difficult tests, the correlations were -.27, -.49, and -.65. thus, the students who were most accurate in their postdictions (having a low squared error of postdiction) tended to have higher test scores and this was particularly true on the most difficult tests (see table 1). from the previous analysis (matched-score format) it is possible that all students are 2 hacker (2000) makes the distinction between predictions and postdictions. predictions are a student’s estimates of their test scores before they take a test. postdictions are a student’s estimates of their test scores after they have taken the test but before their tests are graded. in the present study the students estimated their results after having taken the test but before the test is graded. the students were not asked to make predictions before taking the test. isaacson, r.m., and fujita, f. journal of the scholarship of teaching and learning, vol. 6, no. 1, august 2006. 45 equally good (or poor) at postdicting their test scores, they all postdict that they will do well, and that on any particular test a few students do poorly at random. to insure that it was the students who consistently performed well on their tests who were making the best postdictions, we also computed correlations between the total of the ten test scores (total points) and each of the ten squared error of postdictions. for example, for the first correlation, we took the squared error of postdiction for test 1, and correlated it with the student’s total points for the semester. let us call this type of correlation the total score format. this calculation was done for each of the ten tests. the median of these ten correlations was -0.18. for the three most difficult tests, the correlations were -0.27, -0.24, and -0.53. thus, the students who have the highest achievement across the semester are better at postdicting test results, and their postdiction accuracy is most pronounced on the most difficult tests. this confirms the findings by hacker (2000) and others (maki and berry, 1984; maki, et.al., 1990; maki and serra, 1992) that demonstrates that high performing students are better at metacognitive awareness, knowledge monitoring, and calibrating how they will do on tests in college. are there also differences in the goals students set and the changes that occur over time during a semester? table 1: correlations between test score and the squared error of predicted test score matched-score format total-score format chapter mean stddev r n p r n p 2 3 4 6 7 8 20 11 14 15 43.0 41.2 43.7 43.5 38.4 44.3 41.7 43.5 37.1 37.1 5.2 6.7 5.0 5.7 6.8 6.8 6.2 7.0 6.3 7.4 -0.270 -0246 +0.054 -0.392 -0.274 -01.54 -0.163 -0.055 -0.492 -0.651 81 78 81 81 77 81 80 77 80 75 0.02 0.04 0.64 0.00 0.02 0.17 0.15 0.64 0.00 0.00 -0.228 -0.190 -0.014 -0.133 -0.273 -0.180 -0.161 -0.238 -0.527 -0.160 76 77 78 79 75 78 77 76 77 74 0.05 0.10 0.91 0.24 0.02 0.11 0.16 0.17 0.04 0.00 mean median -0.264 -0.258 -0.210 -0.185 b. group differences by goals across time four roughly equally sized groups were created based on the total performance across the ten tests. to examine the differences between high achieving groups and low achieving groups in relation to goal setting, expected performance, and actual performance across the ten tests during the semester, a repeated measures manova was performed. the three factors were performance group (four levels), measure (satisfaction goal points, pride goal points, expected points, actual points), and time (the ten different tests). the important significant difference for this analysis was the group by measure interaction (f(9,107.2) = 5.76, p<.05, partial 02 = .274). the highest performing group has small differences between goal points, expected points, and actual points. the lowest performing group had very large differences between satisfaction and pride goal points (high), expected points (high), and actual points (low). the two intermediate groups had appropriately intermediate differences between the two extremes. the three-way isaacson, r.m., and fujita, f. journal of the scholarship of teaching and learning, vol. 6, no. 1, august 2006. 46 interaction was not significant. thus, the lowest performing group of students did not adjust their satisfaction or pride goal points, nor their expected points, to the reality of their actual points. this further supports the hypothesis that low achieving college students are less likely to use metacognitive awareness to make adjustments as they are learning in a college course. the actual test scores of the highest achieving students are very similar to their satisfaction goals and their expected points, with their pride goals figure 1: examination points as a function of type of points and performance level of student 35.00 38.00 41.00 44.00 47.00 50.00 53.00 actual points expected points satisfaction goal pride goal measure e x a m in a ti o n p o in ts a students b students c students d & f students approximately 4 points above their achievement. when the highest achieving students are incorrect in their postdiction they are more likely to under-postdict their score which may have led to more extensive studying before the test. as shown in figure 1, he actual test score of the lowest achieving students are significantly less than their expected points and their goals across the entire semester which may have led to less studying before their test. expected points, goals, and self-efficacy can vary dramatically both across individuals and within individuals across the semester. when a high achieving student is able to make accurate postdictions, is that just because the high achieving student lives up to the universal expectation of good performance, or does the high achieving student appropriately lower expectations when they are going to perform at a lower level in a way that the low achieving student does not? that is, in a course which has a weekly test even high achieving students occasionally have a bad day and if the hypothesis about metacognitive awareness is correct these students should make adjustments to their postdictions, self-efficacy, and goals as the semester progresses. c. intra-individual differences across time to focus on differences within individual students across the semester, we calculated isaacson, r.m., and fujita, f. journal of the scholarship of teaching and learning, vol. 6, no. 1, august 2006. 47 within-subject correlations of the test score, postdicted test score, hours studied, pre-test selfefficacy, and post-test self-efficacy within each individual student across the ten tests. each student will have, for example, a correlation between his or her ten test scores and his or her ten postdicted test scores to help explore how changes in test scores compare to changes in postdictions which indicate whether changes in a student's postdictions across the semester reflect changes in their test scores across the semester. the postdictions of students with effective mkm should rise and fall in concert with their test results indicating that they knew when they did well or poorly on test. these individual correlations are characteristics of each student, and as such can be considered a type of individual difference variable. these individual difference variables were themselves correlated across students with each student’s final grade. these correlations have a number of important implications for self-regulation. intra-individual post-diction accuracy across time. each student has a within-subject correlation between his or her postdicted points and actual points across the ten tests, which reflects the extent to which each student adjusts their postdictions to match their actual score across the test tests during the semester. let’s call this within-subject correlation “relative postdictive accuracy.” the mean relative postdictive accuracy across all students is 0.24. thus, using the weekly test scores and the student’s postdictions for each test, the average student is able to make a somewhat accurate relative postdiction of whether he or she will do better (or worse) on this exam than on the other nine exams. the between-subjects correlation of the relative postdictive accuracy with total points is 0.26 (p=.056) meaning that students whose relative postdictive accuracy is greater than the mean tend to score higher across the ten tests during the semester. students who are more accurate in adjusting their estimation of how well they have done on tests from week to week are more likely to achieve more total points during the semester. reliance on effort for pre-test self-efficacy. each student has a within-subject correlation between his or her number of hours studied and pre-test self-efficacy. let’s call this within-subject correlation “reliance on effort.” the mean reliance on effort is 0.30. the average student is more confident the more hours he or she studied for that test. but the between-subject correlation of reliance on effort with total points is -0.24 (p=0.12) meaning that students whose reliance on effort is greater than the mean are less likely to do well in the class. this could be interpreted to mean that the more a student depends on the number of hours they have studied to decide on their confidence for success, rather than on mkm to decide on how confident they are about the test, the less likely they are to do well on a test. it is good to be able to weigh metacognitive feedback during test preparation rather than being forced to depend primarily on the amount of effort expended when making a prediction on how well you’ll be able to do on an exam. metacognitive changes in post-test self-efficacy. each student has a within-subject correlation between his or her pre-test self-efficacy for achieving their satisfaction goal and their post-test self-efficacy for achieving their satisfaction goal across the ten tests. essentially this measures how much each student is likely to change their self-efficacy for achieving their satisfaction goal from before they take the test to after having taken the test. let’s call this within-subject correlation “self-efficacy constancy.” the mean self-efficacy constancy is 0.30. but the between-subjects correlation of self-efficacy constancy with total points is -0.39 (p<.05). thus, students whose self-efficacy constancy is greater than the mean tend to score lower than students whose self-efficacy constancy is less than the mean. students who use the feedback they receive from taking a test to adjust their self-efficacy are more likely to do better on tests isaacson, r.m., and fujita, f. journal of the scholarship of teaching and learning, vol. 6, no. 1, august 2006. 48 across the semester. this is consistent with what we would expect from students with good metacognitive awareness in that they are aware of how they have done after taking a test but before it is graded. these three individual difference variable all support the theory that high achieving students are consistently monitoring their understanding of their learning and adjusting their postdictions (relative postdictive accuracy), adjusting the time they spend studying (reliance on effort), and are better judges of how well they have done after completing a test (self-efficacy constancy) than low achieving students. this begs the question: can metacognitive knowledge monitoring be taught to students and will improvements in mkm lead to improvements in learning? d. metacognitive impact on choosing test questions each of the tests in the course used a variability difficulty variable weight test format where students were given choices about which questions they selected to be graded. this test format allowed students to eliminate questions to which they did not know the answer but at a cost: choosing more difficult questions earns more points but only if you get the correct answer. for each of the 10 exams, students received a number of questions correct out of 30. each student also had a number of points earned for the exam. since students are allowed to choose 30 questions from 40 available questions, and since their total points are dependant upon choosing 30 questions that they are likely to get correct, it is critical that students choose the appropriate questions from week to week depending upon their mastery of the material. that is, some weeks students have clearly mastered the material and can choose questions which are more difficult and worth more points. other weeks when students have not mastered the material as well, the students with good mkm can choose less difficult questions to which they know the answers while students with poor mkm are likely to guess at the answers and guess at which questions to choose to answer. to assess the accuracy of students' choices we correlated the number of points earned with the number of questions answered correctly within student so that each student had a within-subject correlation coefficient that measured the relationship between the number of questions answered correctly and the student’s score on the exam. we will call this within-subject correlation question dependency (qd). if all of the questions on the exam were worth the same number of points, qd would be 1.0 for all students. because students chose which questions to answer, and different questions were worth differing numbers of points, the average qd was 0.94, the minimum qd was 0.74, and the maximum qd was 0.99. while it is clear that the number of questions answered correctly is the key variable in any student’s score, some students are able to assess their understanding of the material and the test question and choose the appropriate test questions to positively influence their grade. students with high metacognitive skill should have a lower qd, because when they know that they do not understand the material well they will choose easier questions to answer, when they understand the material well they will choose more difficult questions which are worth more points, lowering their qd. students with high metacognitive skills know when they understand the material and are more likely to be able to adjust their choice of test questions based on accurately reflecting upon their learning. since the test format allows them to not answer some questions they can make their choices based on their assessment of their understanding. their test points will depend relatively less on the number of questions answered correctly and relatively more on the difficulty of the questions chosen. students with high metacognitive skill should also, on average, score higher on the exams. isaacson, r.m., and fujita, f. journal of the scholarship of teaching and learning, vol. 6, no. 1, august 2006. 49 a correlation across students was calculated between each student’s average test score and that student’s qd. the between-subjects correlation was negative ( r = -0.33, p < .01) as expected. students with low qd had higher average test scores which is evidence of their metacognitive awareness; they used metacognitive strategies to adjust their test item selection based on their knowledge monitoring. the student whose test scores are more question dependant are most likely to do well based on whether they get questions correct, independent of which questions they choose. expert college students in the present study adjusted their choice of test questions to enhance their test scores based on knowing-when-they-knew. it can be inferred that low achieving students were less accurate in selecting the questions they knew based on their metacognitive awareness of their understanding of the material and their comprehension of the test questions. iv. discussion the literature on metamemory and metacomprehension has demonstrated that better learners are able to make more accurate judgments about their learning. the classroom application of these findings for self-regulated learning has not been thoroughly explored although the work of hacker et.al. (2000) begins to shed light on these possibilities. our findings begin to make a connection between the accuracy of students’ metacognitive judgments of their learning and a number of variables related to self-regulation. the initial step, in zimmerman’s three step academic learning cycle, is forethought which includes goal setting and self-efficacy, the second step is performance and volitional control which is guided by self-monitoring, and the third stage is reflection including assessing success or failure and modifying self-efficacy. to be an effective self-regulated learner a student must use mkm to guide this process and make adjustments in goals, judgments of learning, selfefficacy, and task choice. this study begins to explore these relationships within individual learners. the first study of metacomprehension within a classroom context (hacker, et.al., 2000) confirmed the finding that high achieving students are better at predicting and postdicting their learning. our study supports this finding in relation to student postdictions. students’ postdictions correlated significantly to their test scores. of special interest was the finding that this correlation was greater for tests that were more difficult which supports the finding of maki (1995). this relationship has potential implications for metacognition and self-regulation. if difficult tests are better at differentiating high achievers from low achievers the reason may be that difficult tasks require a higher level of metacognition and self-regulation. given the demands for higher level thinking skills that occur when students transition from high school to college, it would be revealing to explore if effective mkm becomes increasing important in academic tasks that require these thought processes. the groups in the hacker et.al. (2000) study were assigned based on their results on each individual test which leaves open the possible interpretation that the reason for the discrepancy between groups could have been a regression to the mean. if the test goals of most students were approximately 80%, and each student were to predict achieving their goal, the group discrepancy between predictions and actual score would be attributable to their initial goals. since the group assignment changed across each of the three tests it is possible that group membership changed while goals, predictions, and postdictions stayed the same. in our study the group assignments were made based on the students’ accumulated test points across all ten exams. group assignment based on accumulated points across the entire isaacson, r.m., and fujita, f. journal of the scholarship of teaching and learning, vol. 6, no. 1, august 2006. 50 semester rules out the possible regression to the mean explanation. the twenty students in the top group were the students with the highest scores across all ten tests. the twenty students in the bottom group were the students with the lowest scores. the repeated measures manova demonstrates that the top group of students had the smallest differences between their satisfaction and pride goals, expected points, and actual points across the course of the semester while the lowest group of students had the widest discrepancy between these variables. future research should examine whether students’ mkm changes over time, what factors might impact these metacognitive skills, and whether these skills can be taught to lower achieving students. the results from hacker et.al (2000) also suggest that high achieving students are not only more accurate in their judgements but more likely to under-estimate their results (leading to under-confidence) while low achieving students were more likely to grossly over-estimate their results (resulting in overconfidence.) in these conditions self-regulation theory would predict that high achievers who are under-confident might be defensive-pessimists (garcia and pintrich, 1994) which would lead to an increase in their efforts and their success. students who overestimate their result may be self-handicapping which may result in a decrease in their efforts which would lead to failure. the relationship between calibration, confidence, and self-efficacy will need to be more fully explored since self-protective perceptions may influence the accuracy of calibration (dembo and jakubowski, 2003). what is the impact when students underestimate or overestimate their mastery of the course material? how does this impact studying before a test? what impact does this underestimation and overestimation have on motivation to study? according to zimmerman (1998), adjusting goals, expectations, and self-efficacy over time is a critical skill in self-regulated learning. the weekly class test format in our study allowed students to set and readjust goals and expectations before each test during the course of the semester. the pattern of pride and satisfaction goals, expected points, and achievement for the high achieving group is consistent with self-regulation, while the pattern for the low achieving students does not reflect the reality of their actual test scores. low achieving students maintain the same general expectations which may lead them to learned helplessness. future studies should explore the affective reactions of students who do not adjust their goals, their expectations in the face of continual failure, and their attributions for success and failure. one of the advantages of the present study is the longitudinal nature of the data set. since students are given weekly tests and the same data is collected for each test, we can examine the intra-individual differences for a number of variables. self-regulation involves small changes over time within, as opposed to across, individuals. to explore changes in selfregulation it is important to examine how the changes of one student week-to-week compare to the changes within another students. these intra-individual relationships can be examined as individual difference which can then be compared to variables such as final course grade. our study explored three such relationships. the first intra-individual difference we are calling “relative postdiction accuracy.” this is a measure of whether a student is able to predict whether they will do better on this test relative to other tests. the average student is somewhat accurate in estimating whether they will do better on this test compared to other tests, but the highest achieving students are better at judging when they will do better or worse on a test compared to their own performance across the semester. while teachers at all levels may view this as "common sense", the ability of students to judge how they have performed on this test compared to other tests is indicative of the mkm that is critical to academic success in college. high achieving students are not only better at estimating their score on an individual test, they are also sensitive to whether they will isaacson, r.m., and fujita, f. journal of the scholarship of teaching and learning, vol. 6, no. 1, august 2006. 51 do better or worse than they usually perform on the class tests. whether the reason is that they were not able to study as much as usual or they estimate that the material will be more difficult, high achieving students are aware that on this test they are likely to perform better or worse than they usually perform. self-regulated learning in zimmerman’s performance stage (1998) is dependent upon on-line monitoring of performance and when students are not able to assess whether they are doing better or worse than they normally do they are less likely to adjust their behavior. the second intra-individual difference is “self-efficacy constancy.” students were asked to estimate their confidence for achieving their satisfaction goal before they took the test, and after they took the test but before the test was graded. students who are metacognitive about the feedback they receive from taking a test adjust their self-efficacy based on this feedback. students who lack metacognitive awareness do not change their self-efficacy. high achieving students are more likely to modify their confidence for an individual test after having taken the test; "that test was harder/easier than i thought it would be." this is also evidence of the metacognitive awareness necessary for self-regulation. when students do not, or cannot, adjust their self-efficacy after taking a test, it is likely they are not metacognitively aware of how they have done. the third intra-individual difference is “reliance on effort.” many students link their test confidence (i.e., pre-test self-efficacy) to the number of hours they study for a test. when a student has studied four hours for a test they are usually more confident than when they have only studied one hour. but students who are truly self-regulating do not tie their confidence solely to the number of hours they have studied. self-regulating students make judgments of when to stop studying based on how well they know the material and whether they believe they will be able to achieve their goals. many students seem to have a pre-designated number of hours they plan on studying and if they complete those hours they believe they are sufficiently ready for the test. when students rely primarily on time to regulate their learning they may be less likely to succeed. the correlation of “reliance on effort” with total points indicates that the more a student relies on time to determine their pretest self-efficacy the lower their total points in the class. the relationship of metacognitive awareness, pre-test confidence, and self-regulation is an important variable to explore. many students depend on time as the leading indicator of learning and the result is often failure. how students decide they have spent enough time learning is an important question to be explored. the process of self-regulation depends on intra-individual changes in expectations, goals, and self-efficacy. students who are skillful at self-regulation modify what they expect based on the feedback they receive from self-monitoring and external input (e.g., tests). they also adjust their goals and self-efficacy based on these results. the process of learning self-regulation requires extensive time and feedback. the present study begins to follow the development of this process in a college classroom with frequent feedback over an entire semester. the variable weight variable difficulty test format in this study places a strong emphasis on student’s metacognitive awareness. given this test format it is possible for a student to get all 30 of the questions they choose correct and still only earn a c, if they choose the easiest test questions. students are told, and quickly learn, that their grade is dependant not only on the number of questions they get correct, but also choosing the correct test questions. this test format encourages students to take questions that are worth more points, which require higher level thinking skills, with the expectation that they will eventually learn to regulate their study time and strategies to go beyond the simple memorization of facts to the application, analysis, isaacson, r.m., and fujita, f. journal of the scholarship of teaching and learning, vol. 6, no. 1, august 2006. 52 and synthesis of information. but learning at this higher level is not the only skill necessary to succeed with this test format. it is also important for students to know-when-they-know. students who used elaboration and organizational strategies in learning would be more likely to do well if all students were required to take all the questions of highest value. if all students took all the higher level questions, the student who got the most questions correct would have the highest score. but since students can choose which questions to take their score is also dependant upon their ability to choose the question they will get correct. the question dependency (qd) variable helps to reveal a student’s ability, across the semester, to choose questions that will enhance their test score based on their mkm. students who lack mkm are less likely to be able to assess whether they understand, and can answer, each individual test item. for these students their test score is entirely dependent upon the number of questions they get correct because they do not make metacognitive choices on individual items on the test. for students with high mkm their test score is influenced by the questions they choose as well as the number of items they get correct. for example, a student with high mkm who, in a given week, is not able to devote as much time as usual to studying for the higher level thinking test questions is likely to choose lower level difficulty test questions on that test. these students are aware that they do not know the answer to the more difficult test questions and, instead of guessing, choose the easier questions for which they know the answer. their mkm influences their total score which makes them less question dependent. but students with poor mkm cannot make these choices because they do not know-when-they-know and therefore are guessing: guessing not only on the answer to the question, but also on whether they should choose the question. this finding invites further exploration of the relationship of metacognitive awareness and learning particularly when higher level thinking is required. many undergraduate students have difficulty when they first enter college because they are not familiar with the academic demands of higher level thinking. if they were effective in memorizing information in high school they were likely to have received good grades and also were likely to have believed their were good at learning. this would lead them to believe they would be successful in college if they were to use the same learning strategies. when they enter college and fail to meet their own expectation, they are then more likely to externalize the blame for their failure to the teacher or an unfair task. this may keep them from engaging in the reflection which is essential to changing their study behavior and improving their metacognitive awareness. are high achieving college students better able to predict their scores on difficult tests because the questions are more difficult (maki, 1995) or because the test requires higher level thinking skills? this study also raises the question of whether metacognitive awareness can be improved over time. it is clear that the highest achieving students in this study were better able to predict their test scores and also choose the right test questions. are the metacognitive skills that allow these students to choose the right test questions a stable characteristic, or can these skills be improved over time? if these skills can be improved, what instructional approaches would facilitate an improvement in their metacognitive awareness that would impact their self-regulated learning and success in school? this study and others (e.g., hacker et.al., 2000; tobias and everson, 2000, 2002; maki, 1995) are demonstrating a strong relationship between mkm and academic performance but the causal relationship is unclear. can an improvement in mkm lead to academic improvement or is improved mkm a result of improved academic performance? this study demonstrates that expert students are effective at estimating their understanding (postdicting their test scores) and they are more inclined to vary their goals and isaacson, r.m., and fujita, f. journal of the scholarship of teaching and learning, vol. 6, no. 1, august 2006. 53 self-efficacy based on past results and the feedback they receive from taking a test. expert students in this study were also more likely to make choices which demonstrate mastery and non-mastery of tasks of varying levels of difficulty (choosing the appropriate test questions.) this has important implications for the teaching-learning process in higher education and secondary education. as students are required to take on academic tasks of increasing difficulty it is critical that they have the metacognitive skills to assess their mastery of the material on a variety of levels. this metacognitive self-assessment is essential to the application of selfregulated learning. this study also invites a thorough examination of the relationship of mkm to learning. it seems clear that expert students are skilled in reflecting on their own learning but the origin and nature of these skills is not clear. does mkm facilitate student learning or does the mastery of a body of knowledge assist students in their judgement of their mastery of the material? does mkm change over time and can it be taught? if mkm is a skill that can be learned, what pedagogical changes in classroom practice are most likely to encourage students to be more reflective of their own learning? what types of assessment practices are most likely to encourage students to be more metacognitively aware? are there procedures that can be implemented in most postsecondary classrooms (e.g., frequent evaluation and immediate feedback) that would encourage students to be more metacognitively aware? can technology be used outside the classroom to assist students to improve their mkm? the author is presently implementing pedagogical approaches that are designed to make students aware of the impact of metacognitive knowledge monitoring and assist them in improving their self-awareness of their learning. this study also raises questions about how to assess metacognitive knowledge monitoring. this study demonstrates the challenges inherent in assessing the metacognitive awareness of students while they are taking a test which reveals the even more difficult task of assessing metacognitive decision making while students are studying which clearly is the more crucial connection between metacognitive knowledge monitoring and self-regulated learning. this study clearly demonstrates that low achieving students frequently over-estimate how well they know a body of information which leads to disengagement early during studying. this process typically results in failing the test which often leads them to blame the teacher/test for their failure instead of examining their own learning. a thorough examination of the impact of metacognition, and pedagogical approaches that might increase metacognition, are an important issue that should be addressed in the scholarship of teaching and learning. references bandura, a. (1997). self-efficacy: the exercise of control.. new york, ny: w. h. freeman. dembo, m., t.g. jakubowski (2003, april). “the influence of self-protective perceptions on the accuracy of test predictions.” presented at the annual conference of the american educational research association, chicago, il. garcia, t., p. r. pintrich (1994). “regulating motivation and cognition in the classroom: the role of self-schemas and self-regulatory strategies.” in d.h. schunk & b.j. zimmerman (eds.), self-regulation of learning and performance: issues and educational application. (pp. 127153). hillsdale, nj: erlbaum isaacson, r.m., and fujita, f. journal of the scholarship of teaching and learning, vol. 6, no. 1, august 2006. 54 hacker, d. j., bol, l., horgan, d. d., & rakow, e. a. (2000). “test prediction and performance in a classroom context.” journal of educational psychology, 92, 160-170. kelemen, w. l. (2000). “metamemory cues and monitoring accuracy: judging what you know and what you will know”. journal of educational psychology, 92(4), 800-810. koriat, a. (1997). “monitoring one's own knowledge during study: a cue-utilization approach to judgements of learning.” journal of educational psychology, 126(4), 349-370. locke, e. a., & latham, g. p. (1990). a theory of goal setting & task performance. englewood cliffs, nj: prentice-hall. maki, r. h. (1995). accuracy of metacomprehension judgments for questions of varying importance levels. american journal of psychology, 108(3), 327-344. maki, r. h., & berry, s. (1984). metacomprehension of text material.” journal of experimental psychology: learning, memory, and cognition, 10(4), 663-679. maki, r. h., j. m. foley, w. k. kajer, r. c. thompson, & m.g. willert. (1990). “increased processing enhances calibration of comprehension.” journal of experimental psychology: learning, memory, and cognition, 16(4), 609-616. maki, r. h., & m. serra. (1992). “the basis of test prediction for text material.” journal of experimental psychology: learning, memory, and cognition., 18(1), 116 126. maki, r. h., & serra, m. (1992). the basis of test prediction for text material. journal of experimental psychology: learning, memory, and cognition, 18(1), 116-126. nelson, t., & narens, l. (1990). “metamemory: a theoretical framework and some new findings.” in g. bower (ed.), the psychology of learning and motivation. san diego, ca: academic press. nelson, t. o., & leonesio, r. j. (1988). “allocation of self-paced study time and the "labor-invain" effect.” journal of experimental psychology, 14(4), 676-686. pintrich, p. r., walters, c., & baxter, g. p. (2000). “assessing metacognition and self-regulated learning.” in g. schraw and j. c. impara (ed.), issues in the measurement of metacognition (pp. 43 97). lincoln, ne: buros institute of mental measurement. pintrich, p., r. (1995). “understanding self-regulated learning”. in p. r. pintrich (ed.), understanding self-regulated learning (pp. 3 12). san francisco, ca: jossey-bass. schommer, m., & surber, j. r. (1986). “comprehension-monitoring failure in skilled adult readers.” journal of educational psychology, 78(5), 353-357. shaughnessy, j. j. (1979). “confidence-judgment accuracy as a predictor of test performance.” journal of research in personality, 13, 505-514. isaacson, r.m., and fujita, f. journal of the scholarship of teaching and learning, vol. 6, no. 1, august 2006. 55 sinkavich, f. j. (1995). “performance and metamemory: do students know what they don't know?” journal of instructional psychology, 22(1), 77. tobias, s., & everson, h. (2000). “assessing metacognitive knowledge monitoring.” in g. schraw & j. impara (eds.), issues in measurement of metacognition (pp. 147-222). lincoln, ne: buros institute of mental measurement. tobias, s., & everson, h. (2002). knowing what you know and what you don't: further research on metacognitive knowledge monitoring. college board report no. 2002-3. college board, ny. weaver, c. a., iii. (1990). “constraining factors in calibration of comprehension.” journal of experimental psychology: learning, memory, and cognition, 16(2), 214-221. zimmerman, b. j. (1998). “developing self-fulfilling cycles of academic regulation: an analysis of exemplary instructional models.” in d. schunk & b. zimmerman (eds.), self-regulated learning: from teaching to self-reflective practice (pp. 1 19). new york, ny: guilford. v10n3slade journal of the scholarship of teaching and learning, vol. 10, no. 3, november 2010, pp. 31 40. the student-authored essay as a teaching tool john r. slade jr.1 abstract: students who enter college dreading their “required” courses are understandably skeptical of their ability to succeed in first-year writing. their lack of preparation added to their skepticism results in students with too little confidence that their writing will ever resemble the models used in textbooks. as a tool of engagement, student-authored essays provide writing models students readily identify with, thereby promoting confidence. this analysis examines student perceptions of the student-authored essay as a writing tool and the implications of its use to improve learning outcomes in first-year writing courses. keywords: composition strategies, essay models, first-year writing, foundation writing courses uncertainty about the effectiveness of my teaching techniques created considerable stress during my first years as a composition instructor. before i was introduced to the formal language of assessment, i employed the rudimentary practice of asking students every few weeks what they found particularly helpful or unhelpful about my teaching. the first time out, i phrased the call for feedback something like this: “write down anything i’m doing that helps you learn the material along with those things that don’t help much.” student essays used as writing models garnered the most frequent and positive comments from my informal assessment. the feedback indicated that students felt closer kinship with essay models written by their peers. more captivating was the self-confidence created from the sense that they could reach levels of writing proficiency exhibited in essays by other students. conversely, most suggested little assurance their writing would ever resemble that of the accomplished authors who appeared in their textbooks. throughout those early years of community college teaching, i employed this crude but seemingly effective technique of formative assessment in most classes i taught. after returning to the classroom after many years in administration, i have resurrected this informal practice of real-time assessment. this paper reports on a formal follow up to that practice and examines whether students perceive studentauthored essays as more beneficial teaching aids than professional writing models. i. background. this study was undertaken in fall 2008 and spring 2009 at nc a&t state university to examine the efficacy of a pedagogical tool, the student-authored essay. one of sixteen constituent campuses of the university of north carolina system, nc a&t enrolled 10,388 students, 8,829 of whom were undergraduates, during the period of this study. of the undergraduates, 1,607 were first-time freshmen with an average sat score of 900 (442 verbal) for in-state students and 1,002 (493 verbal) for out-of-state students (nc a&t, 2010). the university of north carolina (2010) reported the average sat score for all sixteen campuses as 1,083. forty-five percent of nc a&t’s fall 2008 first-time freshmen ranked in the top two-fifths of their high school class, 1 department of liberal studies, north carolina a&t state university, 1601 east market st., greensboro, nc, 27411, jrslade1@ncat.edu. slade, j. r., jr. journal of the scholarship of teaching and learning, vol. 10, no. 3, november 2010. www.iupui.edu/~josotl 32 and slightly more, 47 percent, ranked in the bottom three-fifths (nc a&t, 2010). the verbal score takes on a prescient nature if, as was the case for nc a&t in fall 2009, the sat score is the only factor used in determining student preparation for placement in the first-year writing courses. in fall 2006, north carolina a&t implemented an ambitious general education reform program designed around the concept of interdisciplinarity. critical writing, one of five foundation courses required of all students at nc a&t, uses critical thinking explicitly as scaffolding for developing writing skills (graves, 2006). though approximately 300 freshmen enrolled in a developmental writing course, university policy did not obligate them to register for the preparatory course. this practice of optional placement combined with moderately low selectivity results in writing classes populated with students of wide-ranging abilities. as a member of the writing team, i can attest to the concerns voiced by writing faculty who often find it difficult to gauge how much review and remediation to provide. the division requirement to adhere to a common syllabus and assignments (in part to facilitate the collection of valid assessment data) in some ways conflicts with needed and appropriate remediation based on formative assessment. as with any institution, nc a&t’s student profile influences the day-to-day instructional challenges writing instructors must manage. based on the institution’s results from its participation in the wabash national study of liberal arts education, nc a&t students do not place as high a priority on skills, behaviors, and dispositions generally associated with college success. in 2005 and 2007, the wabash college-led study gauged student opinions and feedback from 27 institutions on several factors considered pertinent to college performance and success. nc a&t students ranked near the top in categories measuring aspirations such as desire to earn high grades, to draw a high income, and to engage in entrepreneurship. however, self-reported dispositions and behaviors normally associated with high ambitions placed nc a&t students near the bottom, 26th, for instance, when asked if they were willing to work hard to earn superior grades. they charted similar results in responses to questions about their reading habits, ranking 25th when queried about their willingness to stay with a good book without falling into boredom (childress and southerland, 2008, pp. 16-17). the dichotomous attitudes students report in the wabash study extend to their perspectives on learning to write competently. my own students generally agree with the utilitarian purposes of obtaining sound writing skills; however, far too many do not consider the far-reaching consequences of poor writing skills to a career. while such student attitudes stem in part from freshman naiveté, their erroneous notions sometimes persist until the alerts they eventually awaken to come too late for comprehensive treatment. ii. objective of study. this study aimed to gain some measure of how students perceive two types of writing models used in a foundation-writing course. in the critical writing course at nc a&t state university, instructors routinely employ as instructional tools model essays supplied by the textbook and its supplements. during fall 2008 and spring 2009, the critical writing courses used critical thinking, reading, and writing by sylvan barnett and hugo bedau (6th edition) as its primary text, which included a mix of writings by professional authors and student-authored essays, mainly intended to model application of various rhetorical modes. it seems intuitive that instructors would rely more heavily on the student-authored essays as models to develop student slade, j. r., jr. journal of the scholarship of teaching and learning, vol. 10, no. 3, november 2010. www.iupui.edu/~josotl 33 writing since students are more likely to recognize their own writing patterns in the work of other students. what seems likely, however, is mitigated by another important goal of the instructor, which is to encourage students to develop reading habits befitting college students and eventual college graduates. as emory university’s mark bauerlein (2008) describes in his book about the digital age’s effects on the intellectual development of today’s youth, students who have relegated their sources of information and models of communication to informal and brief modes such as net magazines, social networking sites, and email are suffering predictable negative consequences. to reverse the decline, instructors battle back in one of the few ways they can, which is to assign well-regarded pieces of writing with hope that students will notice and emulate the techniques of good writers. it is easily understood that students who bring less than adequate preparation for college writing coupled with dispositions that hinder their engagement with the subject are less likely to believe that their writing skills will ever match those exhibited in models highlighted in a textbook. this is the focus of this small study – to note student perspectives of writing models used in foundation writing courses at nc a&t. the broader question is whether the analysis of these perspectives can provide useful guidance in what may be more effective in teaching foundation writing courses, especially at an institution where under-preparedness and negative dispositions toward positive collegiate habits conflate to create even greater challenges to achieving proficiency in a fundamental skill. iii. procedure and results. this study surveyed freshmen students at north carolina a&t state university enrolled in the university’s required foundation writing course titled critical writing. students from five classes (n = 84) were given a questionnaire that solicited feedback about the writing models used during the term. the questionnaire categorized the writing models as “student essays” and “professional essays,” was administered the last day of class, and requested the following assessments from students: which category more clearly represented the concepts taught during the semester, which category they referenced more often when applying the concepts to their own writing, and which category they recommended for future sections of the course. in addition to choosing from given responses, students were given the opportunity to elaborate on their choices. the questionnaire also solicited information on previous college-level writing courses students had taken, what types of writing assistance (if any) did students utilize outside of class, and whether students were repeating the course. twenty-two, or 26 percent, of the respondents completed other college-level writing courses prior to completing critical writing in either fall 2008 or spring 2009, and 12, or 14 percent, were repeating the course. all but 8 of the 84 students who completed the questionnaire sought assistance outside of class during the term. writing assistance utilized while enrolled in the course included such services as the campus-based writing center, internet-based writing tools, tutors, and interaction with the instructor via email and office visits. survey responses shown in table 1 below align with the anecdotal responses received from earlier experiences with community college students. by almost 3 to 1, students sampled in this study preferred the writing models by students to those authored by professional writers. in line with the community college students, freshmen at north carolina a&t state university expressed similar reasons for preferring the student models. the most prevalent reason cited greater identification with the student papers and therefore more confidence in the ability to slade, j. r., jr. journal of the scholarship of teaching and learning, vol. 10, no. 3, november 2010. www.iupui.edu/~josotl 34 successfully complete similar writing exercises. the frequency table below presents the frequency results: table 1. student preference for category of model essay (n = 84). frequency percent student-authored essays 63 75 professional essays 21 25 students surveyed employed the word relate most often in detailing reasons for choosing the student essays. based on the informal feedback from the community college experience, it was expected that students in first-year writing courses at the senior institution would match the sentiment. it is necessary to note that students tended to provide their own gloss to the two core terms used in the survey: professional essay and student essay. any essay appearing in the textbook, no matter its label or author, many students considered professional work. they reasoned that any essay in the textbook had earned its way in and therefore must be of superior quality. they then concluded that any essay, professionally authored or otherwise, between the covers of a textbook created a greater challenge to emulate and learn from (at least psychologically). informal feedback suggested that the more likely students were to judge a textbook essay of “professional” quality, the less they felt it benefitted them as a learning tool. consequently, class activities based on essays from the textbook yielded less discussion than student models provided from present or former students. though students tended to draw some distinction between student essays in the textbook and those from their peers or former students, they overwhelmingly favored the student essay when presented as a generic category. student preference for the student-authored essay far outpaced the rate at which students reported how often they referenced this same category of essay, as table 2 below shows: table 2. type of essay students referenced most often (n = 84). frequency percent student-authored essays 43 51 professional essays 38 45 both about the same 3 4 the results above may mask a wider gap between how often students used one essay type over the other. in an informal follow up, students indicated that they defined a key term more broadly than anticipated. some students interpreted referencing an essay as any use or mention of it. for instance, students considered an essay assigned as homework as a reference to it. some students added to their count in-class uses of the essay, even those instances that i initiated. therefore, it is a reasonable conjecture that the manner in which i referenced the essays in class and how often i directed attention to them likely influenced student responses concerning this variable. this is particularly noteworthy because it is true that i spent more time in sustained discussion of the essays from the textbook. i did so because of students’ reticence when discussing textbookbased essays. in terms of self-selected use, some students admitted that they never referenced the essays in the text beyond homework, class discussion, and explicit directions to review them (which was often). therefore, it is possible that the mismatch noted could be a result of the manner in which students interpreted the term reference. slade, j. r., jr. journal of the scholarship of teaching and learning, vol. 10, no. 3, november 2010. www.iupui.edu/~josotl 35 verbal feedback from students combined with a cross tabulation analysis of the essay type preferred and the category referenced more often provided a clearer snapshot of student perspectives. of the 63 students who judged the student-authored essay as more aligned with concepts taught in the course, 30 said they referenced the student essays more often and 30 made the opposite choice. thirteen of the 21 students who viewed the professional essays as more representative of the course concepts reported using the student-authored essays more in completing their own assignments. the lopsided results regarding recommendations students made for future foundation writing courses are more difficult to understand. as table 3 below illustrates, students responding to the questionnaire overwhelmingly recommended both essay categories: table 3. type of essay students would recommend for use (n = 84). frequency percent student-authored essays only 11 13 professional essays only 0 0 both 70 83 no response 3 4 none recommended the professional essay as a single source, and 11 respondents endorsed the use of student samples only. though the numerical data favored both essay categories in this questionnaire item, written comments tended to nuance the responses. the remarks below typify feedback from students who recommended an even balance of student-authored and professional essays: • although i liked the professional essays better, some students may be able to relate to the student essays. • the professional essays make me think a little harder while the student essays you understand right away. • though professional [essays] may be easier to reference or go along with, the student essays may relate more to me and i can find similarities. the students peer edit and grow in that process. • with professional [essays] it’s easier to see more techniques; with students it is easier to understand. overall, student comments showed clear recognition of the benefits and drawbacks to both types of essays as teaching tools. several comments mentioned that the exposure to professional essays illustrated what students might achieve. in many cases, a student’s desire for reliable and repeatable writing approaches applicable to other course work mitigated desire to emulate writers accomplished enough to have their work appear in a textbook. iv. discussion. it is clear that students in this study responded with marked difference to the two categories of writing samples used in class. students interpreted value in both categories of writing samples but gave more credit for their writing development to student-authored samples, namely because they identified more with writers who were also their peers. perhaps this perspective by students indicates that reading, discussing, and learning from a peer possibly provided a psychological lift that built confidence. the positive response to student-authored essays, which students believed more clearly delineated the structures, techniques, and rhetorical tools taught in foundation slade, j. r., jr. journal of the scholarship of teaching and learning, vol. 10, no. 3, november 2010. www.iupui.edu/~josotl 36 writing courses, suggests that writing models perceived as accessible matter, especially for the marginal writer. in other words, the student-authored essays, especially those outside of the textbook, employed techniques and forms considered more basic and therefore more recognizable to developing (transitioning) writers. but what is basic? to what degree is the foundation-writing course obligated to take a student beyond what some call the formula essay? does the so-called formula essay do more harm than good for students? to address this question, i will examine a common component emphasized in the formula essay that is not always clearly reflected in professional essays used as teaching samples. this element is the topic sentence. crew (1987) points out that rules listed in writing texts and taught by teachers of composition often contradict the practices of professional writers (p. 346). to illustrate his point, he references braddock (1974), who analyzed 25 articles from magazines such as the atlantic and the new yorker regarding the frequency and placement of topic sentences. braddock’s essay famously estimates that only 13 percent of expository paragraphs he reviewed began with a topic sentence and that uses of topic sentences vary by author. many composition experts continue to connect braddock’s statistic to outmoded thinking regarding the necessity of topic sentences. however, reading braddock more closely makes his often-quoted statistic seem out of context. actually, his analysis leads him to the conclusion that composition teachers and writing handbooks should provide fuller disclosure about the lessons they teach and the contradicting practices of professional writers. in fact, as the passage below shows, braddock indicates that more use of topic sentences in the samples in his study would aid reader comprehension, and he advocates that composition teachers should continue to illustrate for their students the benefit of the topic sentence in learning to develop good paragraphs: in my opinion, often the writing in the 25 essays would have been clearer and more comfortable to read if the paragraphs had presented more explicit topic sentences. but what this study does suggest is this: while helping students use clear topic sentences in their writing and identify variously presented topical ideas in their reading, the teacher should not pretend that professional writers largely follow the practices he is advocating (braddock, 1974, p. 301). rather selective readings of braddock’s essay fail to acknowledge the narrowness of his target; he limited his study to articles in popular magazines. as popken (1987) points out, braddock’s analysis is instructive regarding the writing styles of general interest magazine authors, but their use of topic sentences – or lack of use – may indicate more about the interplay among topic, audience, and writing style than it does about the usefulness of the topic sentence. viewed from this perspective, braddock’s article describes the stylistic practices and preferences of professional writers and simply alerts teachers to reframe their instruction regarding the necessity of the topic sentence as a rhetorical tool. therefore, it is inaccurate to suggest that his study assails the topic sentence. however, since braddock’s article is often cited and the implications perhaps too broadly applied, popken’s article calls for further studies to correct misapplications of braddock’s conclusions. d’angelo (1986) responds to those who move past braddock and who actually campaign against the topic sentence as nearly useless in teaching composition. he counters with research in reading comprehension that demonstrates better student recall and efficiency when slade, j. r., jr. journal of the scholarship of teaching and learning, vol. 10, no. 3, november 2010. www.iupui.edu/~josotl 37 students read information organized with topic sentences and other structural techniques (p. 438). he concludes the following: if we base the teaching of writing on the way people actually write (i.e., on rhetorical performance), then the topic sentence will be of limited use in the teaching of writing, since many professional writers do not use topic sentences. but if we base our teaching on what people can accomplish with language (i.e., on rhetorical competence), as it seems to me 19th-century composition theorists did, then the topic sentence can be a useful resource that writers can turn to if the need arises. (d’angelo, 1987, p. 439) comments from students who participated in this study of student-authored essays illuminate a remnant of d’angelo: student writers long for instruction and instructional tools that emphasize immediate and practical uses. as has been pointed out, braddock’s research on the topic sentence limits its analysis to pieces from popular magazines, which led some to question whether its results generally applied to academic forms of writing such as those found in textbooks. smith (2008) returns to braddock’s influential research and re-examines his method as well as replications of it, concluding that braddock’s deduction does not generalize to forms of writing students most often encounter in courses outside of composition and literature classes. smith’s research analyzes journal articles in history and shows that 95 percent of what he calls discourse block units (one or more paragraphs that develop a sub-topic of the main topic) are controlled by an explicit topic idea. more to the point, his analysis finds that a topic idea appears at the beginning of the discourse block two-thirds of the time (p. 89). though smith (2008) applies the implications of his research primarily to developing reading skills, he does mention its applications to writing instruction (pp. 78-79). for a student, writing must be practical. while most students will agree in principle that their writing skills may matter some to their future career, finding a successful formula for college writing assignments supersedes four years hence. from this perspective, students hold greater appreciation for writing instruction that will help them achieve success in other courses. their responses in this study suggest that the student-authored essay as a teaching tool more effectively and efficiently aids the goal of transferring writing skills to other college courses. not only do students psychologically identify with student samples, they also can detect in them a formula they cannot always see in the highly stylized professional writing samples offered up in college readers and composition texts. a writing model derived from a “formula” is likely to remind composition teachers of the much maligned five-paragraph theme, which many instructors consider the result of a mechanical, stifling process that results in empty prose. however, the degree to which students are concerned with the freedom to experiment with various techniques and stylistic embellishments may depend somewhat on the preparedness of the individual student, and it seems reasonable that entering students would desire a basic set of writing tools applicable to most formal composition assignments. not all skills may require even distribution; the level of a student’s competency in a skill depends largely upon how essential it is to success in the discipline. the need for history majors to master math at the same level as chemistry majors is debatable. this is not to say that history majors are incapable of learning complex math or that chemistry majors cannot or should not become good writers. this merely implies that in some cases, proficiency may suffice for a skill that is tangential rather than central to success. average slade, j. r., jr. journal of the scholarship of teaching and learning, vol. 10, no. 3, november 2010. www.iupui.edu/~josotl 38 students need and desire writing models that clearly illustrate rhetorical tools and techniques reflected in learning goals, and the stylized presentations of professional writers often contradict the basics students need. think of it in sports terms. beginning amateur golfers who take lessons will encounter basics, not techniques intended to resemble the greatest golfers in the world. professional athletes, aided by some natural ability, have developed their talents well beyond rudimentary principles. put another way, professionals are so accomplished in the fundamentals of their craft that they can interpret and reconfigure the basics into a style of their own. style is often a seamless combination of small, imperceptible elements that only the trained eye can detect and truly appreciate. and it is style that marks the professional essays often assigned to beginning student writers. as jenkins (2010) aptly comments, “. . . some writers may, over time, move beyond formulas. they may develop the desire to explore meaning more deeply and discover for themselves how organization can derive organically from content. in fact, we have a name for such writers. we call them ‘professionals’” (para. 23). it is true that some students are accomplished enough in their writing to create clever metaphors and turn colorful phrases that tend to impress their teachers. effective essays, however, should not belong to the artistic alone; in fact, jenkins asserts that the formula essay is the right tool to develop good writers, and this seems the prevailing sentiment of freshmen i have taught. before jenkins, haluska (2006) enthusiastically endorsed the formulaic essay as an allpurpose tool for the college student. he argues precisely what students in my freshman composition classes beg for, which is a technique that will serve them effectively from course to course and from one semester to the next. he acknowledges the potentially reductionist nature of quantifying aspects of composition (the five-paragraph theme, for instance), which inadvertently influences some student writers to limit rather than enlarge how they think about and develop their topics. an acceptable trade-off is an efficient and effective tool adaptable to writing assignments common to most college coursework. v. implications for teaching and learning. some will likely argue that advocating that students emulate each other’s writing restricts rather than develops their abilities. capping student development is far from the intention; instead, results of this study show that skillful use of student-authored essays may do just the opposite. this study indicates merit in placing greater emphasis on student-authored essays as a writing tool, as student writers, especially average and marginal ones, need accessible models that clearly reflect the rhetorical elements and skills they are expected to learn. there are richer implications and lessons to take away from this study. first, the students’ feedback suggests that either category of essay deserves more sophisticated handling as a teaching tool. the well-crafted student-authored essay, especially one that illustrates the flexibility and the adaptability of the so-called formula essay, can serve as an all-purpose writing tool that can handle most writing jobs adequately for the two-year or four-year college student. broadly considered and taught well, the formula essay, as haluska (2006) points out, is flexible yet muscular enough “to accommodate reading reactions, term papers, doctoral dissertations, letters home pleading for money, and so forth” (p. 51). since students enter postsecondary institutions with the five-paragraph theme ingrained, it is reasonable to believe that an approach that can build upon what they already know offers some advantage. in other words, students who are unsure of themselves as writers find benefit in a recognized and repeatable approach. slade, j. r., jr. journal of the scholarship of teaching and learning, vol. 10, no. 3, november 2010. www.iupui.edu/~josotl 39 results from this study also imply that students consider peer-authored writing samples less intimidating; therefore, their own confidence as writers seems to receive a boost from exposure to writing by their peers. notably, though, this same cohort shunned in-class activities built around peer critiquing and tended to distrust their peers’ judgment of their writing. this seemed particularly true of better-prepared students whose reading and writing habits were advanced enough to discern the stylized ways in which accomplished writers use different techniques and rhetorical tools. perhaps the general lesson to take away from this study is the need for more deliberate planning in the use of both student-authored and professional essays as teaching tools. students in this study were more likely to discuss the strengths and weaknesses of a student-authored essay because they accepted without question that an essay by a nonprofessional contained room for improvement. on the opposite end, students in this study were more reticent in discussing strengths and weaknesses in professional essays. some students reasoned that professional essays equated to perfection and that only exceptional examples would warrant inclusion in a textbook. to a degree, this sentiment applies as well to student-authored essays that appear as samples in the textbook. therefore, it is worth noting that increased and more skillful use of student essays written by current and former students with no connection to professional documents may prove even more effective in achieving learning outcomes. to return to the general result of this study, the students surveyed preferred and benefitted more from the work of accomplished novice writers to that of professionals. the general implication may be that some students, especially those whose writing skills are marginal, find the work of their peers more recognizable and therefore more attainable in their own efforts. vi. limitations of study. sample size and population profile limit widespread implications of the results. this study included a small sample size limited to first-semester freshmen at north carolina a&t state university over two semesters. though frequency counts of responses are notable, a close examination of the written comments provided more substantive and supportive data than anticipated. the nature of the written responses encourages follow up and refinement of the study design to include formal interviews that might provide greater insight into students’ stated preferences for one type of essay sample over the other. as presented earlier, wide ranges of student preparedness and a voluntary system of placement in writing courses might also influence whether the results of this study can be generalized to other populations of students. vii. conclusion. this small study aimed to validate the degree to which first-year students at a public hbcu find two categories of writing samples useful as teaching tools. students in this study generally perceived professional essays as inaccessible and felt less capable of emulating professional models. as studies referenced in this paper show, professional writers often stylize their prose with techniques too advanced for the average and marginal writer to imitate with confidence. participants in this study characterized the student-authored essay as a more accessible teaching tool than the professional essay model and judged it more representative of learning goals and writing techniques taught in class. focused modeling of student-authored essays shows potential slade, j. r., jr. journal of the scholarship of teaching and learning, vol. 10, no. 3, november 2010. www.iupui.edu/~josotl 40 for strengthening learning outcomes in first-year writing courses, especially for average and marginal writers. references bauerlein, m. (2008). the dumbest generation: how the digital age stupefies young americans and jeopardizes our future (or, don't trust anyone under 30). new york: penguin. braddock, r. (1974). the frequency and placement of topic sentences in expository prose. research in the teaching of english, 8 (3), 287-302. childress, v. and southerland, j. (2008). the status of student learning outcome achievement. greensboro, nc: north carolina a&t state university. crew, l. (1987). rhetorical beginnings: professional and amateur. college composition and communication, 38 (3), 346-350. d'angelo, f. j. (1986). the topic sentence revisited. college composition and communication, 37 (4), 431-441. graves, j. l. (2006). university studies program 2005-06 annual report. greensboro, nc: north carolina a&t state university. haluska, j. c. (2006). in defense of the formula essay. academic questions, 20 (1), 46-55. jenkins, r. (2010, february 21). accordions, frogs, and the 5-paragaph theme. the chronicle of higher education. retrieved from http://chronicle.com/article/a-return-to-the-5paragraph/64255/. north carolina a&t state university. (2010). fact book 2010. greensboro, nc: north carolina a&t state university. retrieved from http://qed.ncat.edu/ir&p/availabl.htm. popken, r. l. (1987). a study of topic sentence use in academic writing. written communication, 4 (2), 209-228. smith, c. g. (2008). braddock revisited: the frequency and placement of topic sentences in expository prose. the reading matrix, 8 (1), 78-95. university of north carolina. (2009). fall 2008 average sat of first-time freshmen. chapel hill, nc: university of north carolina. retrieved from fred.northcarolina. edu/quickfacts/fall/sat086.pdf. 503 service temporarily unavailable 503 service temporarily unavailable nginx/1.14.1 503 service temporarily unavailable 503 service temporarily unavailable nginx/1.14.1 503 service temporarily unavailable 503 service temporarily unavailable nginx/1.14.1 503 service temporarily unavailable 503 service temporarily unavailable nginx/1.14.1 v10n1cardetti journal of the scholarship of teaching and learning, vol. 10, no. 1, january 2010, pp. 80 – 92. insights regarding the usefulness of partial notes in mathematics courses fabiana cardetti1, nirattaya khamsemanan2, and m. carolina orgnero3 abstract: note-taking is a widespread practice used by college students to record information from lectures. unfortunately, even successful students’ notes are incomplete and, therefore, may lack the potential to positively impact their academic performance. research suggests that instructors can help students improve their note-taking skills by using partial notes in their classes. the purpose of this exploratory study was to understand the potential usefulness of partial notes in mathematics courses. findings showed that students perceived partial notes as beneficial to their learning. course examination scores further confirmed that partial notes related strongly to high academic performance. keywords: note-taking, partial notes, mathematics, instructional strategy i. introduction. note-taking is a widespread practice used to record information in written form. this practice is very common among college students; in fact; palmatier and bennett (1974) found that 99% of the students they surveyed took notes during an instructor’s lecture. the practice of note-taking has been the subject of educational studies for over 75 years. in his seminal work of 1925, crawford analyzed the benefits of taking notes during a lecture versus listening without taking notes and concluded that note-taking itself enhances students’ examination performance. more recently, einstein, morris, and smith (1985) analyzed students’ notes and found that students could recall 40% of the material in their notes but only 7% of the material that was not in their notes. they also compared notes of successful and less successful college students and concluded that successful students included more of the main ideas in their notes than did less successful students. unfortunately, even successful students’ notes are incomplete and, therefore, may lack the potential to improve their academic performance. indeed, successful junior students record, at most, 70% of the critical ideas that are presented in lectures (kiewra, 1984), and first-year students include only 11% of such ideas in their notes (hartley and marshal, 1974). researchers have suggested several ways in which instructors can help students improve students’ notetaking skills, such as: lecturing at slower rates (peters, 1972); using verbal cues (titsworth and kiewra, 2004); providing students with lecture notes (kiewra, 1985a). kiewra encourages instructors to address directly the problem of students’ poor note-taking skills. he states, “teachers should be aware of students’ relatively incomplete note-taking behaviors, and, therefore, [should be] encouraged to provide learners with adequate notes for review” (kiewra, 1985a, p.77). this proviso applies to mathematics instructors as well as others, since note-taking directly affects students’ ability to follow and understand mathematics lectures. in this article, we 1department of mathematics, university of connecticut, storrs, ct 06269-3009, usa. fabiana.cardetti@uconn.edu 2present address: sirindhorn international institute of technology. thammasat university, thammasat-rangsit, thailand. 3center for continuing studies and department of educational leadership, university of connecticut, storrs, ct 06269-4056, usa. cardetti, f., khamsemanan, n., and orgnero, m. c. journal of the scholarship of teaching and learning, vol. 10, no. 1, january 2010. www.iupui.edu/~josotl 81 focus on assisting students by providing them with a specific format of partial rather than complete lecture notes. the present work constitutes the first step towards understanding the full potential of partial notes in fostering significant learning in mathematics courses. ii. framework. the literature cited above suggests that instructors should be proactive in student note-taking efforts, for example, by providing lecture notes. studies have shown that when students receive complete lecture notes from their instructors, they achieve higher test scores than when they rely only on their own notes (kiewra, 1985c; maqsud, 1980). it may seem natural, then, to think that instructors can help their students by making copies of lecture notes available to them. however, there are some caveats regarding providing students with complete lecture notes. students may feel that when they receive the notes they do not need to take notes themselves or be attentive in class (pardini, domizi, forbis, and pettis, 2005). in some extreme cases, if students receive complete notes, they may not feel the need to attend lectures at all, thus promoting absenteeism (potts, 1993; russell, caris, harris, and hendricson, 1983). to address these caveats, instructors can provide partial notes instead of full notes, thus requiring students to be present in class to complete their notes; moreover, students have to be attentive to complete their notes correctly, which gives them a reason to attend classes. these partial notes generated by the instructor can take the form of outlines or graphic organizers which, depending on the amount of information they contain, can be categorized as complete, partial, or skeletal notes (katayama and robinson, 2000). several studies have been conducted to compare the effects of partial notes and full notes on students’ achievement. for example, annis (1981) compared examination performance amongst students using their own personal notes, the instructor’s complete notes, or partial notes prepared by the instructor. annis found that students using personal or partial notes scored significantly higher on the essay items of the examination than did students using complete notes. in another study, russell, caris, harris, and hendricson (1983) analyzed the performance of medical students on multiple-choice questions and on critical-thinking questions. students were provided with complete notes, or partial notes, or skeletal notes. russell et al. found that the group of students using partial notes scored significantly higher than the other two groups did, particularly on the critical thinking questions. they concluded that “ the partial handout was the best compromise between the skeletal handout, which promotes alertness during the lecture but may be less valuable for review several months later, and the comprehensive handout, which encourages passivity in class” (russell et al., 1983, p. 636). in a more recent study, cornelius and owen-deschryver (2008) examined the impact of partial and full notes on students’ learning outcomes. students using partial notes performed significantly better than students using full notes on examinations taken towards the end of the semester and on the cumulative final examination. in particular, students using partial notes performed better on cumulative conceptual questions (i.e., questions that demand understanding beyond just the definition of a concept and that require the elaboration of information). these studies suggest that students benefit from the use of partial notes, especially for courses that emphasize problem-solving and critical thinking. reliable lecture notes are especially important in mathematics classes because of the large amount of challenging content that is covered in each class and the cumulative nature of mathematics that requires full understanding of an idea before moving on to the next. in addition, cardetti, f., khamsemanan, n., and orgnero, m. c. journal of the scholarship of teaching and learning, vol. 10, no. 1, january 2010. www.iupui.edu/~josotl 82 course examination questions consist primarily of critical thinking questions that require a solid understanding of the concepts covered and the relationships among them. while the studies previously reviewed were conducted in fields such as psychology, medicine, and education, our review of the literature revealed no studies about partial notes in mathematics classes. however, the literature does suggest that partial notes could benefit students in mathematics courses, and we did find two studies about teaching foundational skills that would prepare math students to deal effectively with partial notes. one of those studies (backman, 1994) was an action research project by a high school mathematics teacher, designed to help students learn effective notetaking for studying, organizing, and remembering information. as a result of this project, students were able to refine their note-taking skills. the other study (eades and moore, 2007) examined the benefits of a systematic note-taking procedure in a developmental mathematics course. results from surveys and instructors’ observations revealed that the system increased student understanding and motivation. the present article attempts to further the understanding of this instructional strategy in mathematics courses. specifically, this work explores the potential usefulness of partial notes in mathematics courses. we base our findings on the analysis of students’ perceptions regarding the use of partial notes and on students’ performance on course examinations. the article concludes with a discussion of the findings, a summary of key recommendations for practice, limitations and future research aimed at learning more about the effectiveness of this instructional technique. iii. method. a. context and sample. one of the authors of this paper taught a calculus course at a large northeastern public university in the united states of america. the framework of the instructor’s lectures was based on her typed personal notes. these notes consisted of complete statements of theorems, explanations, and full solutions of examples. in the fall, 2007 semester, the instructor decided to try a different instructional strategy that would promote active engagement of students with the material. to that effect, she introduced partial notes that she handed out to students at the beginning of every class. partial notes (pn) are typed versions of the instructor’s personal notes. they are called partial because the solutions to examples are omitted and blank spaces are left, so that the students can add their own solution to each of the examples within the pn. the instructor followed this format of pn for her classes because she felt this would be beneficial to her students. certainly, other pn formats are also possible (e.g., leave blanks for definitions, names of properties, etc.) an example of a four pages long pn is provided in figure 1. since partial notes saved the instructor some lecture time because she did not have to write examples or problems on the board, she used the extra time to engage students in interactive in-class activities without sacrificing the amount of material she had to cover. for example, once the instructor had worked through enough examples on the board, students were asked to solve a few extra examples on the partial notes on their own. the instructor also asked volunteers to solve each problem on the board. these volunteers were given extra credit points that counted towards the cardetti, f., khamsemanan, n., and orgnero, m. c. journal of the scholarship of teaching and learning, vol. 10, no. 1, january 2010. www.iupui.edu/~josotl 83 1 2 3 4 figure 1. example of a four-page long partial note. cardetti, f., khamsemanan, n., and orgnero, m. c. journal of the scholarship of teaching and learning, vol. 10, no. 1, january 2010. www.iupui.edu/~josotl 84 weekly quiz score for the course. the students’ perception of this instructional strategy (use of pn) became apparent to the instructor when she read the students’ feedback on the mid-semester and end-of-semester open-ended questionnaires of fall 2007. this feedback was gathered as part of standard educational practices for the course. later on, these preexisting aggregated data became the basis of the inquiry that drove this exploratory study. the instructor had taught the same course in the fall, 2006 semester. that year she used her full personal notes to guide her instruction but did not provide students with either full or partial notes. in the 2007 semester the instructor used the same personal notes to guide her teaching and to prepare pn. thus the material covered in the 2006 semester and the 2007 semester was identical. other aspects that were identical to both courses, in addition to the material and the term of the year, were the days and times of the week in which the lecture took place, the length of each class meeting, and the total number of weeks in the semester. furthermore, there was diversity of ethnic backgrounds with a predominance of caucasian students in both semesters. the main difference between the courses was that in 2007 the students had access to partial notes. b. research questions and data collection. the aim of this exploratory study was to assess the potential usefulness of partial notes in a mathematics course. specifically, the research questions were: 1. among students in the fall 2007 course, what themes characterized their perceptions of the instructor-generated partial notes? 2. to what extent did course examination scores differ between students who did not have access to partial notes and students who did have access? qualitative data were used to answer the first research question. these data were collected anonymously from questionnaires that students completed in the middle and end of the semester in the fall 2007. further details about these questionnaires can be found below. quantitative data were collected from the students’ course examination scores to address the second research question. we compared the examination scores from fall 2006 and fall 2007. for the purposes of this study, scores were imported from the grade books to a separate file, students’ names were removed, and scores were recorded in increasing order for each semester. thus all the quantitative data were analyzed anonymously and in aggregate. further details about the examinations can be found below. mid-semester questionnaire. seven weeks into the semester and 2 weeks after the first examination, students completed an anonymous mid-semester questionnaire. the mid-semester questionnaire was a short, open-ended questionnaire developed by the instructor. its purpose was to obtain anonymous feedback from the students that would help improve their learning experience in the remaining weeks of the semester and to obtain feedback about her teaching effectiveness. students completed the questionnaires in the first 15 minutes of the lecture while the instructor was out of the room. the items on the mid-semester questionnaire were as follows: 1. what is going well for you in lectures? 2. what is not going well for you in lectures? 3. what can i do to further help your learning in this class? 4. what can you do to make the lectures more productive for yourself? 5. other comments/concerns/suggestions. cardetti, f., khamsemanan, n., and orgnero, m. c. journal of the scholarship of teaching and learning, vol. 10, no. 1, january 2010. www.iupui.edu/~josotl 85 end-of-semester questionnaire. as is customary in many institutions, at the completion of each semester students were asked to complete an anonymous official student evaluation of faculty. at our institution this evaluation takes place each spring and fall semester. this evaluation consists of two pages: the first page is a 10-point-scale quantitative form to numerically rate the instructor’s effectiveness; the second page is an open-ended questionnaire wherein students can provide a short narrative of the instructor’s performance. for the purposes of this study, we focused on the second page questionnaire that consisted of the following items: 1. what was the most positive aspect of the way in which this instructor taught this course? 2. what can this instructor do to improve teaching effectiveness in the classroom? the results are used by the instructor for improving teaching performance and for promotion, tenure, and reappointment of the instructor. each instructor receives the results after the semester has ended and grades have been distributed. examinations. three examinations were administered in-class in both the fall 2006 and the fall 2007 semesters. in each semester, the first examination took place during the sixth week of classes, the second examination during the eleventh week of classes, and the cumulative final examination during the official finals week at the end of the term. the examinations consisted of computational questions as well as conceptual questions. only the final examination was cumulative. in 2007, these questions were only slightly modified in content to avoid concerns of contamination across years. additionally, we compared the scores of the two groups on the mathematics section of the scholastic aptitude test (sat) to control for general mathematical ability prior to taking the course. c. data analysis. midand end-of-semester questionnaires. data analysis consisted of several steps. first, each of the authors coded the data separately following the constant comparative method (corbin and strauss, 2008) without seeking to build theory. the constant comparative method “involves continually comparing one unit of data with another in order to derive conceptual elements” (merrian, 2002, p. 8). next, we met, compared, and refined codes. patterns that repeated were collapsed or enhanced to form categories until we identified emerging themes (miles and huberman, 1994). to add rigor to our work, we generated individual analyses, and met periodically to discuss our findings. disagreements about codes, categories, and themes were resolved during our meetings, creating an iterative process of dialogic collaboration (paulus, woodside, and ziegler, 2008). examinations. examination scores of the different groups were compared to determine if significant statistical differences existed. ideally, the authors would have preferred to carry out an ancova with the sat score as the covariate, but this was not possible because the data were only available in aggregate. in lieu of such analysis, tests of equal variances, homogeneity, and skewness were performed to justify application of a t-test that measured examination scores differences. finally, we also performed a t-test to determine if significant statistical differences existed between students’ sat scores. cardetti, f., khamsemanan, n., and orgnero, m. c. journal of the scholarship of teaching and learning, vol. 10, no. 1, january 2010. www.iupui.edu/~josotl 86 iv. findings. a. qualitative data from open-ended questionnaires. the analysis of the responses to open-ended questionnaires was performed to understand students’ perceptions of partial notes. four themes emerged from the analysis: (1) students defined partial notes in their own words. (2) partial notes allowed students to follow the lecture attentively. (3) students perceived the lecture structure as beneficial. (4) students perceived partial notes as a helpful aid for studying. each of these themes is described in detail below. students defined partial notes in their own words. the instructor made no attempt to call students’ attention to partial notes because the objective of the course was to help students learn the fundamental mathematical concepts of the course. consequently, students were never given a specific name for the partial notes. in spite of that, students found ways to define partial notes, using their own words to comment on the course strength both in the middle of the semester and again at the end of the semester when they responded to the questionnaires. for example, many students referred to partial notes as “handouts,” or “packets.” a lower number of students referred to them as “notes.” among the comments made were the following, “the handouts are extremely helpful,” “the packets are a good way of presenting the material…,” and “i really liked how we got a handout every class with all the notes and examples. it really helped me.” overall, the students used their own words to refer to partial notes in their comments. partial notes allowed students to follow the lecture attentively. a key benefit of partial notes is that it allowed students to follow along with the lectures. when students did not need to take notes feverishly, they could concentrate on the highlights of the lecture and devote their efforts to listening attentively to the instructor’s explanations. many students referred to this aspect of partial notes. for example, one student mentioned that the partial notes were effective in keeping up with the instructor: “i like following along in the lesson through the packets. it is quite effective.” another student explained the effect that having the definitions and examples already in written form had on listening to the lecture: “i like how you give us everything (definitions and examples). it makes it easier to follow along and listen to your examples.” when students had to take full lecture notes on their own, their attention was divided between listening, processing information, and note-taking. on the other hand, with the aid of partial notes, students’ attention was focused on what the instructor was saying at the moment, allowing for meaningful processing to take place. students followed the lecture attentively in two distinct ways. some students liked partial notes because they could listen to the lecture attentively without the distraction of taking notes, as the following student explained, “i really like that you give us handouts of class in a worksheet so that we can concentrate more than trying to scribble every word down.” some of these students also commented on the fact that critical lecture ideas were readily available on the partial notes, “the handouts were a big help because they allowed students to focus on the main points of the lecture rather than waste time doing extra writing.” other students liked using partial notes to follow the lecture because they could rephrase the instructors’ comments or explanations in their own words to complement the content of the provided notes, “the handouts are very useful. it helps to write down notes on the papers because i can add my own notes, if needed. but i am not worrying about writing down definitions.” similarly, another student mentioned, “i like the fact that we get a handout for each class that we can follow along with but still fill in the examples.” using partial notes, students were able to either put down their pencils cardetti, f., khamsemanan, n., and orgnero, m. c. journal of the scholarship of teaching and learning, vol. 10, no. 1, january 2010. www.iupui.edu/~josotl 87 and focus on listening to the instructor’s explanation, or they were able to combine attentive listening with meaningful note-taking. students perceived the lecture structure as beneficial. a few students did not refer to partial notes explicitly but alluded to the structure of the class when they reflected upon the positive aspects of the course. for example, one student liked having the opportunity to independently attempt problems before seeing the instructor’s explanation, “dr k. provides thorough explanations in class and also allows us to work at problems prior to explaining them on the board for practice.” another student reflected upon the uncluttered nature of partial notes and on the structure of the class; and in particular alluded to the extra points quiz policy described in the context section above, “gave good explanation of problems without cluttering examples with unneeded information. also gave great quiz opportunities such as examples on the board.” the use of partial notes allowed the instructor to structure the lecture in such a way that all of the above was possible (i.e., clear presentation of topics, extra time to solve problems, etc.), so that she could incorporate extra activities within the allocated time. students perceived partial notes as a helpful aid for learning and studying. some students pointed out that learning the lecture material was facilitated by partial notes. for these students, partial notes helped them learn the concepts covered, “going through the section of focus with the printed out guide is very helpful for me to learn the material.” students also found partial notes useful in the reviewing process after the lecture, when they were studying for examinations, “the handouts each week were very helpful because they are easy to review for test time.” students also mentioned another key skill for learning and studying: organization, “handout materials were wonderful. kept me organized.” overall, students used partial notes as an aid in the cognitive processing stages (learning) and in the review or product function of notetaking (studying) (suritsky and hughes, 1991). b. quantitative data from examinations. for the analysis of the scores, the alpha level was set at .05. effects that produced p values between .05 and .10 are reported as marginally significant effects. first, we looked at the differences between the two groups (2006 and 2007), comparing the mean scores on the course examinations. table 1 shows the descriptive statistics for all examination scores across the two groups. students using partial notes scored significantly higher (m=41.43, sd=6.64) on the first examination than did those without access to partial notes (m=34.67, sd=7.94), t(52)=3.50, p<0.001. the standardized mean difference effect size4 (essm) between the two groups on the first examination was very large (essm=0.90). the analysis of the scores on the second examination indicated that the students using partial notes scored marginally higher (m=38.86, sd=9.13) than did those students with no access to partial notes (m=34.59, sd=7.98), t(54)=1.61, p=0.056. the standardized mean difference effect size between the two groups on the second examination was moderately small (essm=0.41). finally, the results indicated that students using partial notes scored significantly higher on the final examination (m=58.83, sd=14.52) than did those without access to partial notes (m=34.59, sd=7.98), t(55)=2.62, p=0.014. for the final examination scores, the standardized mean difference effect size between the two groups was moderately large (essm=0.67). 4 notation for effect size index (essm) from practical meta-analysis, by m.w. lipsey and d.b. wilson, 2001, thousands oaks, ca: sage. cardetti, f., khamsemanan, n., and orgnero, m. c. journal of the scholarship of teaching and learning, vol. 10, no. 1, january 2010. www.iupui.edu/~josotl 88 second, we assessed the statistical significance of differences in the mean score of the mathematics section of the sat between the two groups. the results of the t-test indicated that the group using partial notes scored significantly lower (m=572.67, sd=76.15) than the group that did not have access to partial notes (m=606.45, sd=73.19), t(59)=1.77, p=0.04. there was a medium standardized mean difference effect size in sat scores between the two groups (essm=0.45). table 1. descriptive statistics for examination scores across groups. without partial notesa with partial notesb examination m sd m sd examination 1* 34.67 7.94 41.43 6.64 examination 2c* 34.59 7.98 38.86 9.13 final examination * 49.7 14.53 58.83 14.52 an=30. b n=29. c two students and one student were absent from each group respectively. *p≤ 0.05 v. discussion. the analysis of the students’ comments on the questionnaires illuminated how students reacted to instructor-generated partial notes. in fact, the qualitative analysis showed that students identified partial notes as a key factor to aid their learning. the instructor did not formally introduce the concept of partial notes to her students. she used partial notes as a way of presenting the material and making sure that students would be engaged with it. what became surprising to her was that students perceived she was using an instructional technique that was making a difference in their learning. thus, students indicated they had noticed this instructional technique by using a variety of names to describe partial notes in the questionnaires they completed in the middle and at the end of the semester. the findings highlighted the importance students attributed to follow along with the lecture. as williams and eggert (2002) pointed out “ attempting to record all the details could detract from one’s understanding of the main idea(s). notetakers can miss main points while recording minutia.” (p.177). partial notes address students’ preferences to process information by listening attentively to the instructor’s presentation without worrying about recording every detail and instead focusing on the information being presented. the evidence also revealed that students perceived the opportunity to explore the concepts in class on their own as beneficial to their learning. research consistently supports the effectiveness of student engagement on a broad range of learning outcomes. astin (1993) reported that student involvement is one of the most important predictors of success in college. using partial notes, the instructor gains considerable time by not having to write everything on the board (e.g., all the definitions, examples, or statements). that time can be used to engage students in other activities like having students work on problems independently or showing their solutions on the board. such activities promote student engagement and involvement with their own learning (smart and csapo, 2007). the final point that the findings suggested is that students also relied on partial notes to learn the material and to review for examinations. we speculated that students’ familiarity with partial notes increased the efficiency of the review process. these conclusions are consistent with results from research studies that confirm the significant correlations between reviewing cardetti, f., khamsemanan, n., and orgnero, m. c. journal of the scholarship of teaching and learning, vol. 10, no. 1, january 2010. www.iupui.edu/~josotl 89 notes and academic achievement (hartley, 1983; kiewra, 1985b). therefore, an effort should be made to make students aware of the benefits of reviewing the notes in preparation for examinations. to summarize, the themes that emerged from the analysis of the qualitative aggregate data are taken as an indication that partial notes may have played a role in students’ improvement of learning. this evidence was further confirmed with the analysis from the quantitative data. results from the comparison of the quantitative data helped us assess differences between the students who did not have access to partial notes (fall 2006) and students who did have access (fall 2007). the mean scores for all the examinations in the course were consistently higher for the group using partial notes. statistical analysis of the first examination scores revealed that the students using partial notes outperformed the group that did not use partial notes. results were not as promising for the second examination, for which the higher mean in the scores of the partial notes group was only marginally significant compared to the group that did not use partial notes. although we do not have a specific explanation for this result based on the available data, we speculated that overconfidence, because of the very high scores on the first examination, may have been somewhat counterproductive in the students’ preparation for the second examination. nonetheless, by the end of the semester, the partial notes group outperformed the group that did not use partial notes in the final examination. given the cumulative nature of the final examination, this evidence suggested that the partial notes group overall performed significantly better academically. these results suggested that the use of partial notes might be a factor in improving student academic performance. finally, analysis of students’ prior mathematical knowledge based on scores on the mathematics section of the sat showed that students using partial notes performed significantly lower than the other group. therefore, high academic performance of the partial notes group cannot be attributed to superior mathematical ability. we concluded that participation in the course using partial notes related strongly to higher academic performance. vi. conclusion. the results of this exploratory study suggest that providing students in mathematics classes with partial notes is beneficial to their learning. the students’ voluntary comments about partial notes indicated that they appreciated the strategy. students emphasized that partial notes were beneficial for keeping them engaged with the lecture, providing opportunities to participate actively in class, and to effectively review for examinations. in addition, the comparison of the examination scores suggests that partial notes might have been a factor contributing to the improved student performance. a. implications for practice. in this work we presented some of the benefits of providing students with one format of instructor-generated partial notes in mathematics courses. for the instructor who ordinarily types the lecture notes, this strategy could be easily implemented in his/her classes. for the skeptical instructor, the results of this exploratory study should be a starting point to show that the benefits might be worth the extra effort involved. the benefits of partial notes go beyond the specifics addressed in this study. partial notes may have the same benefits, detailed in the previous section, in other mathematics courses and in cardetti, f., khamsemanan, n., and orgnero, m. c. journal of the scholarship of teaching and learning, vol. 10, no. 1, january 2010. www.iupui.edu/~josotl 90 many science courses. instructors can adjust the format of the notes to the students’ levels and experience and they may find outlines to be more appropriate for upper-level courses than partial notes as described in this article. partial notes may be particularly beneficial for coordinators of multi-section courses, ensuring uniform coverage of the material across all sections. coordinators can use partial notes to help new teaching assistants and foreign instructors who might not be familiar with a particular educational system. partial notes can also enhance the learning experience of learners whose native language may be different from the language of instructors, and of students with disabilities who are easily distracted and might miss important lecture ideas. students who find the pace of the class too fast and students with bad handwriting may also benefit from the use of partial notes. b. limitations and future research. as an exploratory study, this work offered valuable insights regarding partial notes, but its limitations need to be addressed. one major limitation of this study was that we worked with aggregated data that were analyzed after the semester concluded. therefore, there were no follow-up questions or interviews that could allow for a deeper understanding of what students meant on the open-ended questionnaires. while partial notes were the only instructional difference the instructor introduced in the fall 2007 semester (compared to fall 2006), the researchers could not entirely attribute students’ improvement of their performance solely to the presence of partial notes. therefore, the researchers are cautiously optimistic about the success reported here because of the exploratory nature of this study. the findings of this exploratory study revealed only the tip of the iceberg of this promising strategy in mathematics courses. further research should delve deeper into the students’ pn preferences in mathematics classes as well as examine other formats of partial notes that would eventually lead students to take their own efficient notes. since this instructional strategy is not content-dependent but cognitive-dependent, we feel confident that readers teaching other disciplines will find this article relevant to their contexts. references annis, l. f. (1981). effect of preference for assigned lecture notes on student achievement. journal of educational research, 74, 179-182. astin, a. (1993) what matters in college? four critical years san francisco, ca: jossey-bass. backman, c. a. (1994). finding and effective note-taking system for math students. teaching and change, 2 (1), 73-88. corbin, j., and strauss, a. (2008). basics of qualitative research: techniques and procedures for developing grounded theory, (3rd ed.). thousand oaks, ca: sage publications. cornelius, t. l., and owen-deschryver, j. (2008). differential effects of full and partial notes on learning outcomes and attendance. teaching of psychology, 35 (1), 6-12. crawford, c. c. (1925). some experimental studies of the results of college note-taking, journal of educational research, 12, 379-386. cardetti, f., khamsemanan, n., and orgnero, m. c. journal of the scholarship of teaching and learning, vol. 10, no. 1, january 2010. www.iupui.edu/~josotl 91 eades, c., and moore, w. m. (2007). ideas in practice: strategic note taking in developmental mathematics. journal of developmental education, 31 (2), 18-26. einstein, g. o., morris, j., and smith, s. (1985). note-taking, individual differences, and memory for lecture information. journal of educational psychology, 77, 522-532. hartley, j. (1983). notetaking research: resetting the scoreboard. bulletin of the british psychological society, 36, 13-14. hartley, j., and marshall, s. (1974). on notes and note-taking. universities quarterly, 28, 225235. katayama, a. d., and robinson, d. h. (2000). getting students `partially' involved in notetaking using graphic organizers. journal of experimental education, 68 (2) 119-124. kiewra, k. a. (1984). implications for notetaking based on relationships between notetaking variables and achievement measures. reading improvement, 21, 145-149. kiewra, k.a. (1985a). learning from a lecture: an investigation of notetaking, review, and attendance at a lecture. human learning, 4, 73-77. kiewra, k. a. (1985b). providing the instructor's notes: an effective addition to student notetaking. educational psychologist, 20 (1), 33-39. kiewra, k. a. (1985c). students' note-taking behaviors and the efficacy of providing instructor's notes for review. contemporary educational psychology, 10 (4), 378-386. lipsey, m.w., and wilson, d.b. (2001). practical meta-analysis. thousand oaks, ca: sage. masqud, m. (1980). effects of personal lecture notes and teacher-notes on recall of university students. british journal of educational psychology, 50, 289-294. merriam, s. b., and associates (2002). qualitative research in practice: examples for discussion and analysis. san francisco, ca: jossey-bass. miles, m., and huberman, m. (1994). an expanded sourcebook: qualitative data analysis, (2nd ed.). thousand oaks, ca: sage. palmatier, r. a., and bennett, j. m. (1974). notetaking habits of college students. journal of reading, 18 (3), 215-218. paulus, t., woodside, m., and ziegler, m. (2008). extending the conversation: qualitative research as dialogic collaborative process. the qualitative report, 13 (2), 226-243. pardini, e. a., domizi, d. p., forbes, d. a., and pettis, g. v. (2005). parallel note-taking: a strategy for effective use of webnotes. journal of college reading and learning, 35, 38-55. peters, d. l. (1972). effects of note-taking and rate of presentation on short-term objective test performance. journal of educational psychology, 63, 276-280. cardetti, f., khamsemanan, n., and orgnero, m. c. journal of the scholarship of teaching and learning, vol. 10, no. 1, january 2010. www.iupui.edu/~josotl 92 potts, b. (1993). improving the quality of student notes. washington, dc: office of educational research and improvement. (eric document reproduction service no ed 366 645). russell, i. j., caris, t. n., harris, g. d., and hendricson, w. d. (1983). effects of three types of lecture notes on medical student achievement. journal of medical education, 58, 627-636. smart, k. l., and csapo, n. (2007). learning by doing: engaging students through learnercentered activities. business communication quarterly, 70 (4), 451-457 suritsky, s. k., and hughes, c. a. (1991). benefits of notetaking: implications for secondary and postsecondary students with learning disabilities. learning disabilities quarterly, 14 (4), 7-18. titsworth, b.s., and kiewra, k. a. (2004). spoken organizational lecture cues and student notetaking as facilitators of student learning. contemporary educational psychology, 29, 447461. tomastik, e. c. (2004) calculus: applications and technology (3rd ed.). belmont, ca: thompson brooks/cole. microsoft word venglarfinal the journal of scholarship of teaching and learning, vol. 7, no. 1, may 2007, pp. 64-76. case-based ethics education in physical therapy mollie venglar and michael theall1 abstract: physical therapist students often think ethics content to be less relevant than other course material. the purpose of this study was to assess whether changing from lecture to case-based method, would impact ethics awareness and integration. in focus groups, students in the case-based course reported greater perceived value of the ethics content and felt that the material was easier to integrate into practice, while students in lecture-based course reported that content should be compressed into a shorter period of time and did not integrate it as effectively. the model was also effective in improving critical thinking in clinical practice situations. key words: ethics; case-based; physical therapy. i. background and purpose. in physical therapist education, instructors teach clinical skills by dealing with tissue (such as muscle, ligament, tendon, etc.), diagnosis, or through the use of cases. students are taught how to evaluate a problem, determine the physical therapy diagnosis, and plan a treatment appropriate to the diagnosis integrating the psychosocial, past medical history, financial, and pharmacologic co-variants in the plan of care. carry-over of the didactic instruction of clinical skills content from the academic setting to the clinic is evident through performance assessment during clinical education internships and is required to pass the physical therapist education program. clinical skills competencies are emphasized through practical examinations, clinical education, and clinical performance evaluation tools (anonymous. 1997). traditionally, ethical issues are not integrated into the teaching of the clinical decisionmaking process involved in patient care. the american physical therapy association (apta) has indicated some concern about this aspect of professionalism as the physical therapy profession completes the transition to the direct access role. direct access allows individuals to seek physical therapy services without physician referral. in the apta statement on professionalism, ethical consideration of patient care is prevalent, thus indicating that ethics education and the carry-over of that knowledge to the clinical setting is vital in professional clinical practice (american physical therapy association, 2003). the realm-individual process-situation (rips) model of ethical decision-making (swisher ll, arslanian la, & davis cm, 2005) was designed to assist physical therapist clinicians in identifying clinical situations that could include ethical issues. the model directs the clinician to determine the party with the greatest responsibility for the situation, determine the moral background, and determine the nature of the situation. in doing so, the clinician is better able to understand the potential stake holders of any given ethical situation. the clinician is then empowered to direct the situation to the primary stakeholder and facilitate a solution. table 1. the realm-individual process-situation model of ethical decision-making. 1 department of physical therapy, youngstown state university, one university plaza, youngstown, oh 44555, mcvenglar@ysu.edu and department of teacher education, mtheall@ysu.edu venglar, m., and theall, m. the journal of scholarship of teaching and learning, vol. 7, no. 1, may 2007. 65 (swisher ll et al., 2005) realm individual process situation individual organizational/institutional societal moral sensitivity moral judgment moral motivation moral courage issue or problem dilemma distress temptation silence the rips model is based on the work of glaser,(glaser jw, 1994; glaser jw, 2005) rest and narvaez,(1994), rest, narvaez, bebeau, and thoma (rest jr & narvaez d, 1994; rest jr, narvaez d, bebeau mj, & thoma sj, 1999) purtillo (2005) and kidder (1995). since it is designed for clinical use, the model is appropriate to use in the academic education of entry-level physical therapists. by using cases or scenarios, much like those in the clinical skills content, an instructor can use the rips model as a viable method for teaching ethics content. the rips model directs the student to evaluate the situation, diagnose the components of the situation, identify the stakeholders, and plan necessary action as appropriate; the same steps taken in therapist intervention of a physical problem. an example of how the rips model can be used in provided in the appendix. lecture-style teaching has been criticized for producing passive learners and preventing critical thinking in the classroom (limbach bj & waugh wl, 2005). in contrast, many medical education studies have reported the positive impact of using cases to help students learn course content (hudson jn & buckley p, 2004; jonassen dh & hernandez-serrano j, 2002; keefer m & ashley kd, 2001; mcginty sm, 2000; triezenberg hl & mcgrath jh, 2001). hudson and buckley (2004) studied the perception of case-based teaching in the physiology curriculum for pre-medical students. they found that the case-based method increased confidence of the students when they moved to the clinical courses. the case-based method allowed the students to study physiology in reference to hypothetical patient scenarios, thus creating a non-threatening environment in which the students could attempt to solve the case without fear of consequence (hudson jn et al., 2004). keefer and ashley (keefer m et al., 2001) reported the comparison of student response to ethicists’ responses to ethical problems. they determined that the student responses were based on common morality; the ethicists’ responses were based on professional morality. although common morality is no less important than professional morality the lack of knowledge of moral issues that relate to the professional world limited the students’ abilities to perceive the complete threat and the wide realm of potential solutions (keefer m et al., 2001). without exposure to professional ethics, in a manner that expresses the importance of professional ethical decisionmaking, students can not be expected to grasp the variability of solutions. triezenberg and mcgrath (triezenberg hl et al., 2001) report students’ perceptions following an applied ethics course in which the primary teaching method was through the use of narratives. the authors reported that students perceived the method in a positive manner and that narratives enhanced their learning of the integration of ethical theory and professional behavior (triezenberg hl et al., 2001). historically, physical therapist students are taught ethics content in a remote, nearly third person, sense. unlike the physical intervention and client management content, ethics content is not treated as relevant to daily physical therapy practice. the purpose of this paper is to describe the outcomes of a teaching methodology change in an ethics class in a physical therapist education program, and the effects of that change on perceived value of ethics education venglar, m., and theall, m. the journal of scholarship of teaching and learning, vol. 7, no. 1, may 2007. 66 following subsequent clinical education for master’s level students in a physical therapy program. ii. method/model descriptions and evaluation. the ethics curriculum at the author’s university is half of a two semester-hour course entitled “legal and ethical issues in pt”. the course has two instructors, one for the primary legal content, and one for the primary ethics content. traditionally, the ethics content was taught in a lecture format. the students were assessed via a cumulative course examination (both legal and ethics content) and a paper/presentation based on a directed interview with a clinician. the lecture format for the ethics content was abandoned by the instructor, and a casebased format was adopted by the same instructor using the same topics as in the traditional lecture format. in addition, the traditional textbook was abandoned and replaced with popular articles that were relevant to the topics discussed in the class. eight key aspects of professional ethics were chosen on which to base the case discussions. prior to attending class all students were provided with an article to read that incorporated a case related to the topic of discussion for that day. at the start of class the topic was briefly presented and the students were asked to identify and discuss the key aspects of ethics most prevalent in the article. the students were also asked to discuss the realm, individual process, and situation most prevalent in the article as defined by the rips model. articles were chosen based on relevance to the topic of discussion for that day, and relevance to a health care professional. the articles were not specific to physical therapy so as to provide students with a more global view of ethical concerns in the health care environment as well as to prepare the students for potential participation in a wide variety of ethics-based discussions later in their professional roles. additionally, each student completed two narrative papers on articles discussed during the course. the papers were designed to allow each student to express his/her opinion, with appropriate defense of his/her opinion, prior to initiating in-class discussion. in the paper, as in class discussion, the student was required to identify the aspects of professional ethics most prevalent in the article as well as the realm, individual process, and situation. the student was then required to defend his/her choices of the above in the paper, quoting sources as appropriate. finally, the student was asked to provide insight for how he/she, as a physical therapist, might be impacted by the situation, and thus how he/she would handle the situation. in addition, cases from concurrent clinical skills courses were used in the discussion of the daily topic. the use of cases from other courses allowed for the transfer of the ethics topics to clinical skill-related courses with the emphasis that ethics transcends the defined ethics class and is truly important in everyday clinical practice. assessment of student performance in the course was made via a midterm and final examination (both legal and ethics content), two narratives, and the paper/presentation of a directed clinician interview. iii. evaluation methods. in the summer following presentation of the ethics curriculum the students at the authors’ university embark on their second clinical education experience. students from the final year of the traditional lecture format, and students from the first year of the case-based format were asked to participate in focus group discussions following the clinical education experience. the venglar, m., and theall, m. the journal of scholarship of teaching and learning, vol. 7, no. 1, may 2007. 67 focus groups were run by an individual, unrelated to the department of physical therapy, and experienced in educational methodology and group assessment. the instructor for the class was not present. answers to the focus group questions were presented to the instructor/author in aggregate format by the individual who performed the focus group assessments. because the ethics content is half of the “legal and ethical issues in pt” course, students were asked to reflect on the course as a whole with particular attention to the ethics content. the answers of the two focus groups were compared to determine if the change of curricular format impacted the perceived value of the material presented in the ethics class as well as increased awareness or integration of ethics knowledge during the subsequent clinical education experience. focus groups were used because qualitative methods (patton, 1990) were expected to be more effective for exploration of attitudes and opinions in a situation where it was not possible to collect on-site, quantitative data, and where implementation of ethical principles and behaviors first required progressive gains in the affective domain (krathwohl, bloom and masia, 1956). the incorporation of ethical principles into professional practice can be expected only if students have reached levels three through five of the affective taxonomy (valuing, organization, and characterization by a value or value complex). in other words, ethical values will be exhibited through behaviors that can be assessed qualitatively rather than through measurement in traditional quantitative tests. iv. limitations. this study involved two classes of physical therapist students with a total of 19 students (10 in year a and 9 in year b). thus, given the sample sizes, sophisticated quantitative analyses, and hypothesis testing would have been questionable. while the comments of individual students in the focus group activities might have been isolated, coded, and quantified, using for example, coded responses and student demographics as variables, it would have been problematic to apply even non-parametric tests such as chi square due to small cell sizes. indeed, part of the focus group process involves striving for consensus to extract major issues and thus, individual student comments do not form part of typical reports of results and are not collected as such. another issue to consider is that the incorporation of ethical considerations into professional behaviors and practice is, as noted above, related to the affective domain and thus, the most accurate way to observe and record the incorporation of beliefs into a value system is to observe those beliefs as evidenced by behaviors over time. longitudinal studies were not possible at this stage and even if planned, they would require both time and (again with such small samples) the collection of enough data to allow appropriate analysis. such investigation also requires financial support and at the moment, the studies are in the realm of action research or classroom research, for which funding is limited at best. the way to overcome both of these limitations would be to secure funding for a study on a larger scale involving several physical therapist programs and a collaborative research agenda that would allow comparison and analyses of students’ behaviors and attitudes. during this study, this kind of exploration was not possible. v. outcomes. venglar, m., and theall, m. the journal of scholarship of teaching and learning, vol. 7, no. 1, may 2007. 68 in both groups, the students were reportedly candid and open in their responses. responses, as reported by the group facilitator, to each questions are presented below. year a is the class taught via traditional lecture format. year b is the class taught via the case-based format. a. question 1. “what were the three most important (valuable) things you learned in the course?” year a: students were pragmatic in their views, noting the value of 1) knowing relevant laws; 2) interacting with in-service clinicians; 3) understanding hipaa and medicare rules; and 4) reviewing job descriptions. year b: 1) students reported that they had gained a broader perspective on ethical issues due to having to consider various stakeholder views. the complexity of the issues was also clearer as a result of students having to respond to the cases employed in the course. 2) a related opinion was that the case-based approach led to more application of the knowledge gained in the course both during the course and later, in clinical experiences. 3) students also said that they had gained more/better knowledge of legal issues [through discussion of ethical issues] and that their clinical experiences supplemented this knowledge. b. question 2. “were you able to integrate the course content into your practice/profession when you were taking the course? if yes, then how? if no, then why not?” year a: the consensus was that students were able to integrate content during the course because they had to: 1) recognize situations with legal/ethical implications; 2) follow regulations (e.g. health insurance portability and accountability act); and 3) learn to observe practice in clinical situations. year b: the consensus was that students benefited from the case-based approach because it required this kind of integration. while they felt (as had the previous group) that they entered into the course with mature understanding of ethical issues and the ability to make appropriate ethical decisions, the practical benefit of the case method was that it demonstrated that they could improve their problem-solving skills. c. question 3. “when you had your clinical experience after the course, were you able to better integrate the course content into your practice/profession? if yes, then how? if no, then why not?” year a: students felt that their clinical experiences improved their ability to integrate course content. they cited straightforward responsibilities such as billing, the delegation of appropriate responsibilities to others, and the opportunity to observe a variety of situations, particularly those that repeated previous situations and thus re-emphasized their prior learning and experience. when asked about the extent to which they experienced any direct instruction or assistance (i.e. did they receive any training on the job) they said that any dialogue on venglar, m., and theall, m. the journal of scholarship of teaching and learning, vol. 7, no. 1, may 2007. 69 legal/ethical issues had been initiated by their observations or questions and that they had not received any direct training. in effect, one had to be observant and willing to ask if/when legal/ethical issues arose. no one reported resistance to questions, but there was an indication that students felt that some long-term clinicians might not be as well informed about new regulations as they were. in effect with these individuals, there was no point in asking for advice from someone who didn’t have current knowledge. year b: students felt that their clinical experiences improved their ability to integrate course content. the combination of active practice through the cases and the clinical experience allowed further development of their awareness and the practical application of ideas in the sense of blending legal and ethical knowledge when making professional decisions. d. question 4. “do you believe that the course is important/valuable? if yes, then why? if no, then why not?” year a: students felt that the content was valuable, but the course was less so. they indicated that more emphasis should be placed on legal aspects because they needed specific knowledge of laws and regulations. they indicated that they felt capable of making ethical decisions without as much course work, saying, ‘…we are mature enough to understand ethical dilemmas and consequences.’” year b: students felt that the course was both important and valuable. all felt that they needed to be aware of the issues so that they could deal with situations that arise in professional practice. the extent of this feeling varied as a function of the intended career paths of the individuals. those who were considering private practice or management roles indicated that they would deal with complex legal-ethical issues more often than would those whose career interests were more focused on direct provision of patient/client care. the main legal-ethical issue that students noted was the balance between the business/economic requirements faced by care providers and the need to provide clients with the most appropriate amount of care for the most appropriate amount of time. in applied terms, this issue involved deciding how much care to provide given two possibly conflicting agendas: the need for the clinic to make a profit (i.e. using available insurance coverage to provide services that bring a return to the clinic) and the need to preserve the client’s fiscal protection (i.e. not using up all of the insurance coverage so that the client could get further services if needed). e. question 5. “what would you recommend that would make the course more valuable?” year a: responses to the previous question relate to students feelings that this course requires more time than is necessary. they suggested that this course could be combined with management or positive health courses. one reason offered for this opinion was that when students reported on their visits to clinics and interviews with clinicians, they found very similar situations and got very similar responses. thus, when in-class reports were venglar, m., and theall, m. the journal of scholarship of teaching and learning, vol. 7, no. 1, may 2007. 70 given, the reports were repetitive. after the first few reports, nothing new was learned from hearing those remaining. year b: students offered the idea that more regular integration of legal and ethical topics would help them to develop more sophisticated decision-making skills. they said that having one topic on one day and the other topic on another day tended to separate rather than integrate the two content areas. since they felt that their professional decisions would most often require a blending of legal and ethical considerations, they also felt they would benefit from more frequent combinations of the topics. students also said the use of the case-based methods in treating legal content would help them to develop better problem solving skills in this area. finally, students noted a desire for more discussion. this can be related to the use of cases that require more activity and engagement than do passive methods such as lecture. in the context of discussing cases, students have to verbalize their thinking and respond to each other. discussions would be expected to supplement the thinking required by the cases and to make the various decision options more obvious (i.e. there are many possibilities) as well as more clear (i.e. discussion may reveal the most appropriate courses of action). vi. discussion. the focus group comments indicate that the students taught using the traditional lecture format for ethics content apparently valued little of the content following the course. responses to all of the questions are largely focused on the legal content taught in the course. apparently the content covered during the ethics portion of the course was either already known (as stated in the answer to question 4), or considered less relevant/valuable while the students were in the clinic. in addition, the group comment in question five, asking how to make the course more valuable, indicates that the content is not of value and should be condensed in lieu of other more pressing topics. although knowledge of the legal aspects of the physical therapy profession is vital, ethical practice is also vital. this point, however, was apparently not conveyed well in the traditional lecture format. the responses from the students in year b indicate that using cases and encouraging students to voice and defend their opinions, caused the students to better recognize and integrate what they had learned in class to their clinical experiences. an unanticipated outcome was the report that the case-based format of the ethics content allowed for better integration of the legal and ethical aspects of the class. the class is not formatted such that the two topics are intentionally integrated; however, in all of the case discussions students raised questions of law as well as the pertinent ethical aspects. these students did not feel the course content should be condensed, as had previous students, but rather recommended intentional integration of the legal content and the ethics content with further use of cases. the case-based method, requiring critical thinking and problem-solving for case discussion, appears to have sensitized the students to the ethical subtleties in clinical practice during their subsequent clinical experience. given the importance that the apta has attached to incorporation of ethical standards into professional practice, future investigation is necessary into effective methods of teaching about ethical issues, helping students to understand and apply ethical principles, and motivating students to attend to these issues in their classes, clinical experiences, and professional lives. the venglar, m., and theall, m. the journal of scholarship of teaching and learning, vol. 7, no. 1, may 2007. 71 limitations in this study suggest that an appropriate and perhaps necessary strategy for definitive research would be to secure funding for large-scale exploration of the issues across multiple institutions and over time. with sufficient data, more rigorous analysis could be applied to determine direct cognitive gains with the target of raising the taxonomic levels of learning to application at minimum, and to higher levels in practice. additionally, the duration of the effects could be assessed through longitudinal study of professional practices of individuals accompanied by a practical assessment of the impact of workplace environments, management practices, and operational behaviors in clinical, out-patient, and other settings. vii. conclusions. in the physical therapy profession we teach students to treat individual patients and to treat patients as individuals. each patient is a “case” with its own unique characteristics, but with traits that put that case into a general category for diagnosis and treatment. the expectation that students will naturally translate generalized knowledge into appropriate cases without practice is perhaps unrealistic. students have limited exposure to clinical situations prior to becoming physical therapists, thus they require sensitizing not only to those situations that require clinical skill, but also to those situations which may impact them ethically. the above outcomes demonstrate that the use of cases, relating ethical issues to cases in other courses, and allowing students to explore the articles/cases using a guided decision-making process improves students’ awareness of ethical issues, enhances critical thinking in non-clinical aspects of a client case, and increases the value of ethics education for the students. the students had an opportunity to participate in the critical thinking needed to make the content personally relevant and incorporate it into their approach to patient care (atton c, 2005; limbach bj et al., 2005). the effort of the american physical therapy association to improve professionalism in physical therapy involves ethical decision-making. this approach to ethics education resulted in subjective student reports of increased perceptions of the value of the content, increased awareness of ethical situations, and increased understanding of the prevalence and significance of ethical decision-making in the clinical setting. although the method presented is only one option for teaching ethics content, the outcomes indicate that teaching ethics in a format similar to that used in teaching clinical skills is effective for increasing understanding of the issues, enhancing the value of the content, raising awareness of ethical issues in clinical settings, and developing higher-level professional problem-solving skills. acknowledgements dr. venglar wishes to express sincere appreciation to the center for teaching and learning at youngstown state university (catalyst) for assistance in developing this teaching strategy. this strategy has made teaching of the content much more interesting, as it has apparently improved the interest of learning this content among students. in addition, dr. venglar wishes to thank dr. swisher, author of the rips model, for her guidance in the use of the model with student instruction. her encouragement and interest in this effort were inspiring. references venglar, m., and theall, m. the journal of scholarship of teaching and learning, vol. 7, no. 1, may 2007. 72 physical therapist clinical performance instrument. (1997). alexandria, va. american physical therapy association. professionalism in physical therapy: core values. (2003). alexandria, va. american physical therapy association. atton, c. (2005). using critical thinking as a basis for library user education. journal of academic librarianship, 20, 310-315. cross, k. p. and steadman, m. h. (1996) classroom research: implementing the scholarship of teaching. san francisco: jossey bass. gabard, d. l., martin, m. w. (2003). physical therapy ethics. philadelphia: f. a. davis company. glaser, j. w. (1994). three realms of ethics. (1 ed.). kansas city: sheed and ward. glaser, j. w. (2005). educating for moral action: a sourcebook in health and rehabilitation ethics. philadelphia: fa davis. hudson, j. n., and buckley, p. (2004). an evaluation of case-based teaching: evidence for continuing benefit and realization of aims. advances in physiology education, 28, 15-22. hutchings, p. (ed.) (2000). opening lines: approaches to the scholarship of teaching and learning. menlo park, ca: carnegie foundation for the advancement of teaching. jonassen, d. h., and hernandez-serrano, j. (2002). case-based reasoning and instructional design: using stories to support problem solving. education technology research and development, 50 (2), 65-77. keefer, m., and ashley, k. d. (2001). case-based approaches to professional ethics: a systematic comparison of students' and ethicists' moral reasoning. journal of moral education, 30 (4), 377398. kidder, r. m. (1995). how good people make tough choices: resolving the dilemmas of ethical living. new york: fireside. krathwohl, d. r., bloom, b. s., and masia, b. b. (1956) taxonomy of educational objectives. handbook ii: the affective domain. new york: david mckay. limbach, b. j., and waugh, w. l. (2005). questioning the lecture format. thought and action, 21, 47-56. maiorana, v. (2005). the road from rote to critical thinking. community review 11, 53-64. mcginty, s. m. (2000). case-method teaching: an overview of the pedagogy and rationale for its use in physical therapy education. journal of physical therapy education, 14 (1), 48-51. venglar, m., and theall, m. the journal of scholarship of teaching and learning, vol. 7, no. 1, may 2007. 73 patton, m. q. (1990). qualitative evaluation and research methods. newbury park, ca: sage publications. purtilo, r. b. (2005). ethical dimensions in the health professions (4th ed.). philadelphia: elsevier. rest, j. r., and narvaez, d. (1994). moral development in the professions: psychology and applied ethics. hillsdale, nj: lawrence erlbaum associates. rest, j. r., narvaez, d., bebeau, m. j., and thoma, s. j. (1999). postconventional moral thinking: a neo-kohlbergian approach. mahwah, nj: lawrence erlbaum associates. swisher, l. l., arslanian, l.a., and davis, c. m. (2005). the realm-individual processsituation (rips) model of ethical decision-making. hpa resource, 5 (3), 1, 3-8. triezenberg, h. l., and mcgrath, j.h. (2001). the use of narrative in an applied ethics course for physical therapist students. journal of physical therapy education, 5 (3), 49-56. venglar, m., and theall, m. the journal of scholarship of teaching and learning, vol. 7, no. 1, may 2007. 74 appendix 1. scenario of rips decision-making model. this is a brief scenario to serve as an example of the rips decision-making model. students are asked to identify the realm, individual process and situation, and then to provide appropriate rationale for their choices of each. there is discussion prior to initiating this activity that all opinions are respected when accompanied by adequate rationale, and any legal influence should be considered when it impacts the ethical decision-making. following this activity the class discusses appropriate courses of action based on the results of the model. scenario: john is a therapist with a contract to provide services to the patients of a managed care company. the company is very clear in its contract that john is to follow the critical pathways. he can treat patients for less time than anticipated, but he cannot extend additional care without approval. he is not free to refer patients to outside therapists, nor is he in any way to “undermine” the credibility of the care offered by the company. in the course of treating a 42 year old man for injuries resulting from multipletrauma, john realizes that his patient should have the care of a certified hand therapist. he also realizes that he will not be able to help the patient reach his potential within the number of visits approved by the managed care company. the company has never honored any of his previous requests for extension of visits. if john does not petition for an extension and the patient is harmed, then john may be liable for the harm. but another petition from him might reduce the likelihood that his contract will be renewed and even put him at risk for dismissal. if he informs the patient that he needs a therapist with expertise in hand therapy, and if the patient then demands from the managed care company the expertise that john recommends, john will most certainly be dismissed. however, if john uses his social skills and convinces the patient that he is getting the very best care, it is unlikely there will be any negative repercussions for john. (adapted from “physical therapy ethics” by donald gabard and mike martin; copyright 2003, fa davis) there are two possible arguments in this scenario: rips argument a argument b realm – environmental context individual – john, as the individual employee, is the primary stakeholder of this situation. as the person working directly with the patient, he is the one to make the decisions that will impact the patient’s care and progress. therefore, he “owns” the decision-making organizational – the managed care company, the organization, is the primary stakeholder. their policies have created a situation in which patients can only receive expert care if the expert is already part of the company. managed care companies are often chastised for putting finances ahead of patient venglar, m., and theall, m. the journal of scholarship of teaching and learning, vol. 7, no. 1, may 2007. 75 ability/responsibility care. the scenario appears to fit that notion. if the managed care company allowed patients, even on a case-by-case basis to be referred outside of the company for care not available within the company, this ethical situation may not have existed. individual process aspect of morality moral sensitivity – moral sensitivity is the ability to recognize and interpret a situation as one with ethical concern. john recognized that the need of the patient conflicted with the policies of the managed care company. he also recognized that the steps to follow to meet the patient’s needs have been unsuccessful in the past with previous patients. moral courage – moral courage means having courage to practice ethically as well as persisting in one’s efforts to implement ethical care. john appears to be struggling with moral courage. this may be due to past efforts being unsuccessful; this may be due to concern for his job/livelihood. he appears to know what the ethical course of action is, but is lacking full courage to implement the action. moral motivation – moral motivation is the act of prioritizing ethical values over other values. this scenario demonstrates a lack of moral motivation on the part of the managed care company. although john is in the position to advocate for the patient, the policies of the company will, and have previously, limit the success of his advocacy. the company appears to prioritize finances over ethical, and quality, treatment of patients. situation – nature of the ethical concern dilemma – a dilemma is two apparently “right” courses of action; taking one “right” course compromises the other “right” course. both can not exist together. john has a dilemma in that if he chooses to request an extension of time or refer the patient to an expert he will likely lose his job. if john chooses to convince the patient he is getting better, the patient temptation – temptation is the choice of right vs. wrong in which one stands to profit from choosing wrong. although the company can not state they are choosing to do “wrong” by their patients, the actions of the company in preventing employees from referring a patient for the most appropriate care appear wrong. in preventing the action from venglar, m., and theall, m. the journal of scholarship of teaching and learning, vol. 7, no. 1, may 2007. 76 does not receive the most appropriate care, but john maintains his job. if john serves the patient he puts himself at risk; if he serves himself he neglects the full needs of the patient. distress – distress is when the stakeholder knows the right thing to do but is not empowered to act. john is in distress because his clinical expertise tells him his patient needs the skills of an expert hand therapist. he is restricted by the managed care company in his effort to make the appropriate referral for his patient. he is not empowered to provide the best care for the patient. occurring, the company benefits financially. 503 service temporarily unavailable 503 service temporarily unavailable nginx/1.14.1 5071-apa complete journal of the scholarship of teaching and learning, vol. 14, no. 3, august 2014, pp. 75 92. doi: 10.14434/josotl.v14i3.5071 exploration of undergraduate preservice teachers' experiences learning advocacy: a mixed-methods study kelley massengale1, cherese childers-mckee2 and aerin benavides3 abstract:   applying transformational critical advocacy research in college instruction can be a powerful way to engage students in challenging inequity in society and promoting positive changes. few studies systematically measure the impact of such pedagogy on the development of college students’ beliefs about advocacy. in this mixed methods study, we worked with 21 preservice teachers through advocacy letter writing activities and collected data from pre/post surveys and focus group discussions to explore the impact of such pedagogy. the findings indicated that advocacy letter writing was a meaningful activity for preservice teachers, allowing them a professional opportunity to voice their concerns about personally meaningful issues to entities in power. a significant correlation was found between baseline advocacy experiences and baseline advocacy beliefs, suggesting that the teaching of advocacy, when combined with opportunities for meaningful practice, can contribute to shifts in belief about the importance of advocating. keywords: transformative education, teacher education, advocacy, mixed methods research introduction educators recognize the importance of equipping students with a variety of skills that will benefit them in their personal and professional lives. the ability to advocate for self and others represents a useful strategy for fostering personal empowerment and effecting social change. scholars in a variety of fields have integrated advocacy-type activities in coursework. some of the more prominent have been in the health profession (hearne, 2008; radius, galer-unti, & tappe, 2009) and in teacher education (athanases & de oliveira, 2008, grymes, 2007). in teaching students about advocacy and providing opportunities for them to engage in advocacy within class activities, an underlying assumption is that as students learn about advocacy, their belief in the benefit and worth of advocating will naturally follow. yet, there have been few studies that systematically measure shifts in beliefs as undergraduate students actively engage in advocacy activities. our study design was influenced by the overarching questions of how individuals might take up a critical advocacy lens and how undergraduate students might be empowered to advocate in their personal and professional lives. in this study, we described how advocacy activities were integrated in a college course and examined the impact of such activities on the development of undergraduate preservice teachers’ advocacy beliefs.                                                                                                                             1 doctoral student, university of north carolina at greensboro, department of public health education, kemassen@uncg.edu 2 doctoral student, university of north carolina at greensboro, department of educational leadership and cultural foundations, cdchilde@uncg.edu   3 doctoral student, university of north carolina at greensboro, department of teacher education and higher education, awbenavi@ uncg.edu   massengale, k., childers-mckee, c., & benavides, a.   journal of the scholarship of teaching and learning, vol. 14, no. 3, august 2014. josotl.indiana.edu 76 review of literature theoretical framework the philosophy that undergirds our inquiry into the topic of undergraduate students' experiences learning advocacy is informed by critical advocacy research (shields, 2012). through this particular lens, we foreground the role of critical epistemologies in addressing “questions of inequity and disparity” for the purpose of advocating for “policies and practices that can lead to economic, ecological, and human justice, and a sustainable global future” (shields, 2012, p. 3). we define advocacy in line with shields as “taking a stance on behalf of a person or a position in which one believes” (p 6). critical advocacy research reflects a commitment to social justice, empowerment, and change, while upholding standards for conducting research that are rigorous, contextual, and meaningful. this study is situated within a transformative mixed methods paradigm in which the researchers acknowledge the workings of power in all parts of the research process (mertens, 2003, 2007). this particular design is appropriate for a study of teaching advocacy to undergraduate students in that we seek to “conduct research that is change-oriented and seeks to advance social justice causes by identifying power imbalances and empowering individuals and/or communities” (creswell & plano clark, 2011, p. 96). therefore, research decisions, from research questions to data analysis, reflect a philosophical commitment to empower participants to advocate for themselves and others. this study reflects participatory elements of a transformative design (mertens, 2003, 2007, 2010, 2012). also, it represents an important step toward empowering undergraduate students to cultivate the critical consciousness necessary to engage in transformative-emancipatory work in their personal and professional lives. an exploration of the processes by which individuals take up a critical advocacy lens and come to think of themselves as advocates holds implications for educators of undergraduate students in a variety of fields including education, public health, social work, and many more. advocacy in teacher education both critical advocacy teaching and the transformative mixed methods design mirror conceptualizations of advocacy in teacher education. in a review of standards used by several teacher education programs, grymes (2007) found that typical standards reflected a broad range of activities that could be considered advocacy such as, policy-related advocacy, advocacy based on developing relationships and trust with families, and advocating for individual children with special needs. she concluded that although commonly used standards already reflect the need for advocacy, teacher education programs must continue to equip teachers with necessary skills to be advocates for themselves, their students, and their school communities. sleeter (2008) draws similar parallels by including teacher activism and advocacy as one of two broad strands critical in educating teachers for equity and diversity in order to explore teachers’ perceptions of advocacy, althanases and de oliveira (2008) conducted focus groups of a representative group of new teachers to find out how and why teachers engaged in classroom advocacy. advocacy, as defined by participants and researchers, included meeting diverse needs of students, hunting for resources, tutoring, increased parent communication, fieldtrips, and creating a culture club. teachers identified connections between their acts of advocacy and the training acquired in their teacher education programs. athanases massengale, k., childers-mckee, c., & benavides, a.   journal of the scholarship of teaching and learning, vol. 14, no. 3, august 2014. josotl.indiana.edu 77 and de oliveira (2008) conclude that an advocate views “all aspects of school as problematic rather than given,” “learn[s] to locate expertise inside oneself,” and can envision how schools can more effectively meet all students’ needs (p. 68). similarly, silverman (2010) draws connections between preservice teachers’ beliefs about diversity and feelings of efficacy in advocating for change. in her review of the literature, she contends teacher education programs should increase their focus on helping teachers’ develop feelings of responsibility and commitment to advocate for diversity. data found that many survey respondents, though they identified themselves as advocates, felt low personal responsibility to advocate. additional research is needed to explore the nuances of how the responsibility and desire to advocate are cultivated within undergraduate courses. teaching advocacy the teaching of advocacy in undergraduate classrooms has been used for multiple purposes. as a pedagogical strategy, advocacy-related instruction is thought to increase student engagement and motivation in the classroom (beacham & shambaugh, 2007). beacham and shambaugh (2007) describe two case studies in which advocacy activities represented a type of pbl (problem-based learning) in both an education graduate level course and an interior design undergraduate level course. they describe that advocacy-related activities fostered increased student engagement through enhancing “relevancy, challenge, and uniqueness” (p. 316) of instruction. beacham and shambaugh conclude that advocacy as both a teaching strategy and a learning objective enhanced students’ feelings of agency and empowerment. as service learning becomes increasingly common in undergraduate courses (brundiers, wiek, redman, 2010; butin, 2006; cawthorn, leege, & congdon, 2011), the teaching of advocacy becomes a critical component in preparing students for community engagement. preservice teachers have difficulty making connections between personal experiences of social service and larger societal issues, and this can be made explicit as a course component based on advocacy (wade, 2003). bridging theories on advocacy and service learning, berke, boydsoisson, voorhees, & reininga (2010) describe the experiences of undergraduate family science students who were matched with community agencies and worked on collaborative advocacy projects throughout the semester. students experienced positive benefits from engaging in advocacy activities including knowledge of working with agencies and an increased understanding of advocacy as a macro-level process. the existing studies on the teaching of advocacy in undergraduate classrooms emphasize the benefits of advocacy. further study is needed to better describe variations in effectiveness for different forms of advocacy, variations across content areas, and the role of advocacy experiences in shifting beliefs about the benefits of advocating. additionally, studies on the teaching of advocacy in undergraduate classrooms would benefit from a more robust theory of the stages of advocacy development for undergraduate students. this study provides both an analysis of the effectiveness of an advocacy lesson and activity in shifting beliefs about advocacy and offers a model of the stages of development of beliefs in advocacy. advocacy letter writing advocacy letter writing has been employed to express concerns to people in power and suggest changes on behalf of individuals, minority groups, majority groups, and the environment. massengale, k., childers-mckee, c., & benavides, a.   journal of the scholarship of teaching and learning, vol. 14, no. 3, august 2014. josotl.indiana.edu 78 syndicated cartoonists who were the recipients of advocacy letters changed their behaviors and began drawing seat belts on their characters when depicting them in vehicles (mathews & dix, 1992). writing advocacy letters has been suggested as a method for contacting politicians on behalf of legislative health issues (huntington, 2001), as an effective tool for individuals living with disabilities (white, thomson, & nary, 1997), and as a transformative activity for preservice english for academic purposes and english as a second language teachers (morgan, 2009). within the undergraduate classroom, advocacy letter writing is a tool for engaging students in social discourse. advocacy letter writing on the topic of oil and politics has been used as an assignment to integrate political science and english composition skills for first year undergraduate students (huerta & sperry, 2013). undergraduate nursing students participated in advocacy letter writing as a way to learn about population groups affected by community health topics (eide, hahn, bayne, allen, & swain, 2006). what is unknown about previous undergraduate advocacy assignments is how students’ beliefs about advocacy may have been impacted by participating in the assignments. this paper builds upon the existing literature about advocacy letters by describing the experiences and change in advocacy beliefs of a group of undergraduate preservice teachers who were assigned to write advocacy letters to people in power about issues of personal meaning to them. course context selected courses preservice teachers who are undergraduate students dually enrolled in a course on both elementary education in seminar/internship and advanced science methodology participated in the study. these particular courses were selected because one is an experiential learning course following a theme of cultural awareness. the other course includes a lesson on how to differentiate between environmental education and environmental advocacy. students enrolled in these classes may benefit from learning about advocacy as they are preparing to enter the profession of elementary education in which they may need to advocate for individual students, groups of students, or themselves as teachers. advocacy lesson all students in the courses participated in a lesson on advocacy. the primary instructor of the lesson, a doctoral student with experience teaching undergraduate public health courses, had previously delivered similar lessons to students enrolled in public health courses. three other instructors from the teacher education and educational leadership & cultural foundations departments joined the lesson discussions to share their own perspectives and experiences. the interactive lecture began by asking students to share, with a classmate, a memory from a time when they or someone that they cared about did not have something that was needed or was treated unfairly. students were asked to reflect on how they felt at the time, whether or not any action was taken, whether it was difficult to take action, feelings about the outcome of the situation, and whether they would act differently in a similar situation in the future. next, the primary instructor shared an example from her own experience then invited students to share their experiences with the entire class if they wanted. students shared a wide range of experiences and some students’ stories revealed details about their lives and personal values that massengale, k., childers-mckee, c., & benavides, a.   journal of the scholarship of teaching and learning, vol. 14, no. 3, august 2014. josotl.indiana.edu 79 were previously unknown to their peers and instructors. the class engaged in a discussion on how these experiences were acts of advocacy or revealed a time when advocacy could have potentially changed the outcome. the lesson then proceeded by including the preservice teachers’ input in defining advocacy, determining what an advocate does, and in brainstorming reasons to advocate. preservice teachers learned about the phases of advocacy (defining and researching an issue, planning advocacy activities, thinking of all possible solutions and outcomes, advocating, evaluating the act of advocacy, then planning for any future actions) and role-played effective and ineffective advocacy. the instructor led a discussion on what might happen if the act of advocacy had unintended consequences or made the situation worse. preservice teachers discussed how they might act as advocates in their future roles as in-service teachers and brainstormed when presented with the scenario of how they could advocate for elementary school student fieldtrips given district budget restraints. next, in small groups, preservice teachers evaluated a case study about community development near a marsh and demonstrated critical thinking in determining how different acts of advocacy might lead to different outcomes and what any unintended consequences of advocacy might be. preservice teachers listed pros and cons to appealing a decision made by local legislatures, involving an outside advocacy organization, and taking no action. preservice teachers also listed alternate actions and acts of advocacy they identified on their own. through discussion of the pros and cons they brainstormed, material taught earlier in the lesson about characteristics of effective and ineffective advocates was emphasized. throughout the entire lesson, preservice teachers were encouraged to ask questions and to share any of their own experiences with advocacy. the lesson concluded with a group discussion of a news story about a polish art student who advocated for his own art by hanging it in a museum (scislowska, 2012). the news story allowed the students to review what they had learned during the lesson and to connect it to the real life experience of a student, someone they may have been able to relate to. advocacy letter writing activity at the end of the advocacy lesson, preservice teachers were given an assignment to write advocacy letters of their own. example letters and suggested formats were shared. the phases of advocacy discussed during the lecture were reviewed in dissecting example letters. participants were instructed to each choose an issue of personal meaning to research and compose an advocacy letter requesting a change in the status quo. no restrictions were placed on the advocacy topics; participants were encouraged to reflect on what was important to them in their personal lives noting that the issue could be something that affects a small group of people, a large population, a specific community such as a student’s hometown, or one that was relevant for policy makers. preservice teachers were asked to reflect on what could be done to improve the issues of choice and to determine who had the power to make the changes happen. to increase the likelihood of receiving responses to their letters, participants were encouraged to send their letters to local entities with the power to enact changes on their chosen topics. they were given one week to research their issues, write, and send their letters. proof of letter submission was required to receive credit for having completed the course assignment. massengale, k., childers-mckee, c., & benavides, a.   journal of the scholarship of teaching and learning, vol. 14, no. 3, august 2014. josotl.indiana.edu 80 methods participants the preservice teachers participating in the study were enrolled as undergraduate students at a medium-sized university in a southeastern state. all of the students were seniors, (n = 22, average age = 22 years old) pursuing elementary education as a major and earning a state environmental educator certification. the participants were all female and self-identified as: 90% christian, 81% white, 43% raised in a rural environment, and 57% raised in a suburban environment. average self-reported annual income of the participants’ families of origin was measured using income categories from the us census with 52% reporting income in the upper two-fifths, 29% in the mid fifth, and 19% in the lower four fifths. study design in acknowledging the multifaceted nature of the concept of advocacy, a mixed methods transformative approach could best capture the complexity of the issue (mertens, 2012) and reflect the purpose of empowering and effecting change in participants’ understanding of advocacy. the research design included qualitative research and quantitative research in parallel strands (see figure 1). per creswell and plano clark (2011), the start of many mixed methods studies falls between fixed and emergent designs. our study also fell on this continuum with both fixed and emergent characteristics. as an emergent mixed methods design (creswell & plano clark, 2011), the guiding original transformative study design was altered to include a quantitative/qualitative post survey; this was added during the research process. we as researchers felt the data obtained in the pre-intervention survey about advocacy beliefs would be important to collect again after the focus group discussions, in order to specifically address our research question about significant difference in advocacy beliefs. the one eight-item question on the post-intervention survey was taken from the pre-intervention survey instrument. we decided to add another qualitative question at the end of our post-intervention survey as themes of growth in advocacy emerged from the focus groups. we wanted to give participants a chance to provide any final words on their experiences. the following research questions were constructed to represent a blend of both quantitative and qualitative strands and guided the investigation of students’ experiences learning advocacy: 1) is there a statistically significant change in preservice teachers’ beliefs about engaging in advocacy through a lesson on advocacy and a letter writing activity as evidenced by responses on a preand post-intervention survey? 2) in what ways do the focus groups’ results explain any differences between preand post-intervention survey responses? 3) what meanings do preservice teachers make of advocacy and the advocacy letter writing activity? massengale, k., childers-mckee, c., & benavides, a.   journal of the scholarship of teaching and learning, vol. 14, no. 3, august 2014. josotl.indiana.edu 81 figure 1. research design data collection research participants took a written pre-intervention survey to express their prior participation in advocacy and express their beliefs about advocacy. next, participants engaged in a lesson on advocacy conducted by the research team, then were asked to write a letter to a local entity advocating for change on a topic of personal meaning to the participant. three weeks later, participants were asked to join a focus group to have an opportunity to further discuss the ways in which they understood their experiences participating in the lesson and writing the advocacy letter. finally, all participants took a written post-intervention survey. preand post-intervention survey part one of the participant survey contained questions modeled after those on the social issues advocacy scale [sias] (nilsson, marszalek, linnemeyer, bahner, & misialek, 2011). the sias, created to assess behaviors and attitudes about advocating on social justice issues, measured activities and beliefs relevant for most any field or discipline (nilsson et al., 2011). the instrument used a likert-type scale to measure participation in various advocacy-related activities or agreement with specific beliefs and demonstrated good internal validity (nilsson et al., 2011). modifications were made to collect similar data without a likert-type scale. following these quantitative questions on baseline advocacy activities and beliefs, two openmassengale, k., childers-mckee, c., & benavides, a.   journal of the scholarship of teaching and learning, vol. 14, no. 3, august 2014. josotl.indiana.edu 82 ended qualitative questions allowed the participants to describe prior undergraduate learning experiences and course assignments in which they may have learned about advocacy. part two of the participant survey contained questions modified from those on the social justice advocacy scale which was designed to measure counselors’ social justice advocacy abilities (dean, 2009). modifications were made to measure beliefs about aspects of social justice advocacy relevant to preservice teachers. the responses to the 25 likert-type scale questions in this portion of the survey were tested for internal reliability, and for these items it was high, with a chronbach’s alpha of .895. the exact cutoff for what is considered high reliability is debated, but it is generally accepted that high reliability in chronbach’s alpha is anything above .8 (field, 2013). focus groups all participants in the study submitted their letters to an instructor by hard copy or email. the research team met the day before focus group discussions to read and sort the letters into three groups of mixed topics. the letters were written to a range of addressees, from local officials to large corporations, and were evenly distributed among the three focus groups, purposefully giving each focus group a variety of topics. the items included in the focus group protocol were guided by the qualitative research question, what meanings do preservice teachers make of advocacy and the advocacy letter writing activity? consistent with the convergent parallel research design, the focus group protocol was written at the onset of the study. during the three focus groups, students first introduced themselves and the topic of their letters. the rest of the focus group was guided by a semi-structured interview protocol. in analyzing focus group transcripts, initial coding reflected hatch’s (2002) conceptualization of an inductive analysis approach that identifies patterns of meaning in data in order to make statements about the phenomenon under investigation. interrater reliability was assured by all three researchers listening to audio recordings of all three focus group sessions before coding data, then reaching consensus on coding within each transcript. after initial open coding, each transcript was read multiple times and recoded by each member of the research team to identify additional codes in efforts to triangulate interpretations. then, all codes were placed into categories which were then thematized, paying close attention to outliers that ran counter to initial interpretations (hatch, 2002; strauss & corbin, 1998) results changes in advocacy beliefs through analysis of quantitative data collected on the preand post-intervention surveys during the study, we sought to address our first research question to determine if there was a statistically significant change in preservice teachers’ beliefs about engaging in advocacy through a lesson on advocacy and a letter writing activity. prior to participation in the advocacy lesson and letter writing activity, participants reported 96 total acts of advocacy from among 14 different advocacy activities. participants, on average, had previously engaged in 4 to 5 different types of advocacy activities with discussing bills and legislative issues of personal importance with family and friends (81%), voting in local elections (62%), and using social media to influence massengale, k., childers-mckee, c., & benavides, a.   journal of the scholarship of teaching and learning, vol. 14, no. 3, august 2014. josotl.indiana.edu 83 others regarding issues they cared about (52%) being the most popular activities. advocacy experience was found to have a positive correlation (p = .011) with average scores on the likerttype scale questions about advocacy beliefs. prior to the advocacy lesson, 76% of students had previously learned about advocacy in another undergraduate course. several students named a required educational leadership and cultural foundations course which explored education as a social institution and advocating for racially, culturally, and linguistically diverse groups. one preservice teacher described a course activity that required students to attend an event or setting outside their comfort zones as one which taught her about advocacy. others mentioned learning about environmental advocacy or response to intervention (rti) to accommodate individual learning needs. the preservice teachers, on average, agreed with 5 out of the 8 advocacy belief items in part one of the survey prior to participation in the letter writing activity. almost one third (29%) agreed with all eight items. the first four items were beliefs related to the effects of societal forces on health, wellbeing, and educational performance as well as the effects of state and federal policy on access to education and social services; the majority of participants (70%) agreed with each statement. the highest agreement was with the belief in societal forces affecting access to education and resources (95% agreed). the last four items were beliefs in the professional responsibility to confront those who discriminate against the elderly, disabled, those of a different culture or ethnicity, or of a different sexual orientation. we also found a significant correlation between individual advocacy experience and beliefs scores (number of items respondents agreed with on part one of the pre-intervention survey) and the average individual score on the likert-type scale questions in part two. the overall mean frequency score for beliefs and experiences in advocacy for part one was 4.57. the overall mean likert score for part two on the pre-intervention survey was 2.6, on a scale of 0 11. the range of mean likert scores for the preservice teachers was from 0.36 5.68. testing for a correlation between beliefs and experiences scores in part one, and beliefs scores in part two, a positive correlation is statistically significant (p = .011) at the .05 alpha level (2-tailed). the strong baseline positive correlation of experience in advocacy with belief in advocacy in this sample is a compelling argument for recommending that participants experience advocacy as part of transformative coursework. on average, none of the eight items on both surveys showed a decrease in advocacy beliefs pre-post, and the last four items showed an increase in advocacy beliefs. in an analysis of mean differences (using spss paired samples t-test) between individual preand postintervention survey responses (n = 19), there was a statistically significant mean difference increase in the average number of individuals’ advocacy beliefs (𝑥 pre = 0.64, 𝑥 post = 0.76, α = .05, df = 19, tcrit = -2.62, p = .018). participants’ explanations of belief changes qualitative data collected during the focus groups enabled us to address our second research question in which we sought to determine in what ways the focus groups’ results explained any differences between preand post-intervention survey responses. in all three focus groups, facilitators began with the same focus group protocol—although we also permitted participants to introduce and address additional topics that came up in the course of conversation. in this way focus groups were semi-structured in order to encourage a more participatory style that in some instances allowed participants to control the flow of the conversation. due to time constraints massengale, k., childers-mckee, c., & benavides, a.   journal of the scholarship of teaching and learning, vol. 14, no. 3, august 2014. josotl.indiana.edu 84 focus groups lasted approximately an hour and participants were encouraged to discuss their feelings and experiences with the letter writing activity and any plans for future advocacy. topics ranged from training the police force in the state to recognize and prevent human trafficking to improving maintenance at a breezeway at the participant’s apartment complex. one preservice teacher said she used a “sandwich” of a compliment, the meat of the letter, and then ended with a compliment, to request a large cereal manufacturer omit a potentially cancer causing chemical from their cereal, and soon after her comment, another student in her group used the same terminology, “sandwich,” to describe the letter she wrote. clearly, as focus group discussion progressed, there was listening, reflection, and learning taking place. one student said she knew she should have researched before writing to parking services, but her letter was from emotion. the act of discussing the process of writing reminded participants of the recommended steps from the advocacy lesson. there was explicit expression of learning through discussion, as a student exclaimed, “oh, i hadn’t thought of that before. . .” over 30 individual codes were generated from the focus group transcripts, then grouped thematically into the following 6 categories: 1) awareness of the benefits of advocacy, 2) negotiating who benefits from advocacy, 3) affective dimensions of advocacy, 4) barriers to advocacy, 5) negotiating the process of becoming an advocate, and 6) embracing advocacy. while the preand post-intervention surveys were crucial to answering the question of whether or not beliefs about advocacy changed, the focus groups contributed to an understanding of the nuances and idiosyncrasies of changes that occurred in participants’ awareness, beliefs, or action around questions of advocacy (see figure 2). awareness of the benefits of advocacy. participants described the extent to which the advocacy activities shifted their beliefs and perspectives about advocacy or increased their awareness of a particular issue. shifts in awareness occurred in several important areas. participants expressed that they became more aware of how to advocate and why advocacy is important. one described, “i think this assignment opened up my eyes to the importance of advocacy. even if your advocacy does not result in change, you put your voice out there and stood up for what you believe in.” another remarked, “it brought to my attention how showing you care can have an overall effect.” therefore, an awareness of ways to effectively advocate as well as the belief that advocacy is important work in tandem in the process of becoming an effective advocate. negotiating the process of becoming an advocate. this category represents a broad theme that encompasses data related to specific advocacy skills learned during the advocacy lesson and how students responded to the process of learning to be advocates. learning about the process of advocacy reflected one of the most important components of their burgeoning advocacy development. one participant stated, “first time advocating for anything. interesting to learn the process.” another stated, “it’s a very authentic learning experience.” participants also added that the experience gave them added knowledge about different ways to advocate and broadened their perspective of what advocacy is. one stated, “when i thought of an advocate, a lot of times i thought of an extremist.” therefore, many participants viewed letter writing as an effective way to advocate that was potentially lower risk than other forms of advocacy like marching or protesting. while most participants expressed that they enjoyed the lesson and felt it was beneficial information to know, it was not assumed that this implied they would necessarily engage in future advocacy. therefore, additional themes were identified that further contribute to the advocacy development process. massengale, k., childers-mckee, c., & benavides, a.   journal of the scholarship of teaching and learning, vol. 14, no. 3, august 2014. josotl.indiana.edu 85 figure 2. advocacy development model negotiating who benefits from advocacy. in keeping with the theoretical framework that privileges participants’ involvement in the research process, participants possessed complete control over the topic of their letters. the choice of whether to advocate for self or others represented a salient theme throughout all three focus groups. one participant stated, “the activity was very beneficial and helped me learn the importance of advocating for myself and what’s important to me.” many other participants echoed similar sentiments, while others chose to advocate for issues that would impact others more than themselves. while it could be interpreted that negotiating for self shows a more novice understanding of advocacy, upon deeper analysis of focus group responses, we suggest that negotiating the beneficiary of advocacy activity represents a key component in the development of an advocacy identity. in learning to advocate for themselves, some participants developed a greater confidence in advocating for others in future endeavors. affective dimensions of advocacy. the feelings and emotions associated with engaging in advocacy represented recurrent themes in all focus groups. participants expressed the personal advocate negotiating the process of an advocate awareness of the benefits of advocacy   negotiating who benefits from advocacy   affective dimensions of advocacy barriers to becoming an advocate fluid overlapping contextual “it’s a very authentic learning experience, especially if they get something back.” embracing advocacy “i think writing a letter is a good first step in pursuing something you want to be advocating for” “i pay $304 a parking pass. i should be able to park in a paved parking lot close to campus.” “i should have sent it to the dean, or somebody higher up.” “i’m just one person.” “i’m kind of a passive person in general to be able to write to them and support my argument was kind of empowering” massengale, k., childers-mckee, c., & benavides, a.   journal of the scholarship of teaching and learning, vol. 14, no. 3, august 2014. josotl.indiana.edu 86 satisfaction gained through advocating, feelings of frustration or disappointment at not receiving a response to a letter, and the sentiment that being able to advocate for something personally meaningful gave them a voice. affective dimensions of advocacy were also critically important in fostering feelings of empowerment among participants. one student shared, “i’m kind of a passive person in general…so to be able to write to them and support my argument was kind of empowering.” one stated, “it’s been a continual frustration. this letter was a great way to get if off my chest and i felt better when i was done writing it, cuz i was able to vent.” another remarked, “when i sent this letter and received feedback it made me feel like i had taken a step towards change and that felt good.” as illustrated in this response, responses to letters seemed to play an integral part in participants’ feelings about the experience. those who received responses expressed excitement and feelings of encouragement, while some of those who did not expressed frustration and feelings that letter writing was ineffective. others who did not receive responses simply expressed a renewed commitment to continue trying. overall focus group data about the affective dimension of advocacy suggest that students who expressed strong emotions about a particular topic were more empowered to express their concerns and take action for change. barriers to advocacy. although barriers to advocacy were mentioned at least once in each focus group, participants’ hesitancy in taking up identities as advocates was more pronounced in one focus group discussion in particular. although negative sentiments about advocacy were primarily espoused by two individuals in the group, others silently concurred with certain statements. one participant stated, “you put all this time and hope into researching for a letter and the other end doesn’t even care to say anything back.” another responded, “the majority of the time they’re gonna do whatever they wanna do and it has to do with money.” several participants worried that their one letter would not be significant enough to effect change. we suggest that in developing an understanding of the process by which participants develop an advocacy identity, negotiating barriers to advocacy represents a real and valid concern. meaningfulness of advocacy letters to address our final research question, “what meanings do preservice teachers make of advocacy and the advocacy letter writing activity,” we considered our analysis of the advocacy letters themselves, the quantitative and qualitative analysis previously described, and a qualitative question added to the post-intervention survey. as part of the study intervention, students were asked to compose and send to an entity in power a letter on a personally meaningful topic about which they wanted to advocate. the participants (n = 22) who wrote advocacy letters all provided evidence that their letters had been written and sent. some provided the letter to researchers to mail, while others produced a screen shot or confirmation page of the letter being sent electronically. letters were analyzed by topic, beneficiary, and whether or not a response was received. in analyzing the beneficiaries of the letters, or the individual(s) for whom the participant intended to advocate, three categories emerged: self advocacy, advocacy for self and others, and advocacy for others. letters written to advocate for self included letters written to the university parking services (n = 4), letters written to campus dining services (n = 2), participant’s employer (n = 1), and the health insurance provider of a participant (n = 1). letters written for the purpose of self-advocacy represented 37% of letters written. interestingly, an equal percentage of letters were written to advocate for others (37%). in contrast to self-advocacy letters in which 6 of the 8 letters written related to issues specific to participants’ university massengale, k., childers-mckee, c., & benavides, a.   journal of the scholarship of teaching and learning, vol. 14, no. 3, august 2014. josotl.indiana.edu 87 experience, the letters written to advocate for others were more varied. topics included letters to the state governor (n = 2) about increasing funds to state school systems, a letter to a representative about increasing awareness of human trafficking in the state, a letter to a local school superintendent to reduce class size by hiring an additional teacher, a letter to emergency services to provide an ambulance for rural parts of a local county, a letter to a national corporation about the chemical bht in cereal marketed to children, a letter to a restaurant about working collaboratively with local high schools to employ teens, and a letter to campus disability services to improve ramps and make campus more wheelchair accessible. the remaining (27%) were written for self and others and represented similar topics to those just described. due to the potential influence of rate of response on feelings about advocacy, the letter analysis documented whether or not each participant received a response. of the 22 letters written, 37% (n = 8) of participants showed proof that they received a response to their letter. all but one of the letters that received a response were written to local entities. the rate of response varied, with some participants receiving immediate responses while others received responses after a couple of weeks. all of the responses were polite and encouraging, some articulated future plans to address the requested change, and others suggested alternatives that the letter writer could explore, but none of the letters promised immediate changes. although the advocacy letters were a significant piece of the data analysis, there were many questions left unanswered. how did participants choose a topic? how did the classroom lesson and discussion contribute to their letter writing experiences? did participants perceive letter writing as an act of advocacy? how did receiving a response potentially influence participants’ feelings about the experience? therefore, the focus groups represented an important vehicle for both understanding participants’ feelings about the process of letter writing to advocate for a cause, as well as continuing the discussion about advocacy and empowerment. two-thirds of all participants responded to the optional open-ended question at the end of the post-intervention survey where participants could have the final say on the project. all responded positively and similarly to tell us just how meaningful the experience was: “i feel that this assignment was beneficial in educating us about advocacy, and it gave us the opportunity to be an advocate for something we believe in. because of this project i feel that i am more clear on what advocacy is and different ways you can choose to advocate.” another student wrote: “this has helped inform me of advocacy and i now want to advocate more about things i am concerned and passionate about!” participants’ understandings of and feelings about advocacy fell along a fluid continuum. although participants reported prior advocacy experiences at baseline, after taking part in the letter writing experience all embraced advocacy to some extent, even those with the lowest baseline scores. this was revealed in a systematic analysis of a cross section of all data collected for each individual. through both focus groups and post-intervention survey open ended responses, participants identified themselves as advocates, making future plans to advocate, recognizing the complex dimensions of advocacy, expressing they had learned how to advocate, and showing an understanding of the role of power in advocacy. massengale, k., childers-mckee, c., & benavides, a.   journal of the scholarship of teaching and learning, vol. 14, no. 3, august 2014. josotl.indiana.edu 88 discussion interpreting preservice teachers’ prior advocacy experiences preservice teachers’ shared experiences prior to participating in the intervention could account for some but not all of the responses on the pre-intervention survey. each of the participants had experience as an intern in local public schools for over three semesters prior to completing the survey at baseline. the overwhelming agreement with the statement that society affects access to education and resources was not surprising as teaching was their specialty, and they had firsthand knowledge of inequities in local area public schools. due to their common major and required coursework, participants typically take the same core group of classes. despite having been enrolled in the same classes throughout the past year, participants had vague and inconsistent understandings of advocacy prior to participating in advocacy activities in this study. activities in prior courses that some participants recognized as advocacy activities were not recalled by others. this finding supports the justification for including specific instruction on advocacy in undergraduate courses coupled with the opportunity to advocate for something. results from all phases of this study provide encouragement for continuing to incorporate advocacy lessons and assignments in undergraduate courses. first, results suggest that the process of advocating for something personally meaningful empowered participants and transformed their ideas about advocacy. therefore, the combination of the advocacy lesson, actual practice advocating for something personally important, and the experience of discussing the process with others contributed to participants’ greater understanding of the process and importance of advocacy. second, beliefs shifted through learning about advocacy and engaging in advocacy experiences. the cross analysis of quantitative and qualitative data shows the transformation that occurred in students’ beliefs and understandings of advocacy. these findings corroborate and expand upon other research of students’ learning of advocacy (beacham & shambaugh, 2007; berke et al., 2010) by explicitly drawing connections between experiences engaging in advocacy and beliefs about advocacy. further, the findings illustrate the usefulness of advocacy teaching as a pedagogical intervention to connect theory to practice or illustrate the real-world usefulness of advocacy in professional settings. as some students indicated that writing and sending the advocacy letters was the first time they had ever advocated for something, the opportunity to advocate for something within the context of an undergraduate course allowed them the guidance and experience of their instructors as they interpreted their own actions and reflected on the experience of advocating for the first time. having had the opportunity to advocate in the context of their own learning experiences, the preservice teachers may feel more confident and prepared to advocate in a professional setting once they are no longer in the student role. finally, learning to advocate is a fluid, contextual process in which discussion, reflection, and the co-construction of ideas is important. integrating data from all results, we have theorized a model for understanding advocacy development or the way in which individuals come to view themselves as advocates. within this model (figure 2), individuals develop an awareness of the benefits of advocacy while negotiating various components within the process of becoming an advocate including who benefits (self, other, or both), affective feelings about advocacy, and barriers and obstacles to seeing oneself as an advocate, before finally embracing an identity as an advocate. although the model is circular, it should not be interpreted as an illustration of massengale, k., childers-mckee, c., & benavides, a.   journal of the scholarship of teaching and learning, vol. 14, no. 3, august 2014. josotl.indiana.edu 89 sequential steps that individuals go through on the path to becoming an advocate; instead, embracing a critical advocacy lens involves a fluid, contextual, and interconnected process of negotiation influenced by many components. limitations of the study while the participants found the advocacy lesson and letter writing activity to have personal meaning for them, the sample size was small and relatively homogenous. more research is needed to understand if the same lesson and activity would be an effective way to teach a more diverse group of undergraduate students about advocacy. the timeframe of the study required all activities to be complete within one semester. this short timeline did not allow for providing participants with much guidance on continuing to advocate for their specific topics after their initial letters were sent and any initial replies were received. also, detailed discussions of power, privilege, social justice, and inequality require more time than we were afforded in the research site. while these components of advocacy represented an integral part of our research lens, due to time restrictions, they were not explicitly presented to participants during the advocacy lesson. therefore, future research in this area could explore the ways in which discussions of social justice influence the choices of letter topics and the overall development of advocacy in undergraduate students. between the preand post-intervention surveys, the students read and discussed a book on workplace diversity for a separate assignment. their experiences reading the book may have also contributed to the increased number of students who indicated on the postintervention survey that they would feel compelled to act if a colleague discriminated against others based on differences in culture or ethnicity. recommendations for teaching advocacy incorporating lessons and class discussions about advocacy in undergraduate courses is encouraged as a way for students to gain awareness on when and how they may advocate for themselves and others in situations when they do and do not have power over others. lessons on advocacy may be most effective when they involve both small and large group discussions so that students who are not comfortable sharing personal advocacy experiences with the larger group may still have the opportunity to share with others. examples of effective and ineffective advocacy from students’ intended careers offers suggestions to students on ways they themselves might act as advocates. a case study example may provide students with the opportunity to engage in critical thinking about the potential outcomes and consequences of different acts of advocacy that could all occur in response to the same situation reinforcing advocacy concepts previously learned during a lesson. letter writing assignments have the potential to teach students about the importance of research and professionalism when contacting entities in power to make changes and allow students the chance to practice advocacy concepts examined during a class lesson. allowing students to choose the issues for which they advocate naturally encourages students to become personally invested in the outcomes. visiting the topic of advocacy over multiple course meetings allows students to see how advocacy is applied in multiple contexts and provides the students with guidance as they interpret their own advocacy experiences and develop their own identities as advocates. while this study enrolled students in the field of education, undergraduate students in any discipline may benefit from learning about advocacy as it relates both to their fields of study and to their personal lives. massengale, k., childers-mckee, c., & benavides, a.   journal of the scholarship of teaching and learning, vol. 14, no. 3, august 2014. josotl.indiana.edu 90 instructors who incorporate advocacy lessons and activities in their undergraduate courses may benefit from preparing for a range of emotions and reactions students may experience when assigned to contact entities in power to encourage a change in the status qua. the portion of students in our study who contacted campus service providers for parking, dining, and accessibility services demonstrates that students are continually reflecting on and interpreting their college experiences of which instructors play integral roles. during focus groups, some students expressed disillusionment with local community leaders in power while others expressed discouraging thoughts about the potential change that could come from their seemingly meager acts of advocacy. historical and current examples of successful advocates can remind students of their own potential as advocates. still, students demonstrated creativity and thoughtfulness in their words and advocacy actions, leaving us encouraged as they will soon grow in their advocacy identities as teachers. conclusion in their future roles as in-service teachers, preservice teachers will encounter many instances when advocating for a change in the status quo may benefit an individual student, an entire class, a school, or school system, or themselves as teachers. some of the instances may require immediate action, others may require cycles of research, planning, action, and reflection to achieve the greatest possible impact. empowering preservice teachers with effective advocacy skills before they become in-service teachers may give them the confidence to advocate in both their professional careers and in their personal lives. specific advocacy instruction along with an advocacy letter writing assignment is an effective combination for introducing undergraduate students to the topic of advocacy, providing guidance as students advocate, possibly for the first time, and grow into their identities as advocates. acknowledgments we acknowledge our preservice teachers for their participation and for providing us with the privilege of understanding their advocacy experiences, some of which were deeply personal. we appreciate the support for our study of dr. melony allen and dr. catherine matthews. we are grateful for the support and guidance of dr. ye he. references athanases, s., & de oliveira, l. (2008). advocacy for equity in classrooms and beyond: new teachers' challenges and responses. the teachers college record, 110(1), 64-104. beacham, c. v., & shambaugh, n. (2007). advocacy as a problem-based learning (pbl) teaching strategy. international journal of teaching and learning in higher education, 19(3), 315-324. berke, d.l., boyd-soisson, e. f., voorhees, a. n., & reininga, e. w. (2010). advocacy as service-learning. family science review, 15(1), 13-30. massengale, k., childers-mckee, c., & benavides, a.   journal of the scholarship of teaching and learning, vol. 14, no. 3, august 2014. josotl.indiana.edu 91 brundiers, k., wiek, a., & redman, c. l. (2010). real-world learning opportunities in sustainability: from classroom into the real world. international journal of sustainability in higher education, 11(4), 308-324. doi: 10.1108/14676371011077540 butin, d. w. (2006). the limits of service-learning in higher education. the review of higher education, 29(4), 473-498. doi: 10.1353/rhe.2006.0025 cawthorn, m., leege, l., & congdon, e. (2011). improving learning outcomes in large environmental science classrooms through short-term service-learning projects. journal of environmental studies and sciences, 1(1), 75-87. doi: 10.1007/s13412-011-0001-8 creswell, j. w. & plano clark, v. l. (2011). designing and conducting mixed methods research (2nd ed.). thousand oaks, ca: sage. dean, j. k. (2009). quantifying social justice advocacy competency: development of the social justice advocacy scale. (unpublished phd). georgia state university, atlanta. retrieved from http://scholarworks.gsu.edu/cps_diss/40/ eide, p. j., hahn, l., bayne, t., allen, c. b., & swain, d. (2006). the population-focused analysis project for teaching community health. nursing education perspectives, 27(1), 22-27. field, a. (2013) discovering statistics using ibm spss (4th ed.). london: sage grymes, j. m. (2007). advocacy and teacher preparation programs. childhood education, 84(2), 94-d. hatch, j. (2002). doing qualitative research in education settings. albany, ny: state university of new york press. hearne, s. a. (2008). practice-based teaching for health policy action and advocacy. public health reports, 123(suppl 2), 65. huerta, j. c., & sperry, r. (2013). pulling it together: using integrative assignments as empirical direct measures of student learning for learning community program assessment. learning communities research and practice, 1(1), 16. huntington, c. g. (2001). legislative advocacy for health professions educators. education for health change in learning & practice, 14(2), 241-250. doi: 10.1080/13576280110059228 mathews, r. m., & dix, m. (1992). behavior change in the funny papers: feedback to cartoonists on safety belt use. journal of applied behavior analysis, 25(4), 769-775. doi: 10.1901/jaba.1992.25-769 mertens, d. (2003). mixed methods and the politics of human research: the transformativeemancipatory perspective. in a. tashakkori & c. teddlie (eds.), handbook of mixed methods in social and behavioral research (pp. 135-164). thousand oaks, ca: sage. massengale, k., childers-mckee, c., & benavides, a.   journal of the scholarship of teaching and learning, vol. 14, no. 3, august 2014. josotl.indiana.edu 92 mertens, d. (2007). transformative paradigm: mixed methods and social justice. journal of mixed methods research, 1(3), 212-225. doi: 10.1177/1558689807302811 mertens, d. (2010). transformative mixed methods research. qualitative inquiry, 16(6), 469474. doi: 10.1177/1077800410364612 mertens, d. m. (2012). what comes first? the paradigm or the approach? journal of mixed methods research, 6(4), 255-257. doi: 10.1177/1558689812461574 morgan, b. (2009). fostering transformative practitioners for critical eap: possibilities and challenges. journal of english for academic purposes, 8(2), 86-99. doi: 10.1016/j.jeap.2008.09.001 nilsson, j. e., marszalek, j. m., linnemeyer, r. m., bahner, a. d., & misialek, l. h. (2011). development and assessment of the social issues advocacy scale. educational and psychological measurement, 71(1), 258-275. doi: 10.1177/0013164410391581 radius, s. m., galer-unti, r. a., & tappe, m. k. (2009). educating for advocacy: recommendations for professional preparation and development based on a needs and capacity assessment of health education faculty. health promotion practice, 10(1), 83-91. doi: 10.1177/1524839907306407 scislowska, m. (2012, january 4). polish art student hangs own painting in museum. chicago sun-times. retrieved from http://www.suntimes.com/news/world/9809511-418/polish-artstudent-hangs-own-painting-in-museum.html shields, c. (2012). critical qualitative research. in s. steinberg & g. cannella (eds.), critical qualitative research reader (pp. 104-114). new york, ny: peter lang. silverman, s. k. (2010). what is diversity? an inquiry into preservice teacher beliefs. american educational research journal, 47(2), 292-329. doi: 10.3102/0002831210365096 sleeter, c. (2008). equity, democracy, and neoliberal assaults on teacher education. teaching and teacher education, 24(8), 1947-1957. doi: 10.1016/j.tate.2008.04.003 strauss, a. l., & corbin, j. m. (1998). basics of qualitative research: techniques and procedures for developing grounded theory. thousand oaks: sage publications. wade, r. (2003). teaching preservice social studies teachers to be advocates for social change. social studies 94(3), 129-133. doi: 10.1080/00377990309600195 white, g. w., thomson, r. j., & nary, d. e. (1997). an empirical analysis of the effects of a self-administered advocacy letter training program. rehabilitation counseling bulletin, 41(2), 74-87.   503 service temporarily unavailable 503 service temporarily unavailable nginx/1.14.1 3123-11909-1-ce journal of the scholarship of teaching and learning, vol. 13, no. 3, august 2013, pp. 48 – 62. a novel approach for practitioners in training: a blended-learning seminar combining experts, students and practitioners vicky j. meretsky1 and teresa a. n. woods2 abstract: a joint student and professional practitioner seminar used distance technology to allow remote experts to present to, and remote practitioners to participate in, a university-based learning experience. participants were professional practitioners from the us fish and wildlife service who were mandated to receive training and on-campus graduate students in environmentally focused programs who were enrolled for credit. seminars providing training in high-demand or cutting-edge topics may be especially valuable to practitioners outside the university in business, agency, or organization positions, if they can attend as distance learners. such classes create opportunities to bring students and professionals together to interact with expert presenters, who may present from distant locations. presenters model expert thinking for students and engage them in discussions in which they practice such thinking. students gain additional insight into their field of practice by observing interactions between practitioners and presenters, as well as by working directly with practitioners, in discussions and, potentially, in assignments. as a result, at little cost to any participant, students are engaged in authentic learning that is not regularly available in a classroom setting and practitioners gain access to a series of experts as well as access to student views and, potentially, student work. instructors must relinquish considerable control of some aspects of the learning environment, but as mediators can increase the value-added aspects of sharing the class with professionals. professional programs seeking to prepare students for professional practice often combine both more traditional classroom learning and experiential learning during thesis preparation, service learning or internships. seminars such at this provide a valuable addition to this mix. keywords: professional training, reflective practice, distance teaching and learning, cognitive apprenticeship, experiential learning, decoding i. introduction. the training of practitioners, traditionally undertaken through apprenticeships, and the training of scholars, traditional undertaken through university degrees, come together in the training of professional practitioners. practitioners-in-training are scattered throughout the academy in schools of business, public administration, engineering, nursing, social work, education, law, design, as well as in professional programs in the applied sciences. practice—the experience of doing of things—has long been recognized as distinct from and important to content-based education (dewey, 1938). 1 associate professor, indiana university bloomington, school of public and environmental affairs, meretsky@indiana.edu 2 doctoral student, university of minnesota, natural resources science and management, wood0752@umn.edu meretsky, v.j., and woods, t.a.n. journal of the scholarship of teaching and learning, vol. 13, no. 3, august 2013. josotl.indiana.edu 49 professional practice requires integrating factual information with a nuanced appreciation of the context in which that information is to be brought to bear. the ability to apply the appropriate information in the appropriate way has been considered a form of art (e.g., schön, 1987). perhaps for this reason, professional practice is often considered part of the “hidden curriculum,” a phrase first used to connote a wide range of behaviors and mores that are not taught explicitly (e.g., snyder, 1970) and that may be less easily seen and adopted by some student than by others, depending on their backgrounds. although much of the hidden curriculum debate focuses on concerns for students disadvantaged and discouraged by unspoken rules, among educators in medical and pharmacological fields in particular, the phrase describes aspects of professional practice and behavior that may not be made explicit in academic training, in parallel with schon’s (1987) discussion (bradley, steven, & ashcroft, 2011; jaye, egan, & parker, 2005; masella, 2006). another conception of this need to learn by practicing is the notion of cognitive apprenticeship (collins, brown, & newman, 1989). collins et al. (1989) defined apprenticeship as “embed[ding] the learning of skills and knowledge in their social and functional context” (p. 454). ideally, the training of practitioners results in students who have a good start in this art and sufficient understanding of the undertaking to direct their own learning and advancement, as schön’s phrase, reflective practitioner, suggests. the teaching needs that schön set out in educating the reflective practitioner are echoed in pace and middendorf’s decoding the disciplines (2004). although pace and middendorf discuss training students to become experts in academic disciplines, an analogous process that might be called decoding the practices is clearly called for in training practitioners and indeed is an apt paraphrase of the approach advocated in schön (1987). traditional university courses teach requisite academic skills and subjects, but in isolation these lack the context and synergy needed in order for student to become practitioners. internships, project-based classes, and service-learning classes provide opportunities to work with one or more communities of practitioners and these may or may not provide explanations of practice or opportunities for reflection concerning practice. as a faculty member (author1) and an agency practitioner (author2), we created a graduate seminar designed to provide training, in a cutting-edge topic, simultaneously to graduate students in a traditional classroom and practitioners from a federal agency who attended remotely. in bringing these two groups together in a seminar we created what lave and wenger (1991) have termed a community of practice: a group that comes together to focus on gathering and sharing knowledge on particular aspects of practice in a professional field. situated learning (lave & wenger, 1991; resnick, 1989), cognitive apprenticeship (collins et al., 1989) and experiential learning (kolb, 1984) are all aspects of cognitive learning that stress the importance of experience and authenticity for effective learning. in this case study, we discuss the value-added aspects, beyond standard course content, that this combination afforded graduate students who were in training to become professional practitioners. we describe the aspects of experience and authenticity our course format provided and explore ways to increase opportunities for reflection in future iterations. we discuss the course practices and outcomes in the context of distance learning and blended learning and describe the role of instructor-facilitators in enhancing the learning opportunities of the course and in maintaining a sense of presence for all participants. meretsky, v.j., and woods, t.a.n. journal of the scholarship of teaching and learning, vol. 13, no. 3, august 2013. josotl.indiana.edu 50 ii. background and methods. in 2007, the us fish and wildlife service (fws) directed its various regions to provide training to many of their regional wildlife managers and biologists on the topic of climate change. the mandate specifically directed that training was to include more just a handful of staff, so that an in-depth understanding of climate change would rapidly become widespread in the agency. region 3 (the upper midwest) sought to provide such training in a way that would avoid contributing to climate-change by transporting staff, possibly repeatedly, to some common learning site. to that end, author2, from region 3, contacted author1 at the school of public and environmental affairs at indiana university to investigate the possibility of a combined seminar that would train both graduate students from the university and agency personnel from the region. in addition, we sought to eliminate most expense and travel (and its accompanying impacts on climate change) by having experts present from their home locations. similarly, although one practitioner who lived nearby usually attended the class in person, the remaining practitioners participated individually and in groups from across the region via electronic and telephone connections. figure 1 diagrams the resulting synchronous learning environment. figure 1. components of the blended-learning seminar. the physical space was an oncampus classroom. the virtual space was an adobe connect session with a toll-free conference telephone line to permit synchronous question-and-answer and discussion. an expert lecturer led lecture-discussions remotely; the facilitators, together, guided discussions. to accommodate the fws practicing professionals, the class met once each week, for 2.75 hours. typically, an expert speaker discussed some aspect of climate change for the first 1.25 hrs, including discussion with students and practitioners which we both facilitated. after a 15-min break, the last 1.25 hr was used for a discussion, again facilitated by both of us, of primary literature we had chosen to complement but not duplicate the expert’s material. distant expert speakers used adobe connect and a toll-free telephone line provided by the fws to share powerpoint presentations with the classroom and with remote practicing professionals who participated from a distance. facilitators used the same system to present the reading under discussion in the second part of the class. synchronous learning environment physical space virtual space academic facilitator graduate students expert lecturer practitioner facilitator practicing professionals (fws employees) meretsky, v.j., and woods, t.a.n. journal of the scholarship of teaching and learning, vol. 13, no. 3, august 2013. josotl.indiana.edu 51 adobeconnect is a desktop-sharing program that allows users to exchange control of the screen. distant experts could control their powerpoint and other materials themselves. there is also a chat window that allows participants to type questions or comments during presentations without stopping the flow of the presentation. we encouraged presenters to use the setting that put their presentation on the full screen while instructor-facilitators used the setting that showed presentation and chat. in this way, presenters were not distracted and facilitators could interject with questions from chat or from the classroom at appropriate pauses. reading discussions, being less formal, used chat less, with distant participants commenting over the telephone. each speaker was given a short training session in the week before their presentation to assure that software issues were minimized and to discuss the logistics of speaking “blind” – without visual contact with the audience. due to their other duties, the number of practitioners varied from week to week. the phone link was piped into the classroom through an audio link with the toll-free line, and remote attendees and expert presenters used speaker-phones. author1 attended all classes in the classroom, serving as a facilitator and coordinator during expert presentations and as facilitator during discussions of primary literature. author2 attended all classes on the phone link and facilitated and amplified practitioner comments and discussion. remote participants contributed to discussion either by telephone or by typing questions into the chat window. due to the large and varying number of practitioners attending, we did not use video of participants at all during the class. the format required front-end time to learn the online technology and set up the slate of expert speakers, but most other aspects of the course occurred at a moderate pace throughout the semester. overall, the time needed to prepare and present the course was not more than and was possibly less than would have been required for a traditional lecture/seminar combination. our primary aim in developing the seminar was to provide students and practitioners alike with information about impacts of climate change on fish and wildlife conservation, through presentations by experts from academia, agencies, and ngos. we hoped that having both students and practitioners in the audience would enrich discussion and that each audience group would benefit from the knowledge of the other. seminars, owing to the importance of discussion, already involve collaborative learning. because this format was, so far as we know, entirely novel, we did not know how learning in this format might differ from more traditional academic seminars. in particular, we were aware that differences among the audience subgroups, both in their prior knowledge and experience and in their methods of experiencing the course, might affect the modes of learning and the nature of the knowledge learned. we, the authors, spoke weekly during the course, discussing and reflecting on the nature of the interactions and learning observed as the course progressed. we realized that the learning environment we had created was providing a useful window for our students into the world and practice of our practicing professionals, as well as into climate-change science. we began to ask professionals who were commenting during question-and-answer and discussion periods to explain the contexts in which they applied climate-change science and the aspects of their work that raised questions about climate change, in order to enrich this practitioner-oriented aspect of the class without taking substantial time away from the primary topic. in addition, we sought to capture information about the nature of the practitioner-oriented learning that our students were experiencing. this scholarship-of-teaching-and-learning aspect of the seminar was not planned at the outset; rather, we acted on an unexpected opportunity in order to learn how our format added value to student and practitioner experiences. meretsky, v.j., and woods, t.a.n. journal of the scholarship of teaching and learning, vol. 13, no. 3, august 2013. josotl.indiana.edu 52 with human-subjects approval, we surveyed students, expert speakers, and participants at the end of the course, asking for demographic information, previous experience with web-based or distance education through a variety of likert-scale and open-ended questions addressing the experience of the course. some questions were designed to assess logistical aspects of the course (e.g., for speakers, ease or difficulty of presenting without receiving visual cues from the audience); others asked participants to comment on the potential benefits or costs to learning associated with the course format. relevant portions of the student and expert surveys are included in appendixes. we will provide details on the perspectives of the practicing professionals in a future publication. iii. results. we focus here primarily on results associated with introducing graduate students to their community of practitioners. a. participation. the class (n = 28) included one undergraduate student in environmental management and 27 students pursuing master’s degrees in environmental science (mses), mpa degrees with a focus on environmental policy and natural resource management or sustainability, or dual mses/mpa degrees. one local agency practitioner often joined the class in the classroom. another 10-80 practitioners (approximately, depending on the day) joined the class remotely, using the agency telephone line and adobe connect software. a handful of practitioners were regular participants but most only attended sessions of particular interest; all practitioners had occasional absences due to conflicts with work schedules. b. recruiting expert speakers. expert speakers universally responded positively to invitations to speak to the class and typically cited both the minimal time commitment and the opportunity to address practitioners in the primary federal agency tasked with wildlife conservation as attractive aspects of the invitation. all of our first-choice outside speakers who did not have time conflicts accepted invitations to speak (n = 12). most made their presentations from their offices, but one presented from home, and another presented while on sabbatical leave in europe. of nine speakers responding to the speaker survey, all indicated they were somewhat or extremely likely to consider a distancelearning/ distance-teaching format for future classes. eight were somewhat or extremely likely to consider an agency-university mix. c. student responses regarding interactions with agency practitioners. of 27 students responding to “do you believe that a collaborative agency/university approach to learning helped you learn about breaking issues more than a traditional classroom approach y/n” all answered yes and 25 provided additional comments. one student wrote “provides insight into ‘real’ world issues agencies are facing and challenges they are trying to overcome.” another offered “we get to hear what agency officials deal with every day. if we want to work for them, this is helpful.” the following themes appeared repeatedly in comments: meretsky, v.j., and woods, t.a.n. journal of the scholarship of teaching and learning, vol. 13, no. 3, august 2013. josotl.indiana.edu 53 ● better understanding of how climate change information is being used in the real world/outside perspective (n = 13), ● opportunities to interact with professionals (n = 8), ● better understanding of agency issues (n = 5), and ● usefulness for future interviews/employment (n = 2). d. agency responses regarding student interaction. of 54 agency personnel responding to “is attending such a seminar with university students a plus, a minus, or a neutral aspect of the experience?” 19 ranked it a plus, 17 were neutral, and three ranked it as a minus. positive respondents overwhelmingly mentioned students’ fresh or different perspectives (11of 14 who provided comments). others mentioned the opportunity to bring science into the agency, and recruiting opportunities with students. e. logistics. invited to speak specifically about any aspects of expert presentations that were lost in distance teaching formats, 11 of 27 students providing survey responses had no comments on this question, eight noted a loss of visual cues such as body language, three indicated they found the experience less personal, and one mentioned awkward interruptions (possibly due to the rare technology glitch). five of nine experts had not previously used web-based training or seminar software and four of ten had never participated in a web-based seminar or training session. for eight remote speakers responding to the question, scores for difficulty of use of the software on a scale of 1 (easy) -7 (hard) ranged from 2 to 5.5, with an average of 3.9—almost exactly the middle of the scale. iv. discussion. a. a community of learning and a community of practitioners. opportunities for cross-training in this format ran in all directions among graduate students, practicing professionals and experts, as is appropriate for communities of learning and practice. most of the graduate students were in programs in applied ecology and in environmental policy, rather than in wildlife ecology, and they brought that wider training to their discussions in class. the practicing professionals, primarily wildlife biologists and planners, often took time to clarify the role and responsibilities of the fws, the relationship between the public and the fws, and the constraints under which the service operates, in order to provide experts and instructorfacilitators with context for their questions. if context was important but unclear or absent in a question or comment, facilitators often asked professionals to provide additional information. experts ranged from purely academic to very applied and thus varied in their familiarity with environmental policies, with the fws, and with attitudes of the public towards wildlife-related issues. thus, while experts provided knowledge in their area of specialty, they were often recipients of knowledge concerning application of the information they provided. students enrolled in the seminar class clearly found that the student/practitioner mix provided opportunities for learning that were substantively different from those in traditional meretsky, v.j., and woods, t.a.n. journal of the scholarship of teaching and learning, vol. 13, no. 3, august 2013. josotl.indiana.edu 54 university seminars. their comments focused specifically on aspects of the class that provided a view into the community of practitioners that they were being trained to join and on the opportunity to begin to make contacts in that community. agency participants were encouraged to consider whether they had projects or information needs that students could address in their term papers; two students wrote such papers. one student was hired by fws. his first post was at a national wildlife refuge that had had personnel among the practitioner participants for the course. the most common means of introducing students to communities of practice involve active experiential learning in situations such as internships, service-learning, and project-based classes. the mses and mpa programs at indiana university require two of these (an internship and a project-based capstone class) of all students. both of these approaches put students directly in contact with practitioners but, in both cases, contact may involve only a single office or a single practitioner. the format discussed here did not have as its primary purpose creating practitioners from graduate students. nevertheless, it gave students an opportunity to hear and interact with practitioners who held many different positions, from several states and from two different branches of a major federal agency. in addition, the involvement of practitioners likely improved our ability to attract very high-quality speakers, thus fulfilling one of our primary goals. by creating, simultaneously, a community of learning and a community of practitioners that were jointly interactive, we provided students with an unusually rich opportunity at no extra cost to them and essentially no extra cost to the university (the use of a distance-learning classroom to support the telephone connections to speakers and practitioners). survey results clearly showed that students saw and appreciated the agency perspective. this behind-the-scenes look into agency operations provided aspects of decoding practice by allowing students to observe reflective practice as practitioners discussed how respond to climate change in their various positions. thus, by bringing practitioners to the students, remotely, our format brought situated learning into the classroom (lave & wenger, 1991). more importantly, opportunities to watch experts interact with practitioners provided students with deeper understanding of the relationship between research and practice and the processes by which research informs practice. pace and middendorf (2004), in decoding the disciplines: helping students learn disciplinary ways of thinking, speak to the problem of creating experts from naive students and develop a model of identifying learning bottlenecks and making visible to students the thinking that experts do in any given discipline. in this, they address the problem posed by collins et al.’s (1989) discussion of cognitive apprenticeship and lave and wenger’s (1991) discussion of situated learning. decoding a discipline or practice is accomplished by breaking the thinking or practice down to its component pieces and modeling each step for the students before asking them to practice that type of thinking and, later, to practice doing, on their own. the teaching approach we describe here is not a full decoding solution to bottlenecks in training professionals in that it lacks the overt step of explicitly describing practice and allowing students opportunities for guided practice. rather, our format adds extensive interactions with practitioners, which give students the opportunity to observe practitioner approaches and narrative decoding of practitioner thoughts, to a teaching environment that is already rich with learning possibilities. meretsky, v.j., and woods, t.a.n. journal of the scholarship of teaching and learning, vol. 13, no. 3, august 2013. josotl.indiana.edu 55 b. adding reflection. the seminar-discussion format gave students opportunities for reflection, but not at the level suggested for reflective practice. benefits of reflection could be increased substantially by constructing a course/community of practice with an explicit goal of reflection. both experts and practitioners could be asked to explicitly decode aspects of practice in the discussion, when possible. in addition, students could undertake assignments that promote reflection (such as focused reflective journals) and that apply what they hear about practice from experts and practitioners in course-related projects and assignments. with sufficient advanced planning, agency-sponsored projects could become part of the class so that a truly experiential component could be added. two of our students had such experiences, but our timeline did not permit developing such opportunities for all students. our initial format allowed students to take important steps from knowing to understanding practice in their field (wiggins & mctighe, 2005); the modifications we suggest here would allow students to deepen that understanding and would provide opportunity for assessing and validating this kind of learning. c. blended and distance learning. our format uses distance methods in order to expand the variety and number of people involved in the experience. both experts and practicing professionals attended almost exclusively in virtual space—as distance teachers and learners. during expert-led discussion, the three segments of the learning community—expert, practicing professionals, and graduate students— were all at a distance from one another. readings were made available on-line but were primarily peer-reviewed literature that did not originate in an on-line format. students submitted reading discussion questions through the course-management system, but then met with the instructor in class for reading discussion that included distant practicing professionals. except for reading the discussion readings, all class components were synchronous. thus, our format fits milne’s (2006) wider definition of blended learning: “a course or program that is accessible by distance and on-campus students simultaneously (supported by videoconference, for example). narrower definitions of blended learning (e.g., in garrison & vaughan, 2008) focus on the opportunities for reflection that arise from asynchronous uses of online material. the usual nature of a graduate seminar is of synchronous discussion, and our practicing professionals faced time constraints that also made the synchronous format attractive. however, in the asynchronous reading of peer-reviewed articles and creation of discussion questions, our format also partook of this narrow definition. graduate students’ questions reflected their variety of backgrounds and interests, and graduate students used the opportunity to ask their learning colleagues among the practicing professionals to reflect on the impacts of climate-change issues on resource management and policy in practice d. facilitating a multifaceted learning experience. the seminar posed unique challenges to us as instructors and facilitators. a new expert spoke every week, so we, as instructor-facilitator and practitioner-facilitator, provided continuity and an ongoing sense of presence for both graduate student and practicing professional participants. sense of presence is important for engaging distance learners (lehman & conceição, 2010), and we were careful to communicate regularly with distance learners by email to provide them first meretsky, v.j., and woods, t.a.n. journal of the scholarship of teaching and learning, vol. 13, no. 3, august 2013. josotl.indiana.edu 56 with information and support for setting up the software and then to provide information on upcoming speakers and readings. this regular communication allowed practitioners to stay in touch with the class even if they participated only rarely, and we had no problems associated with uneven participation. even with a consistent physical facilitator presence in the classroom, some students missed the visual contact with the experts. some noted that it helped them to see a photo of the speaker projected before the start of class, and one speaker included a photo at the start of her presentation, to introduce herself visually to the audience. some challenges were merely logistical. because presenters could not see students, they could not respond to a raised hand or a quizzical or confused expression. similarly, the instructor-facilitator could not visually signal to expert presenters that students were or were not familiar with particular concepts. we encouraged presenters to focus on their presentations, and allow the facilitators to monitor the chat area of the adobe connect screen where participants typed questions during presentations. as a result, the responsibility for managing the flow the presentation rested entirely on the facilitators, who became meta-communicators in this context, communicating about communication cues. after the formal presentation, the question-and-answer discussion presented different challenges. both students and practitioners had questions. in addition, there were opportunities for facilitators to ask experts to expand on answers for the benefit of students and practitioners. similarly, facilitators often asked practitioners to expand on practitioners’ questions or answers either to receive the best benefit of expert knowledge or to provide clearer insights to students regarding the nature of agency interests. during these interchanges, the facilitators had to be comfortable with letting control of the flow of discussion rest as much as possible with the participants while remaining alert for opportunities to nudge topics or speakers briefly to enhance learning of many kinds. collison, elbaum, haavind, and tinker (2000) described the roles (‘guide on the side,’ instructor, and group-process facilitator) and range of voices (generative guide, conceptual facilitator, reflective guide, personal muse, mediator, and role player) available to moderators in online settings. although our setting was more complex, these roles and voices are apt for it as well. during expert presentations, we acted primarily as facilitators of the group learning experience, occasionally stepping into the role of ‘guide on the side’ to highlight overarching themes and connections to other presentations or to readings. as guides, we used the voices of generative guide (to facilitate discussion), conceptual facilitator (to clarify questions to experts and answers to graduate students and professionals), and reflective guide (to encourage deeper investigation of a point during question-and-answer discussion. the sessions that focused on discussions of readings were often dominated by student and facilitator interactions as agency personnel were less able to prepare and tended to listen (at most) during these sessions. in this context, we acted most often as ‘guides on the side’ of discussion and sometimes simply as traditional instructors when elaborating on the content of a reading. the voices of generative guide, conceptual facilitator and reflective guide were all useful to us as these discussions ranged from topics with which participants were familiar and engaged to topics that were less familiar or more complex. as guides, we not only invited consideration of the reading in isolation or in relation to expert presentations, but also often posed questions to prompt graduate students to consider the relevance of the reading to practitioners such as their professional co-participants. meretsky, v.j., and woods, t.a.n. journal of the scholarship of teaching and learning, vol. 13, no. 3, august 2013. josotl.indiana.edu 57 the agency perspective was available to graduate students both in expert presentations and in readings discussions as both of us participated in both kinds of sessions and both of us have worked with and for multiple federal agencies. further, some of the students had internship or employment experience with agencies and could contribute agency perspective from these experiences. nevertheless, agency issues were typically a smaller part of discussions of readings than of discussions with expert speakers. e. limitations of the format. the reality of agency work is that time rarely suffices for the work at hand, especially in an era of straitened financial circumstances. as a result, top-down prioritization of class activities is important to bring agency practitioners into the classroom. deadlines, meetings, and other responsibilities create a constant tension for agency personnel, however much they may value the opportunity for learning. our second iteration of an agency-university seminar had no accompanying directive from a regional director and attendance rarely exceeded a scant handful of agency practitioners, despite the importance of the topics to fws work and the excellence of the speakers. the opportunity to address a major federal agency was clearly attractive to expert speakers and suggests that facilitators of similar efforts should try to ensure some base level of agency buy-in and participation. in the case of the first iteration, top levels of the agency clearly indicated that training was to involve many people; such cases are likely to be rare, however. more often, it may be easier to ensure agency participation by spreading the impacts among staff members by matching personnel to the topic or topics most closely associated with their work. such matching was likely going on during the course we have described here, accounting for some of the variability in numbers of participants. the format is sufficiently flexible that agency participation need not be a constant, but could be linked to particular topics. where long-standing relationships exist between agencies and universities, experiences of this sort could be negotiated well in advance and the ties between students and staff could be increased. students were very enthusiastic about the opportunity to research issues of interest to the agency for their research-paper assignments. such projects increase the experiential aspect of a course and provide students with closer contact and deeper understanding of the practitioner role. where such relationships do not exist, the opportunity to leverage the university’s ability to attract top speakers and to make use of student workers to fill information needs can be incentives to create or strengthen agency-universities ties. changes in policy and regulations and advances in technology and practice create demand for learning in agencies that must respond to the changes. such changes thus create opportunities for applying this format to the benefit of both practitioners and practitioners-in-training. even in the best circumstances, seminars such as this one cannot provide the duration or depth of practitioner interaction of an internship or major project-based experience. the process of transforming graduate students into practitioners is generally a long one, and cannot be entirely encompassed within the academy. however, classes in the format we describe here build on familiar class types (lecture-discussion and seminar) and make available to students a community of practitioners with little additional cost, while leveraging university resources on behalf of the participating agency. technological constraints on distance learning and teaching are continually diminishing. in addition to adobe connect, other proprietary software such as skype now supports conference meretsky, v.j., and woods, t.a.n. journal of the scholarship of teaching and learning, vol. 13, no. 3, august 2013. josotl.indiana.edu 58 calls and could be used for classes like this one. class size is limited primarily by the facilitators’ willingness to try to support discussion among the all participants and by any limitation on the number of callers into the conference or skype line. if students are to use the distance option when they are home ill or travelling, then a short training session at the beginning of the course is advisable as instructors cannot easily offer technical assistance during regular class sessions. f. conclusions. the format we describe relies on significant agency commitment to an educational or training opportunity. when such commitment is available, it affords students and instructors an opportunity for insight into authentic thought and practice that may exceed even experiences such as internships which may not involve reflection and synthesis. expert presenters were very generous with their time in part due to the minimal time requirements of the distance format and in part for the opportunity to speak directly to agency practitioners. having expert presenters gives students the benefit of the best sources of information, reduces time commitments for instructor-facilitators and allows agency partners to leverage the capacity of university partners to attract such speakers. the distance format allows agencies to train many of their personnel without travel costs, although the time commitment remains. we found the experience of facilitating a multi-faceted learning experience to be immensely satisfying professionally. we, our students, and our agency colleagues received cutting-edge training in an area of great importance in our professional lives; we made new contacts with experts in our field and gained training in online technology and facilitation. the overall costs in terms of our time were entirely reasonable. we recommend this format to agency-university partnerships in all fields when need for training is urgent or high-demand topics create high agency commitment to the process. acknowledgments we gratefully acknowledge our graduate students and agency colleagues for their participation and for their help in understanding this learning experience from their perspectives, and robyn thorson, regional director, us fish and wildlife service for her support of this experiment. writing was supported by a indiana university scholarship of teaching and learning writing retreat award. craig e. nelson and george rehrey provided considerable advice and editing. two anonymous reviewers were also provided helpful comments. appendices appendix 1. student survey. only survey questions relating to the practitioner aspect of the class are shown. formatting and numbering do not match the original survey. likert-scale questions scored 1-5 discussion moderators managed discussion traffic well. meretsky, v.j., and woods, t.a.n. journal of the scholarship of teaching and learning, vol. 13, no. 3, august 2013. josotl.indiana.edu 59 agency personnel contributed important points and questions to discussion students contributed important points and questions to discussion points and questions raised during discussion were relevant and interesting learning i learned useful things about the class topic from guest presentations i learned useful things about the class topic from discussions hearing from agency personnel in discussion was useful and informative readings were a useful part of learning in this course readings were relevant to the topic of the class for which they were assigned i would take another course in this format if the topic interested me the different backgrounds and experiences of class participants (students and agency personnel) was a positive aspect of the course open-ended questions 1) please comment on how effectively the guest presentations, readings, and discussion helped you learn about course topics. 2) for these three aspects of the course, were there specific techniques or practices that you felt made that aspect particularly effective? guest presentations readings discussion 3) for these three aspects of the course, were there specific things you felt reduced effectiveness and should be avoided in the future? guest presentations readings discussion 4) the circumstances that gave rise to the course suggested that we keep class size fairly large in order to provide maximum opportunity for training both students and agency personnel in this one-time situation. please comment on any impacts of class size on your ability to benefit from the class. please suggest discuss upper limits (of in-class participation, distance participations, or both or neither) that should be set on future uses of this format. you need not repeat points you have made in earlier answers. 5) please comment on the effectiveness of a collaborative agency/university approach to learning about breaking issues. 6) please comment on the effectiveness of distance teaching for learning about breaking issues. 7) if you attended some or all classes remotely, please comment on the effectiveness of distance learning/distance teaching for learning about breaking issues. meretsky, v.j., and woods, t.a.n. journal of the scholarship of teaching and learning, vol. 13, no. 3, august 2013. josotl.indiana.edu 60 8) are there aspects of presentations that are lost in the distance-teaching format that are important to you as a student? if so, please describe these, briefly. 9) are there advantages or disadvantages to you as a student to using the distance-learning format to bring agency personnel together with students to provide training on breaking issues? if so, please describe these, briefly. 10) please provide any additional comments here. appendix 2. expert survey. only survey questions relating to the nature of the class are shown. formatting and numbering do not match the original survey. 1) prior to this course, please indicate the total number of web-based seminars in which you have been a participant. 2) prior to this course, please indicate the total number of web-based seminars in which you have been a presenter. 3) how many times per academic year do you use conference calls to conduct business? 4) prior to presenting in this course, had you previously attended multi-session, web-based training or seminars? if yes, please describe the general nature of the training or seminar(s) below. 5) prior to presenting in this course, had you previously provided web-based training or led webbased seminars? if yes, please describe the general nature of the web-based work below. 6) have you previously used other web-based training or seminar software? if yes, please provide the name of the software. 7) please evaluate the moderators on the following attributes by circling a point on the scale that corresponds with your judgment for that attribute. 8) how likely are you to consider using a distance-teaching/distance-learning format for courses or training you direct? please comment briefly and mention changes you would make for effectiveness. 9) how likely are you to consider using a university/agency mix for courses or training you direct? please comment briefly and mention changes you would make for effectiveness. meretsky, v.j., and woods, t.a.n. journal of the scholarship of teaching and learning, vol. 13, no. 3, august 2013. josotl.indiana.edu 61 references bradley, f., steven, a., & ashcroft, d.m. (2011). the role of hidden curriculum in teaching pharmacy students about patient safety. american journal of pharmaceutical education, 75(7), 1-7. doi: 10.5688/ajpe757143 collins, a., brown, j.s., & newman, s.e. (1989). cognitive apprenticeship: teaching the crafts of reading, writing and mathematics. in l.b. resnick (ed.), knowing, learning, and instruction: essays in honor of robert glaser (pp. 452-494), hillsdale, nj: lawrence erlbaum associates, inc., publishers. collison, g., elbaum, b., haavind, s., & tinker, r. (2000). facilitating online learning: effective strategies for moderators, madison, wi: atwood publishing. dewey, j. (1938). experience and education, new york, ny: kappa delta pi. garrison, d.r., & vaughan, n.d. (2008). blended learning in higher education: framework, principles, and guidelines, san francisco, ca: jossey bass. jaye, c., egan, t., & parker, s. (2005). learning to be a doctor: medical educators talk about the hidden curriculum in medical education. focus on health professional education, 7(2), 1-17. kolb, d. a. (1984). experiential learning: experience as the source of learning and development, upper saddle river, nj: prentice-hall. lave, j., & wenger, e. (1991). situated learning: legitimate peripheral participation, new york, ny: cambridge university press. lehman, r.m., & conceição, s.c.o. (2010). creating a sense of presence in online teaching: how to “be there” for distance learners. san francisco, ca: jossey-bass. masella, r.s. (2006). the hidden curriculum: value added in dental education. journal of dental education, 70(3), 279-283. available online at http://www.jdentaled.org/content/70/3/279.full.pdf+html milne, a.j. (2006). designing blended learning space to the student experience. in d. oblinger (ed.), learning spaces (pp. 11.1-11.12), louisville, co: educause. accessed online 21 october 2012. url: www.educause.edu/learningspaces. pace, d. & middendorf, j. (2004). decoding the disciplines: a model for helping students learning disciplinary ways of thinking. new direction for teaching and learning, 98, 1-12. resnick, l.b. (1989). introduction. in l.b. resnick (ed). knowing, learning, and instruction: essays in honor of robert glaser (pp. 1-24), hillsdale, nj: lawrence erlbaum associates, inc., publishers. meretsky, v.j., and woods, t.a.n. journal of the scholarship of teaching and learning, vol. 13, no. 3, august 2013. josotl.indiana.edu 62 schön, d.a. (1987). educating the reflective practitioner, san francisco, ca: jossey-bass. snyder, b.r. (1970). the hidden curriculum, new york, ny: alfred a. knopf. wiggins, g., & mctighe, j. (2005). understanding by design, 2nd ed. alexandria, va: association for supervision and curriculum development. journal of the scholarship of teaching and learning, vol. 23, no. 2, june 2023, pp.96-99. doi: 10.14434/josotl.v23i2.33715 increasing student engagement with self-assessment using student-created rubrics alex wayne chambers william paterson university signalexc@gmail.com elizabeth a. harkins monaco william paterson university harkinse@wpunj.edu abstract: self-assessment is a formative process where students evaluate the quality of their own work. this paper describes a strategy for using student created rubrics as self-assessment tools to increase student engagement. initially, the instructor models how to identify criteria for mastery in an assignment. this is followed by students in small groups identifying criteria for mastery in their assignment. these criteria can be used to create either an analytic or holistic rubric. students use this rubric first to practice the assessment process through peer-review, followed by self-assessing their own work. students can also choose to conference with their instructor to discuss their feedback as well as reflect on the quality of their work. the student will then apply this feedback to their work prior to final submission. keywords: self-assessment; criteria-based rubric; instructor feedback: student engagement self-assessment self-assessment is a process in which students evaluate their work to see how well it meets the criteria required for the assignment (andrade & du, 2007) and is a critical skill for all learners to continually improve their work (carless et al., 2011). according to andrade and du (2007), the process of selfassessment begins with establishing expectations for an assignment, followed by students comparing their work to these expectations, often presented in the form of a model or rubric, generating feedback for themselves, and then applying their own feedback to improve their work. additionally, selfassessment should be an ongoing formative assessment process (andrade & valtcheva, 2009). however, it can be problematic if students lack experience in self-assessment (lew et al., 2010). therefore, instructors need to support students understanding the connections between feedback, their work, and how those connections can help them improve their work (quintion & smallbone, 2010). one way to do this is to involve students in creating the assessment process and using selfassessment. this way, students independently focus on the quality of their work, rather than relying on their instructor (andrade & valtcheva, 2009). increasing engagement with self-assessment munns and woodward (2006) identified three behaviors of student engagement that should occur simultaneously, which are: 1) students understanding what they are learning; 2) students valuing what they are learning; and 3) students actively participating in what they are learning. successful selfassessment increases student achievement, as well as actively involving students in their learning process (andrade & valtcheva, 2009), thus aligning with the behaviors of student engagement. when students engage in the process of self-assessment to create their best work, they can increase their mailto:signalexc@gmail.com mailto:harkinse@wpunj.edu chambers and harkins monaco journal of the scholarship of teaching and learning, vol. 23, no. 2, june 2023. josotl.indiana.edu chances of success, leading to higher levels of self-efficacy, and thus increasing their overall engagement in the learning process (mcmillian & hearn, 2008). by involving students in the process of creating a rubric (andrade & valtcheva, 2009), followed by students having opportunities in class to self-assess their work (carless et al., 2011), as well as reflecting on instructor feedback (quintion & smallbone, 2010), the instructor can further increase the chances of eliciting the behaviors of student engagement. modeling mastery criteria the first step in preparing students to self-assess their work is for the instructor to model the process of identifying the criteria to be assessed. the instructor will use an example assignment, (preferably one like the assignment students will self-assess), and model how to identify the criteria of that assignment required to demonstrate mastery. this can include the instructor explaining what criteria are required, and how they contribute to demonstration of mastery of the assignment. this should be followed by engaging students in discussion about the importance of establishing a core set of criteria for an assignment (inoue, 2005), while also pointing out how the means to demonstrate these criteria can vary. for example, if one of the criteria is for students to express a clear understanding of a concept or skill, this can be demonstrated in various forms; written as an essay, as a presentation, or a multimedia project. it is important for students to understand how to monitor their progress of the assignment by focusing on their performance, then comparing it to the established criteria (andrade & valtcheva, 2009) to achieve mastery. the instructor should explicitly inform students this exercise is to prepare them to identify the criteria for an upcoming assignment that they will assess themselves. collaboration to identify criteria at this stage, the instructor will review the assignment for self-assessment with the class and refer to the process of identifying criteria required for mastery that was previously modeled. this should include discussion on how this assignment aligns with the goals of the unit or lessons that have been taught so that students can get a clear understanding of the purpose of the assignment. the instructor will assign students to small groups, with no more than five students in a group to achieve optimal cohesion and sociability (akcaoglu & lee, 2016). each group will be given time to discuss and identify the criteria for mastery. each group will then present their criteria, with the instructor recording their responses in a medium visible to all students. the instructor can then ask students to identify the common criteria as a starting point to eventually finalize a list of the criteria for mastery. as the discussion continues, the instructor should remind students that while there may be disagreements, the class will have to eventually come to a consensus (inoue, 2005). this process of students being involved in the creation of the rubric and identifying the criteria for mastery helps students become more familiar with the assignment (andrade & valtcheva, 2009). rubric creation once the instructor has established a consensus among the students, they can present options in which the criteria will be presented. one option is a criteria-based analytic rubric, which focus on each criterion separately (brookhart, 2013). the instructor can model how to create an analytic rubric for the assignment that contains individual criteria aligning to the objective of the assignment, as well as a description of performance quality of each of these criteria. this includes how to create a performance-level description for each criterion. the performance descriptions should be clear, 97 chambers and harkins monaco journal of the scholarship of teaching and learning, vol. 23, no. 2, june 2023. josotl.indiana.edu provide a description of the performance across a continuum of quality, be distinguishable between levels of quality, and contain a description of the assignment objective at the appropriate performance level that is easily identifiable (brookhart, 2013). as outlined in table 1, instructors can demonstrate point-based scores for each level of performance to easily establish an overall score and assign terms that indicate levels of proficiency. the addition of terms that indicate proficiency clarifies the quality of student work, as opposed to simply using numerical scores. table 1. sample criteria-based analytic rubric. advanced (5 points) proficient (4 points) approaching proficient (3 points) below proficient (2 points) novice (1 point) criteria 1 performance description performance description performance description performance description performance description criteria 2 performance description performance description performance description performance description performance description * this table illustrates an example of a criteria-based analytic rubric. however, some may find creating an analytic rubric quite time consuming. another option is a holistic rubric, see table 2. as opposed to a criterion-based analytic rubric that contains objectives with performance levels, a holistic rubric includes all criteria to be assessed along with a performance description of each criterion (brookhart, 2013). while this rubric does not identify specific levels of performance, it allows for evaluation of criteria, and it is less time intensive to create as opposed to an analytical criterion-based rubric. the instructor can survey students as to which rubric they would prefer, then decide if they want students to create the rubric as part of the assignment, or to create the rubric themselves using the agreed upon criteria for mastery, and then distribute it to the students. table 2. sample holistic rubric. criteria 1 criteria 2 • performance description • performance description • performance description • performance description * this table illustrates an example of a holistic rubric. peer-review practice and self-assessment initially, students will use the student-created rubric to monitor the quality of their work as they progress on completing the assignment by comparing their work to the agreed upon criteria. once they have completed a first draft of the assignment, the instructor will pair students up to peer-review each other’s work, using the student created rubric. this will create an opportunity for students to practice assessing the assignment using the rubric, to discuss their reasoning and provide feedback to each other. once students have practiced assessing the assignment in peer review, they will use the rubric to self-assess their work. students will then submit their self-assessed assignment for instructor review. the instructor will provide feedback, aligning comments specifically to the rubric. the student should have the option to conference with the instructor to discuss and reflect on the feedback provided, as well as to provide a rationale to the instructor, supporting their assessment decisions. 98 chambers and harkins monaco journal of the scholarship of teaching and learning, vol. 23, no. 2, june 2023. josotl.indiana.edu this way, both the student and instructor can come to a consensus before the final version of the assignment is submitted and graded. references akcaoglu, m., & lee, e. (2016) increasing social presence in online learning through small group discussions. international review of research in open and distributed learning, 17(3), 1-17. andrade, h. & du, y. (2007). student responses to criteria‐referenced self‐assessment. assessment & evaluation in higher education, 32(2), 59-181. https://doiorg.liblink.uncw.edu/10.1080/02602930600801928 andrade, h. & valtcheva, a. (2009) promoting learning and achievement through self-assessment. theory into practice, 48(1), 12-19 https://doi.org/10.1080/00405840802577544 brookhart, s.m. (2013). how to create and use rubrics for formative assessment and grading. acsd. carless, d., salter, d., yang, m., & lam, j. (2010). developing sustainable feedback practices. studies in higher education, 36(4), 395–407. https://doi.org/10.1080/03075071003642449 inoue, a. (2005). community-based assessment pedagogy. assessing writing, 9, 208-238 lew, m. alwis, w. & schmidt, h. (2010). accuracy of students' self‐assessment and their beliefs about its utility. assessment & evaluation in higher education, 35(2), 135-156. https://doiorg.liblink.uncw.edu/10.1080/02602930802687737 mcmillian, j., & hearn, j. (2008). student self-assessment: the key to stronger student motivation and higher achievement. educational horizons. 87(1) 40-49. munns, g., & woodward, h. (2006) student engagement and student self-assessment: the real framework. assessment in education: principles, policy & practice. 13(2), 193-213. doi: 10.1080/09695940600703969 quintion, s., & smallbone, t. (2010) feeding forward: using feedback to promote student reflection and learning – a teaching model. innovations in education and teaching international, 47(1), 125135. 99 https://doi-org.liblink.uncw.edu/10.1080/02602930600801928 https://doi-org.liblink.uncw.edu/10.1080/02602930600801928 https://doi-org.liblink.uncw.edu/10.1080/02602930802687737 https://doi-org.liblink.uncw.edu/10.1080/02602930802687737 microsoft word pryorfinal.doc the journal of scholarship of teaching and learning, vol. 7, no. 1, may 2007, pp. 77-101. three professors’ teaching philosophy of education: strategies and considerations for undergraduate courses caroline r. pryor, kris sloan and funmi amobi1 abstract: this study investigated the impact of teaching about philosophical approaches on preservice teachers’ coherence-non-coherence perspectives. participants were 56 preservice teachers from two research universities in two states, and three professors at these universities. data were collected using (a) a 105-item philosophy of education scale (poes) (pryor, 2004b), and (b) professor and student self-reflections. a correlational matrix was used to determine the relationship among five philosophical orientations and seven dimensions of educational practice. results indicate that students are more consistent in their ratings of approaches they are less likely to adopt in their teaching than those they are more likely to use. implications include suggestions for enhancing foundations courses and the use of the poes as a reflective tool. i. introduction. professors of undergraduate teacher education courses often hear students express anxiety about demonstrating their teaching skills. although they ponder, write and re-write lesson plans and reflect about possible problems they might face (i.e. student behavior, appropriate content), we suspect that instead of solely seeking advice on tools, tips and tricks for their lessons, these preservice teachers might be better served by drawing on foundational understandings to support the tools they use. foundational knowledge in teacher education (philosophy, history and sociology of education) has long afforded teachers a means for self-knowledge about their beliefs about the goals of education—a self-knowledge that leads to clarity in making classroom decisions (feinberg & soltis, 2004; oliva, 2005; wactler, 1990 and others). however, many who teach in preservice teacher education programs have noticed that foundational courses (in particular philosophy of education courses) have become either less a staple or have entirely disappeared as stand alone courses in undergraduate programs. one response to this concern is that other areas (e.g., mathematics education) have become more attractive to doctoral students than coursework leading to foundations concentrations in the degree plan. or, perhaps the paucity of trained foundations professors (carbone, 1991; henry & shea, 1986) suggests that these professors have been courted to teach graduate level courses. if this latter statement is true, will undergraduate teacher education programs concentrate solely on pedagogical practice (methods, management and media)? to determine how professors who do teach in undergraduate programs might enhance students’ expression of their philosophical beliefs about teaching (i.e., their beliefs about how to approach teaching decisions), this study reviewed strategies and course methods used in three undergraduate courses. because foundational courses help pre-service teachers better understand the philosophic beliefs that underlie their goals and purposes of education (feinberg & soltis, 2004) or selection of 1 department of secondary education, southern illinois university edwardsville, box 1122, edwardsville, il. 62026, capryor@siue.edu, college of education, st. edward’s university, kriss@stedwards.edu and college of teaching education and leadership, arizona state university, funmi.amobi@asu.edu. pryor, c. r., sloan, k., and amobi, f. the journal of scholarship of teaching and learning, vol. 7, no. 1, may 2007. 78 particular teaching pedagogies (pryor, 2004b) —the study investigated students’ cohesive and internally consistent philosophic approach to teaching. teaching about philosophical foundations of education has a long-standing and wellregarded position in teacher education, a status that has remained fundamentally unchanged (hlebowitsh, 2005, oliva, 2004; tanner & tanner, 1995). for example, an overwhelming number of teacher educators continue to [state that] they believe philosophy of education plays an important role in undergraduate education (e.g., gunzenhauser, (2003); rainer & guyton, 1999 and others). teacher accreditation organizations also remain dedicated to the benefits of including educational foundations in undergraduate course work. the national council of accreditation of teacher education (ncate) (2000 revision) states that candidates of accredited institutions should be prepared with a sound foundational base of the role of the teacher in education. in order to respond to this requirement, the council of learned societies in education (clse) developed two standards used in ncate institutional evaluation: (a) standard ii, candidates demonstrate skill in interpretation of the goals of education, and (b) standard viii, appropriately prepared faculty3. it seems appropriate then that the mission to enhance philosophy of education courses should not be underestimated. morey (2001) compared the quality of teacher education programs in universities with programs offered by for-profit providers with a discernable effect noted in programs with a provision for the translation of theory into practice. morey states: a disturbing fact about [for-profit providers] is its reliance on practical experience and practitioners at the expense of theory. the ability of these practitioner-trained teachers to make judgments about effective educational practices could be seriously impaired by their lack of understanding of educational theory [and] probably will not produce the type of reflective practitioners that many believe are essential for the improvement of today’s schools (pp. 309-310).” it appears, therefore, that the long-standing inclusion of foundations knowledge integrated into an undergraduate course is not a poorly thought out idea. anyone who has tried his or her hand at carpentry knows that predicting the outcome of a project built with a poor foundation is not difficult. what is difficult is how to respond to two programmatic concerns, vibrant course delivery and professor preparation. central to program delivery, butin (2004) suggests, is the provision for “substantive inquiry, intellectual debate, and deep reflection” (p. 7). unspoken in this second concern is a substantive discussion of high-impact strategies that support faculty delivery of this foundational content and an understanding of the insights students’ might derive as a result of learning about their philosophical approach to teaching. ii. literature review. the suggestion that knowledge of philosophy of education can be extrapolated as practical knowledge, that is, it can serve to inform, frame, justify and clarify the work of practitioners, is well regarded by many (arnstine, 2002; leahy & corcoran, 1996; petress, 2003; pryor, 2003a; schonwetter, sokal, friesen, & taylor, 2002; soltis, 1986). for preservice teachers, however, these benefits sometimes appear oblique. to counter this difficulty, petress 3 in 2002, ncate included these two standards as part of the narrative in all standards; see for example, standard 3 on teacher dispositions. pryor, c. r., sloan, k., and amobi, f. the journal of scholarship of teaching and learning, vol. 7, no. 1, may 2007. 79 (2003) urged us to provide students with “guideposts” or opportunities for students to draw on their developing philosophies when thinking about classroom instruction, a reflexive weighing of educational decisions. similarly, schonwetter, (2002) favored the notion that understanding one’s philosophical approach would foster evaluation of teaching decisions, particularly as teachers find themselves evaluated on external measures such students’ standardized test scores. as professors seek to link philosophical understanding with classroom decision-making, the actual beliefs held by these preservice teachers can become marginalized to more pressing issues of learning how to teach (e.g., how to administer a spelling test, or grade essays) (leahy & corcoran, 1996). providing students with a means to identify their beliefs is one strategy from which they might draw clarification when selecting teaching strategies. a. benefits of philosophical coherence. the aim of reflecting on philosophical approaches is not to cement preservice teachers’ orientations into pre-figured, categories that could minimize their efforts to make sense of the complexities of classroom life. rather, the aim is for preservice teachers to draw upon prompts such as a metaphoric image or an analytic survey to facilitate a self-examination process. the goal of this self-examination is the creation of a coherent philosophic framework, which makes possible the navigation of classroom complexities. a coherent philosophic framework is one that is internally consistent epistemologically, ontologically, and axiologically. antonovsky (1987) uses the phrase "sense of coherence" to describe a belief system in which the world is viewed as comprehensible, manageable, and meaningful. comprehensibility is the degree to which one perceives a predictable, ordered, and explicable world; manageability is the degree to which one believes that he or she has the personal and social resources to handle a demand. complementary to these beliefs is meaningfulness in which one believes that demands are challenges worthy of investment and commitment. the research on cohesion indicates that sense of coherence is a construct that significantly influences an individual's adjustment to the complexities of daily experiences (antonovsky, 1987; korotokov, 1998; lustig, rosenthal, strauser, & haynes, 2000; motzer & stewart, 1996; soderberg, lundman, & norberg, 1997; szymanski, hershensen, enright, & ettinger, 1996) a framework that lacks internal coherence is apt to result in what emerson (1841) called a “foolish consistency,” that in the end leads to novice teachers either misapplying or misusing theory (haggerson, 2002). at times this misapplication might not appear salient to the preservice teacher as their urgency to “get my lesson ready” is --understandably—prescient, but inadequate for resolving dissonance within practice. thus for many preservice teachers, an incomplete understanding of assumptions underlying coherence can result in later professional frustration or stagnation (kalimo & vuori, 1990). growing a preservice teachers’ sense of philosphic coherence might moderate these effects (noddings, 1995; szymanski, hershensen, enright, & ettinger, 1996). thus some discussion regarding programmatic pathways to growing these understandings might prove helpful to teacher educators. b. program concerns. the case that foundations courses can be well taught, and relevant to students and faculty remains a programmatic challenge (bredo, 2002; burbules, 2002; edel, 1972). steiner (2004) claims the preparation of teachers is intellectually barren and focused on indoctrination, which pryor, c. r., sloan, k., and amobi, f. the journal of scholarship of teaching and learning, vol. 7, no. 1, may 2007. 80 has been refuted by butin (2004). foundations topics do appear in undergraduate education programs, however there are some concerns about the delivery of this content. the first concern is that philosophy of education is often presented as a limited section within an introductory course, thus fragmenting philosophy from the overall course topic (appleton, 1979; petress, 2003; steiner, 2004). the second concern regards unevenness of content delivery when often it is taught by under-prepared faculty (shea, sola & jones, 1987). towers (1991) suggests that even among prepared faculty, teaching philosophy and history of education is less favored than coursework in current social issues. however, in particular programmatic constructs, when course activities facilitate practical decisions (deciding on a grading policy, a homework policy) towers posits, understanding the foundations of education can become highly valued by both students and professors. some course activities are particularly well regarded by students. for example, gross (1996) used a didactic questioning framework to foster the link between preservice teachers’ reflections on the role of the teacher, course readings, and observations during students’ field experiences. wactler (1990) found that student teachers’ journal reflections on the role of the teacher were efficacious to understanding personal teaching beliefs. rainer & guyton (1999) noted that learning about their philosophical approaches provided preservice teachers with a basis for discerning which of their mentors’ practices they might want to adopt in their own teaching. lastly, fen (1967) suggested that acquiring a personal philosophy of education enables preservice teachers to answer questions about how they substantiate their practice, particularly when conferencing with a classroom mentor teacher. c. how can teacher educators help pre-service teachers understand philosophy? teacher educators face several challenges as they attempt to support students understanding of their philosophical approach to teaching. first, although the benefits of understanding one’s approach to education is well described, some literature on the legitimacy of philosophy of education presents an unflattering picture of its place and possibility for survival in teacher education (bredo, 2002; burbules, 2002; carbone, 1991). in part, this survival is challenged in the present educational climate by an emerging predominance of competencybased outcomes (cbo); there are some indications that cbo might serve to marginalize efforts to portray the effectiveness of understanding one’s educational philosophy (guzenhouser, 2003). second, the construct of traditional approaches to teaching philosophy might also have exacerbated the perceived lack of importance of learning about philosophy. for example, in most educational foundations textbooks, the traditional approach is didactic, using explicit content delivered in a linear-lecture model (butin, 2004). this approach typically begins by defining the term “philosophy” and related explanatory terms such as “realism,” “idealism” and others. having provided this scaffold, the instructor delves into an exposition of the various systems of educational philosophy, beliefs about the purpose of education, curriculum and role of teachers and students. before moving on, the instructor, mindful of the need to have students construct their own philosophies of education, offers a concluding exercise to that effect. critics therefore can easily point to the dissonance between philosophers’ and educational practitioners’ perspectives about the application of philosophy of education in finding workable solutions to educational issues. in this regard, philosophy of education is often deemed too abstract to provide guidance to the everyday concerns of practitioners (carbone, 1991). in the everyday realities of classroom life teachers are not always guided by episteme —a pryor, c. r., sloan, k., and amobi, f. the journal of scholarship of teaching and learning, vol. 7, no. 1, may 2007. 81 theoretically created procedural of teaching. invariably, teachers’ reactions are driven by what korthagen and kessels (1999) termed phronesis, that is, situation-specific knowledge of teaching created by the teacher. programmatic concerns would do well to consider processes that help preservice teachers combine an episteme-phronesis gestalt to good advantage during their preservice preparation. to facilitate this gestalt to teaching educational philosophy, teacher educators increasingly turn to more reflective approaches. reflective approaches. reflective approaches provide experiences that elicit introspection on the assumptions and implications of philosophies of education (preskill, 1979). a common method of promoting reflection is through the use of educational surveys, such as the witcher-travers survey of educational beliefs (1999), which assesses tendencies toward transmissive or progressive beliefs. this instrument is composed of stem items related to educational beliefs (“tests are a good measure of student knowledge”) each of which is rated on a five-point likert-type scale and scored for two oppositional philosophical approaches— behaviorism and progressivism. another instrument, and the one used in this present study, the philosophy of education scale (poes) (pryor, 2004b) proffers cross-classification of five approaches to teaching (executive, humanist, subject specialist, citizen, and explorer) with seven dimensions of instruction such as classroom environment, lesson plans, or classroom management. other reflective approaches involve the development of metaphors of teaching or reflection based on field experience observations, or journal writing (wactler, 1990). according to amobi (2003), “…writing a metaphor of teaching requires tapping into one’s personal experiences to inform one’s teaching” (p. 28). thus, the main focus of employing reflective strategies in teaching educational philosophy is to expand preservice teachers’ self-understanding at a time when they are developing value judgments about teaching decisions. professors’ expectations. belief in reflective approaches are the work of the three professors involved in this present study; however each utilizes different strategies to achieve reflexivity among the preservice teachers in their classes. below, we offer first person descriptions of our goals for (a) students’ understanding of the purpose of education, (b) the deficit of current affordances used in our courses (i.e. philosophy statements, metaphor, autobiography), and (c) expectations of the use of the philosophy of education scale in each of our courses. these offerings are reflections of authors a (pryor), b (sloan) and c (amobi). although we used reflection strategies in our courses, our expectations for this reflection differed. our common expectations for our students was that reflection on philosophic approach would enable a student to (a) become more philosophically coherent in selecting teaching strategies, (b) make sense of coherence in their approach, and (c) use philosophical approach to disaggregate school controversy. at the end of the results section, we reflect on some changes we plan to use in our courses. professor a: my purpose in using the poes was to provide students with a strategy to evoke a personal reservoir of meaning, or at least an approximation of meaning of beliefs about teaching (wactler, 1990). not unlike the experience of professor b, i found my preservice teachers either lacking or hesitant in their ability to draw from their autobiographical memories of school, or from case studies provided in initial coursework when they attempted to explain their educational beliefs. for example, when asked to explain the role of the teacher, the prompt questions i used were, “think about the stories you remember about teachers and school,” and “when you consider the role of the teacher, what comes to mind?” in response, one student wrote: “my role is to help each student become the best person they can be.” what appeared missing from this student’s beginning explanation was a deep understanding of how her pryor, c. r., sloan, k., and amobi, f. the journal of scholarship of teaching and learning, vol. 7, no. 1, may 2007. 82 perception of the role of the teacher will be acted on; that is, given her belief, what is her target teaching action and how consonant is this action with this her beliefs about teaching? the prompts (indicators on the poes described below) appeared to me to function as markers (events in currere, (pinar, 1975) much as ausubel’s (1963) advanced organizer functions as markers of experiences remembered and useful in the projection of expectations and hopes. i hoped the poes prompts would unveil coherence-non-coherence of beliefs en route to a students’ next ontological step about selecting teaching practices. my thinking was that the poes could provide students with a strategy for disaggregation of their beliefs about schooling and enhance their understandings of the beliefs underlying the decisions they would make about classroom practice. professor b. although autobiography has been a rich source of inquiry and theorizing in the curriculum field and teacher education since the 1970s when pinar (1975) first purposed currere, i have frequently been frustrated in my attempts to utilize autobiographic techniques in my teacher education courses. although currere has been described as both a method and theory of curriculum (pinar & grumet, 1976), a significant number of pre-service teachers in my courses struggle in their attempts to produce self-focused autobiographical narratives that further their own philosophic understandings of curriculum and pedagogy. worse, i sense that some preservice teachers simply perform autobiography to prove to their professor they have "learned" something. thus, i turned to the use of the poes as yet another reflective strategy, admittedly a more structured strategy that both promotes self-understandings and helps pre-service teachers develop a more coherent philosophic understanding of their educational decisions. in this investigation, i hoped to learn how the use of the poes might help me better understand preservice teachers’ perspectives on philosophic approaches and develop additional strategies to foster self-understanding and philosophic coherence. professor c. i had hoped to use the poes to provoke students to make sense of experiences that structures their emergent philosophies of education. i hoped the instrument would become a triggering event for students to reflect on past and present knowledge and experiences—and spur their philosophic classifications (dewey, 1933; pryor, 2003a). i compared the poes to the witcher-travers (1999) philosophy scale, finding these complementary to each other on three points: (1) the witcher-travers provides three broad classifications, while the poes provides five philosophic themes; (2) the witcher-travers incorporates questions that subsume indistinct commonplaces of teaching, while the poes includes prompts that are derived from distinct commonplaces of teaching and learning (i.e., classroom environment, lesson plans, classroom management knowledge/instruction ); and (3) the online witcher-travers provides immediate, detailed feedback to survey-takers, while the poes results are calculated and aggregated by participants, and/or external researchers. i respond in this article to my use of the poes and its efficacy for providing a contextual environment for class discussions. given our interest in understanding the philosophical beliefs of our students this study investigated philosophical coherence-non-coherence among preservice teachers. as an outcome of this investigation, we will present course strategies that might enhance preservice programs. pryor, c. r., sloan, k., and amobi, f. the journal of scholarship of teaching and learning, vol. 7, no. 1, may 2007. 83 iii. methodology. a. participants. the 42 students in instructor a and b’s courses were enrolled in a research i university. instructor a’s course was composed of 27 first semester seniors enrolled in a field-based course in elementary methods of teaching social studies; instructor b’s course was composed of 15 second semester juniors enrolled in curriculum development and instructional strategies in early childhood education. instructor c’s students (n=14) were second semester secondary education juniors and seniors enrolled in critical issues in secondary education at a branch campus of a research i university in another state. all three professors were trained in philosophy of education in their doctoral programs. b. course procedures. professor a. in a program entitled citizen teacher (pryor, 2003b; 2004a), students discussed theories related to three themes of democracy, liberty/freedom, justice/fairness/ and equality/equal opportunity using one text on democratic practice and a second on philosophy of education. the social studies methods course was linked to a 20-hour per week fieldexperience in which preservice teachers observed and participated in limited entry-level teaching. in addition to discussions related to social studies content, the course experience included reflection on: (a) educational biography, (b) observations of mentors’ practice, and (c) observation forms centered on identifying democratic practices. the capstone activity included completion of the poes and development and analysis of a philosophy of education statement. professor b. this course examined the curriculum and pedagogies used in early childhood education, often of opposing viewpoints, included: classroom debates, viewing of films, and presentations by community-based educators. students were encouraged to think critically about the promises and limitations of various models of curriculum and pedagogy and afforded extended opportunities to mine their previous experiences through reflective essays. lastly, students were asked to develop, over the length of the semester, metaphoric images of the classroom and their vision of teaching practices (connelly & clandinin, 1988). these metaphors were considered “figurative tropes” (coffey & atkinson, 1996, p. 84) that helped preservice teachers better access their own philosophic framework as well as epistemological, ontological, and axiological assumptions about curriculum and pedagogy. such imagery unveiled personal and situated knowledge that might otherwise remain unrevealed, or, as in a failed or mixed metaphor, revelations of levels of confusion or ignorance not otherwise seen (coffey & atkinson, 1996). professor c. this course examined past and current controversies regarding public schooling focusing on secondary education. students were encouraged to develop their own viewpoints about controversial issues. it was hoped that exploring these viewpoints through the lens of philosophic background knowledge would enhance students’ experience and abilities to analyze the underpinnings of ideas espoused by “respected voices” in education whose writings spanned the readings for the course. students completed the witcher-travers survey and the poes and reacted to the outcome of classifications, after which the class interpreted the philosophical tendencies of major characters in popular high school movies such as to sir, with love, dead poet’s society, dangerous minds, or mr. holland’s opus. following this activity, pryor, c. r., sloan, k., and amobi, f. the journal of scholarship of teaching and learning, vol. 7, no. 1, may 2007. 84 students wrote metaphors of teaching, which were then compared to their perceptions of a teaching metaphor in their chosen ‘movie’ teacher. the purpose of combining traditional and reflective approaches in teaching was to encourage the propensity that “philosophy of education is not just the way we think, but also the way we do” (amobi, 2003, p. 27), and to provide students a framework for analyzing the philosophical assumptions that previously stranded the controversial issues brought forth in the course. c. instrument. philosophical orientations. five categories of philosophical orientations or beliefs central to the poes have been described in the literature as philosophical teaching approaches (feinberg & soltis, 2004; oliva, 2005; pinar, reynolds, slattery, & taubman, 2000; tanner & tanner, 2000). the five approaches used in the poes are: (a) executive (behaviorism, a production model), (b) humanist (progressivism, student centered), (c) subject specialist (perennialism, content focused) (tanner & tanner, 2000), (d) explorer (deconstructivism, revealing social myths, pinar, reynolds, slattery, & taubman, 2000), and (e) citizen teacher (essentialism, core civic values, ravitch & thernstrom, 1992). additional description of these approaches is found in appendix a (see also pryor, 2004b). dimensions. the philosophy of education scale (poes, see appendix b) is composed of seven dimensions of teaching, derived from the core standards of the interstate new teacher assessment and support council (intasc) (1992), and the national council of accreditation of teacher education (ncate) standard one (2000 revision) of effective teaching: classroom environment, lesson plans, classroom management, activities, grading/evaluation, knowledge, and teacher’s role. the seven dimensions of the poes are triangulated across five philosophical teaching approaches described in the literature. in all, the poes is comprised of 105 indicators, each independently rated, and 35 philosophical approach items, each ranked. studies of the 105item poes investigating preservice teachers (pryor, 2003b; pryor & eskirmireh, 2004), report reliability ranging from .61 to .68, considered well above the benchmark range of 0.50 to 0.60 set by nunnally (1967) for an instrument intended as an analytic tool.4 reliability in this present study is r =.71. d. data analysis. philosophical classification. each philosophical approach (e.g. executive) calculation was determined by classifying an individual into an approach type—termed an identifier—if their obtained score reached one half of one standard deviation above the mean score of the total sample score; the total score possible is 105. for example, if the average score for this sample of 4 within each dimension (e.g., lesson plans), five cells represent each approach; each cell is composed of three indicators representing a particular approach. each indicator is independently rated using a five point evaluative scale, after which the cell is compared across the five approaches of a dimension and each approach is ranked using the five-point scale. the summated ranked scores derive an overall philosophical orientation score. these ranked scores also portray the contribution of each dimension to overall philosophical approach. reliability studies of the poes were developed by (a) determining indicator-cell coefficients (cronbach’s alpha), (b) determining coefficients for each of the seven dimensions and five corresponding philosophical approaches, and (c) averaging the mean coefficients of either the five approaches or the seven dimensions, leading to the same result. in this present study, data were collected by each professor from participants at the middle of the semester as part of their coursework, and results were discussed in class; small sample size prevented replication of reliability. pryor, c. r., sloan, k., and amobi, f. the journal of scholarship of teaching and learning, vol. 7, no. 1, may 2007. 85 students for the executive approach is 80 (sd=10), then a score of at least 85 will result in an executive categorization. an individual can be classified into more than one approach category or not classified into any one particular category. the sum of the number of participants in each category, then, will not total the sample n. this procedure was selected to respond to the analytic nature of the instrument in which participants draw on more than one philosophical approach when evaluating classroom practice (wactler, 1990). coherence matrix and coherence pairs analysis. a correlational matrix was developed to determine the degree of coherence in each philosophical approach. this matrix was developed by correlating the seven dimensions of teaching on the poes with each of the five philosophical approaches. further analysis was conducted to disaggregate the coherence matrix data to determine the high and low correlations pairs for each philosophical approach (e.g., coherence of lesson plans and classroom management in the executive philosophy). correlational pairs of at least r=.50 were considered highly correlated. in order to provide information to enhance development of instructional course construct, the data pairs reported below regard only those pairs in which high correlation exists; the remainder of the correlational pairs were considered low and not reported here. iv. results. a. do preservice teachers hold a coherent philosophical approach to teaching? as portrayed in table one, students were more consistent in what they believe they are not (philosophically) than what they believe they are. this is true among all the philosophical approaches except the executive approach, in which students who are primarily categorized as “executive” consistently indicate low coherence to all of the approaches. the average correlations describing the level of coherence for those who are identified as belonging to a particular approach are: executive (r=0.19, sd=0.27), humanist, (r=-0.01, sd=0.21), subject specialist (r=0.23, sd=0.28), explorer, (r=0.14, sd=0.18), and citizen teacher (r=0.06, sd=0.25). students were consistent about which philosophical approach they were not. for example, students who were classified as humanists were not only the least aligned to their own overall philosophical orientation (-0.01), they were also most consistent in describing themselves as not executive (r=0.63, sd=0.15), not subject specialists (r= 0.49, sd=0.13), not explorers (r=0.48, sd=0.16), and somewhat not citizen teachers (r=0.32, sd=0.19). table 1. correlations among philosophical identifiers by approach category. executive humanist subject specialist explorer citizen teacher m sd m sd m sd m sd m sd classified as executive .19 .27 .16 .19 .26 .24 .31 .21 .14 .23 classified as humanist .63 .15 -.01 .21 .49 .13 .48 .16 .32 .19 classified as subject specialist .27 .25 .42 .27 .23 .28 .55 .19 .49 .18 classified as explorer .67 .13 .29 .23 .38 .18 .14 .18 .25 .25 classified as citizen teacher .67 .14 .11 .24 .30 .19 .28 .21 .06 .25 pryor, c. r., sloan, k., and amobi, f. the journal of scholarship of teaching and learning, vol. 7, no. 1, may 2007. 86 the seven dimensions of teaching on the poes provide contextualizion of the five philosophical approaches that frame initial understandings of teaching decisions. of these, four relate to pedagogy (classroom environment [ce], management [cm], lesson plans [lp], and activities [a]), one to domain area (knowledge [k]), and two are related to school policy (grading [g] and the role of the teacher [tr]). as an example of information that can be learned from the disaggregation of students’ teaching beliefs across these seven dimensions, we provide in table 2 one type of dissaggreation of the poes using the executive approach as an example. in order to more fully understand the [dimensional] source of these executive identifiers’ coherence-non-coherence, we portray in table 2, high correlational pairs indicating the executive identifier coherence to the approach, and high correlational pairs indicating their noncoherence to the approach. the data in table 2 was developed as we asked the following question: do student scores indicate an internally consistent approach as we look across the seven teaching dimensions? we learned that some non-coherence to an approach can be identified, and which teaching dimension pairs were related to this non-coherence. in this study, executive identifiers are coherent in their approach on the dimension pairs of lesson planning, classroom management, and classroom activities. these three areas were important to the executive identifier—students valued (and might likely use) the executive approach in these three dimensions of teaching. in a classroom activity about the findings of the poes, a student in professor a’s course explained why she might use the executive approach in classroom management: my mentor teacher knows the students in her classroom very well. she knows how to respond to students when they are off task and she understands the reasons behind their actions. i am learning to understand the students, but i don’t want the entire class to “fall apart” when i teach my first few lessons. i want things to go smoothly, so i give them rules however, when these executive identifiers think about the “knowledge” (content or domain area knowledge) dimension, they are not coherent in their philosophical approach. their pair ratings regarding the nature of knowledge were higher in approaches other than the executive. thus, even as executive identifiers, these preservice teachers held non-executive beliefs about the construct of knowledge. as one student pondered her goals as a teacher she noticed how strongly she believed in the value of students’ opportunity to experience the breadth and wide perspective of knowledge. she explained: here i want students to become critical thinkers. to do this, i will plan in-depth content units, such as a very broad unit on texas history, or a really good unit about money in my math lesson. i want them to be critical consumers, to know what a society has to offer them. if they only memorize facts, they lose out on thinking about possibilities—they miss reflection on content (professor a). through discussions and reflections, we also noticed how a teaching dimension such as “knowledge” might add or detract from coherence within a particular approach. for example, finding that those classified as executives were not influenced by the structure or content of domain area content knowledge (i.e., “i believe history or mathematics information should be taught only in a linear-behaviorist approach—that is teach the facts) when making classroom pryor, c. r., sloan, k., and amobi, f. the journal of scholarship of teaching and learning, vol. 7, no. 1, may 2007. 87 decisions gave us some insight into these students’ beliefs about how they might frame a lesson plan, or which activities they might select to use in the classroom. we noticed however, that although the knowledge dimension does not predict coherence for executive identifiers, we now have information that allows us to pay attention to the attributes of knowledge that does motivate the executive identifier. in this case, the executive identifiers’ high coherence pairs regarding their beliefs about the construct of knowledge was found not in the executive approach, rather in the humanist, subject specialist, and explorer approaches, and these three approaches were then paired to several other dimensions of teaching (“i’m an executive overall, however, i am a humanist in my beliefs about the relationship between knowledge and classroom management, r =.52). it is important to note that these scores portray a consonance-dissonance correlation at an identified point in time in students’ professional development. the scores also enhance professors’ understanding about what students believe about teaching and where gaps in their teaching knowledge might exist. when viewed by professors of preservice courses the sample correlations among dimensions can provide information that suggests revisions in course strategies. for professors teaching undergraduate courses, using strategies such as currere in which a students learn to project possibilities (pinar, 1975)— of a teaching approachprovides students with an opportunity to develop a “conceptual fund” upon which they might draw their future decisions about approaches to teaching (wactler, 1990). professor a asked her students to project/reflect on which teaching strategies comes to mind when addressing the issue of consonance-dissonance within a practice—such as grading policies: i always use the dimension of grading as a probe about student beliefs. i ask if they believe their professors grade fairly and if so, how would these same policies be used in k-12 settings. after all, they see us grade their work, they have had k12 teachers grading their work over the years, yet we expect these preservice teachers will develop a broad repertoire about approaches to teaching that will lead them to decisions based on a typically undiscussed rationale--their philosophy of education. few students link their philosophy statements to the choices they think they might make in classroom practice. they tell me that philosophy is abstract –like a made up story about teaching. they do not see how philosophy is useful in the field experience where they have to demonstrate what they know about teaching. in other words, pedagogical decisions, such as grading is not seen as a philosophical act (professor a). mindful of the suggestion of preskill (1979) to seek opportunities to discover how philosophy and practical activities might align, another common dimension of teaching—grading—is used as a discussion topic here. in this particular study, only the dimension of grading/evaluation was highly aligned with the other six dimensions (in each of the five approaches). this finding indicates that the dimension of grading more than any other dimension best represents the direction of an individual’s philosophical orientation. given the nature of grading, an evaluative process resulting in self-categorization (good student-bad student, kohn, 1986), it is not pryor, c. r., sloan, k., and amobi, f. the journal of scholarship of teaching and learning, vol. 7, no. 1, may 2007. 88 surprising that novice teachers, still in the evaluative environment of a university, can easily target coherence within their own approach using the grading referent. b. which dimensional pairs contribute most to coherence or non-coherence within each approach? as seen in table 1, high pair correlations differed between philosophical approach identifiers (e.g., “the executive approach is most like me) and non-identifiers (“the executive approach is not like me.”) these identifier-non-identifiers were also different when their philosophical approach ratings were triangulated using the seven poes dimensions of teaching (see, for example, executive approach correlations such as lp correlated with cm in table 2). we continue our use of the executive approach as an example in the discussion and table 2 below in order to highlight correlational pair results of approach identifiers and non-identifiers. preservice teachers who were categorized as “executives” were highly consistent in their ratings of two paired correlations, lesson planning and classroom management and classroom activities and classroom management. among non-identifiers, numerous dimension pair correlations were found above the .50 level. this finding corroborates the notion that pre-service teachers appear to know what they are not, rather than what they are. one example of nonidentifier high pair correlation concerns the dimension of grading, which is common among approaches of highly correlated pairs. for example, the correlation of ca and gr is r=.55 in the citizen teacher approach, indicating that citizen teachers strongly believe that they are not executive when they consider the relationship between ca and gr. table 2. high correlational pairs on the poes using an executive identifier example. executive ratings of executive approach lp & cm (.68) ca & cm (.62) ----------------------------------------------------------------------------------------------------------- executive ratings of humanist approach kn & cm (.52) ----------------------------------------------------------------------------------------------------------- executive ratings of subject specialist approach lp & kn (.64) kn & gr (.64) cm & gr (.54) lp & gr (.52) ----------------------------------------------------------------------------------------------------------- executive ratings of explorer approach cm & kn (.71) cm & gr (.58) lp & kn (.55) ca & tr (.53) ----------------------------------------------------------------------------------------------------------- executive ratings of citizen teacher approach ca & gr (.55) note. ce represents the dimension of classroom environment, lp represents lesson planning, cm represents for classroom management, ca represents classroom activities, gr represents grading, kn represents knowledge, and tr represents teacher’s role. pryor, c. r., sloan, k., and amobi, f. the journal of scholarship of teaching and learning, vol. 7, no. 1, may 2007. 89 interviews with our students provided us with feedback about how the poes helped with the process of self-identification of philosophical approach. one of professor c’s students told her: the poes classified me as a humanist. the results were very similar to the witcher-travers survey results. a pro [positive attribute] for the [poes] scale was the simple and clear way to categorize yourself. there were no awkward sentences to un-jumble. it was just picking what sounded more like you. one of professor b’s students told him: taking the [poes] survey was particularly meaningful to me because i was able to see myself more clearly, or the teacher i don’t want to become. often times, while i am working with my students [in my field placement], i find myself wanting to control the situation or guide them to the right answer. too often i will underestimate their intelligence or understanding of the situation and just sort of “give” them the answer for fear that they will become frustrated or lack the confidence to produce the right answer. in my head, i know that this is not the approach that i want to take, but something inside of me just wants to “baby” my students. in addition to seeking clarity in the format of scales or surveys, students also wanted instruments to help them make sense of their philosophic approach. wactler, (1990) found that in addition to word or phrase prompts, students wanted to discuss their approach with their professor – somehow checking to see if their approach is coherent in light of school concerns. as one student explained to her professor: the [poes] seemed to classify me correctly as well. i do feel that the teacher’s role is more than that of a facilitator, even though i am math [secondary]. the [poes] was easier to complete than the witcher-travers. however, i felt that the items were just scratching at the surface instead of asking directly about the issues (student in course, professor c). another preservice teacher responded to professor b: [our discussion of the poes] showed me my “executive” tendencies. for example, as a teacher i hope to plan lessons and activities a year in advance, but i also know that i have to take into account problems and situations that might evolve. my humanist tendencies, however, remind me take into account that i do not yet know the children i will have for that school year. overall, i’ve come to realize that it’s okay to be have different tendencies so long as i am flexible and responsive to children in my class. finally, we noted that students turned to their professor for guidance and discussion about the breadth of philosophical approaches that they might hold (i.e. do i approach teaching solely using one approach? should i only have one approach?). here, these students tell us, it is helpful pryor, c. r., sloan, k., and amobi, f. the journal of scholarship of teaching and learning, vol. 7, no. 1, may 2007. 90 for professors to contextualize school settings, so students can recognize how they might change approaches during their field experiences. a third student in professor c’s course explained: a con [negative] for both surveys is, from a statistical point of view, how accurate can they be? overall, i felt that both surveys were very accurate at this time [emphasis added] in my evaluation. i am eclectic and [likely think of myself as] a teacher as an executive to access the complete analysis of correlational data pair information see pryor1. c. which attributes of philosophical approaches might explain coherence or non-coherence? personal beliefs about the goal of education frame teachers’ stated lesson objectives (wactler, 1990) and these beliefs provide insights about how the approach might be applied in the classroom. as preservice teachers become experienced in describing their ideas about the goal of education, a tool such as the poes can be helpful in the developing the remaining construct of their approach to teaching. questions such as “under what teaching conditions would you want to use a particular approach?” are a helpful prompt to use. the poes offers students information beyond that of self-identification as it provides an opportunity to disaggregate which of the seven dimensions contribute to philosophical coherence. for example, teaching efficiently (using a quickly paced approach with linear and easily observed procedures) is the purpose of the executive approach; it would follow therefore that consistent, internally harmonious curriculum decisions would be developed by a teacher selecting this approach (ediger, 2003). table 2 above portrays these results. however, for other approaches, particularly those composed of attributes less objectively observed, complex, or non-unilaterally defined, approach coherence is often less apparent to the novice teacher (feinberg & soltis, 2004). the explorer approach is one example of complexity as this approach is centered on beliefs that abhor externally imposed objectives. in less objectively observed approaches, identifiers select teaching objectives that demythologize the benefits of the changing nature of knowledge (haggerson, 2002). kincheloe (1999) suggests that a democratic (citizen teacher) approach is strengthened by the discourse of chaos, and personal interpretation of objectives. from the perspective of the humanist approach, objectives should be unbounded without formulae (feinberg & soltis, 2004). similarly, in this present study, the subject-specialist approach was prescient in non-coherence, as the construct of this approach rarely includes use linear procedures, particularly on the dimension of grading (see pryor for full correlational data). students participating in this study noted that a two-dimensional survey such as the poes provides for convergence in their thinking about their teaching approach. this convergence within philosophical approach supports students as they begin to try out their teaching practice, in part because they have yet to establish a currere or experiential base in which their philosophical rationale resides (wactler, 1990). they need to practice their beliefs— try out what works, and seek convergence in what appears to them as “best practice” (rainer & guyton, 1999). dewey (1933) supported the need for application –trying out beliefs--stating, “the formation of purposes [philosophical approach] and the organization of means [practices] to execute them are the work of intelligence” (p.72). unlike the witcher-travis survey in which a student must judge a stem item in a likert-type question, the cells in the poes are composed of pryor, c. r., sloan, k., and amobi, f. the journal of scholarship of teaching and learning, vol. 7, no. 1, may 2007. 91 three indicators of a dimension, and the dimension is composed of five approaches. the rating of three indicators within a cell function in a less dichotomous manner than a stem item/likert rating format (huck, 2000). the three-indicator/cell format allows the indicators to converge into a cell score for the item, as well as to reveal the contribution of each independent indicator to a cell with stand-alone scores per indicator (see appendix b). one student explained how this type of dissaggregation helped them make sense of their teaching approach: the [poes] gives you more classifications. you weren’t bunched together [one stem item linked to a single score]. in addition, you had more control of the survey. you were able to rate yourself, and you knew where the classifications came from. i was able to see all of my sores in each row to see which areas are strongest in me. it gave me more options to consider. (student in course, professor c). d. are preservice teachers attracted to the objective, linear nature of an approach? preservice teachers find objective, linear teaching procedures efficacious (e.g., wactler, 1990), particularly when encouraged by mentors during the field-experience (pryor & kuhn, 2004). it also might be natural for preservice teachers to value the linear nature of an approach which they believe will allow for a strong level of classroom control and demonstration of teaching competence (enz, freeman, & wallin, 1996; veenman, 1984). in fact, wilkins-canter (1996) reported that the most requested information preservice teachers hoped to receive from mentors are strategies for “discipline.” however, students do indicate that they are eager to learn how to implement “hands on inquiry” and “exploratory projects,” the capstone activities of a humanist approach to teaching (guyton, rainer, & wright, 1997). for students to develop a belief in the importance of a non-linear philosophical approach to teaching, they must also believe that they are capable of using a classroom management plan in which they will not struggle with the use of open-ended inquiry approaches to teaching. e. implications for teacher education. the professor-researchers in this study identified the following three enhancement areas they planned to add to their courses. these enhancements reflected individual purposes, (a) professor a, unveiling philosophical assumptions of practice (feinberg, 2004); (b) professor b, developmentally appropriate practice (dap, bredenkamp, 1997 and others) and (c) professor c, ontological synthesis (bredo, 2002). to focus attention on the personal nature of the integration of these enhancements rather than suggest replication, these enhancements are discussed below in first person narratives. professor a. i focused my first class session on an introduction of the assumptions of each philosophical approach on the poes, followed by a session in which students discussed how each of these assumptions might be linked to beliefs about the role of the teacher and sample classroom activities. i hoped students would notice that the poes indicators represented personal meanings, rather than feel compelled to use the indicator/prompt as the only frame for their response. however, at this early point in the course students had little field experience from pryor, c. r., sloan, k., and amobi, f. the journal of scholarship of teaching and learning, vol. 7, no. 1, may 2007. 92 which to frame their poes ratings and discussion, and many simply replicated the language used in the prompts. i realized i needed strategies to foster students’ unveiling/demythogyzing of beliefs, and i identified the following course goals and used three teaching strategies to achieve these : goal 1. enhance student engagement students will develop a rationale to foster their engagement of particular classroom practices (wactler, 1990). teaching strategy: lengthen the autobiography section in philosophy statement. goal 2. understand assumptions students will be able to describe philosophical assumptions underlying goals of education (feinberg & soltis, 2004). teaching strategy: add philosophical content focus during classroom reflection discussion on students’ observations in the schools . goal 3. describe belief coherence-dissonance students will describe the coherence-dissonance of their beliefs (korotokov, 1998). teaching strategy: class discussion using poes results with students identifying where coherence-dissonance exists in their teaching observation reflections using personal rationale. by the end of the course, evidence of students’ understanding of their beliefs about teaching began to emerge in their responses during course discussions (e.g., “now that i understand my mentor teacher’s approach, i think my mentor teacher should…”) and some (but not all) responses grew into insightful analysis statements (e.g., “lesson plans that are objective [executive] focused are not always considerate of students’ needs. if i were the real teacher, i would change that part of the lesson plan”). professor b. drawing heavily on the guidelines for developmentally appropriate practices (dap) as defined by the national association for the education of young children (naeyc), (bredekamp & copple, 1997), the content of my early childhood courses is decidedly constructivist. through such an orientation, i emphasize the importance of instructional approaches that foster open-ended, child-determined experiences and guided discovery rather than teacher-determined experiences and teacher lecture. thus, i was surprised to find that most of the students in my course more closely aligned themselves with an executive approach. throughout the semester, i explicitly challenged tenets more closely associated with the executive approach, in particular the teacher-as-leader role and the limitation of direct instructional approaches in the classroom. these findings suggested to me that i had not done an adequate enough job of shaking free these preservice teachers from traditional (what i often call “default”) modes of teaching whereby the institutional circumstances and traditions of school tend to favor teacher directed, even teacher dominated, approaches. more importantly, however, the findings of the poes demonstrated that i needed to offer the preservice teachers in my class more explicit experiences with instructional approaches in which teachers serve as facilitators of learning rather than directors, even dictators, and that i needed to provide the pre-service teachers with more structured experiences in how to both manage and assess open-ended experiences through which children present multiple interpretations of their understandings. most of all, the findings of the poes demonstrated to me that it is not enough to talk about constructivist approaches to teaching and learning, but to demonstrate and operationalize such approaches(burlubes, 2002). to this end, i have begun to use more video vignettes of classroom teachers as a means to not pryor, c. r., sloan, k., and amobi, f. the journal of scholarship of teaching and learning, vol. 7, no. 1, may 2007. 93 only understand what a teacher is doing, but also speculate about possible alternatives that would foster child-directed inquiry and discovery. professor c. the differences in survey-question construction and procedures notwithstanding used together, both instruments provoked students’ reflectivity on their educational beliefs along two important lines. first, students had the opportunity to make meaning on their philosophical orientation separately on each instrument. second, they had the opportunity to commingle these isolated reflection-responses into an interpretive analysis by responding to both questionnaires. the reflective cogitations that the two instruments sparked in my students’ class discussions appear to me to corroborate dewey’s pronouncement that “the formation of purposes [educational philosophy] and the organization of means to execute them are the work of intelligence” (amobi, 2003, p. 77).5 v. conclusion. this study investigated three instructors’ course methods developed to help pre-service teachers better understand their philosophic foundations of education. the poes used in this study provided coherence information about preservice students approaches to teaching. professors and student comments suggested that more importance should be placed on developing course strategies that enhance opportunity for preservice teachers to portray emerging beliefs about their early experiences in schools. the data in this study indicated how a tool, such as the poes can be used to unveil non-linear relationships between seven contentpedagogical dimensions of teaching and philosophical teaching approaches. for example, even for those who held strong beliefs in which linearity is common—such as executive (behaviorist) beliefs—coherence is demonstrated contextually (e.g. “i’m ok with it [the executive approach] in developing activities for the kids, but not in classroom management. some of these kids really need the teacher to help them.”). we learned from both our course discussions, and in our administration of the poes that our preservice students were more likely to describe what they were not (“i am not an executive when i grade students”) than what they were (“i’m not always an executive”). the three professors in this study considered how these results might impact their teaching goals, and suggested three program areas and related activities for possible use in similar courses. the purpose of developing these philosophy to practice translations during actual coursework time and before the summative evaluations period occurring in the more intense field experience semester (semester two-student teaching) was twofold: (1) to mediate the potential of “washing out” of the use of theoretical rationale in teaching decisions (zeichner & tabachnick, 1981), and (2) to create a format for long term reflection on practice (wactler, 1990). however, this study was limited by the following factors: (a) participants represented two research universities in two separate states, (b) preservice teachers were enrolled in different program levels (elementary or secondary), (c) field experience either differed, or was not a part of the course, and (d) structured and non-structured texts were not used in a similar manner. several concerns remain regarding how course construct might effectively engage students’ knowledge about their beliefs about teaching. in part, little is known about the circumstances that contribute to students’ engagement in philosophical discussion—especially as 5 however, huck (2000) suggests that item response on similar topics or multiple instructions should not be interpreted as a main effect. huck writes that tests (such as a three-way anova) of these effects could be used. pryor, c. r., sloan, k., and amobi, f. the journal of scholarship of teaching and learning, vol. 7, no. 1, may 2007. 94 autobiographical-personal beliefs might be consonant or dissonant with the beliefs of others (classmates, mentor teachers, professor). because of the importance of philosophical beliefs to the decisions teachers make about their teaching, additional research is needed to determine how students’ previously held or current beliefs are enhanced, disengaged, or impeded by important others (their mentor teacher, other students, teachers they know), particularly as they progress in their preparation program. interaction among faculty outside the domain area of teacher education would enhance this discussion. for example, hager, pryor and bryant (2004), compared approaches to designing a field experience (called an internship) across four domain areas: political science, health science, teacher education and construction science education. although hager described the programmatic organization of internships in several domains, there was no discussion about the goals students might hold for themselves, their beliefs about how to best implement theories of practice learned in the university program, or their understandings of the implications of use of particular practices. for example, health sciences faculty might want to evaluate the course activities they use to enhance student belief in holistic diet, or exercise programs when these students are engaged in a parallel experience in a field-based internship. will their students hold beliefs similar to their professors? are students’ beliefs coherent with their field mentor? as faculties in teacher education (and other applied programs) review the scope and sequence of courses provided in their preparation programs, alternative program design formats such as seminars and blocked sections within courses (appleton, 1979; pryor, 2003a) should be evaluated so that implications might be shared among university domains. to advance this discussion, we propose two resources useful for continuing discussions about our course practices: (a) a newly developed url containing the online version of the poes, (http://texascbt.tamu.edu/survey/philosophy_of_education/scale.htm), which includes immediate participant feedback on scoring and spss data analysis, and (b) use of interactive media formats to augment development perspectives. these suggestions might lead to similar scale development and investigations helpful in other domain areas. importantly, scales and other reflective methods might reveal that students’ non-coherence portrays an unintended outcome— student independence from the normative saliency of promoted philosophical orientation. references amobi, funmi (2003). “reflections on the transformation in a teacher educator’s teaching of educational philosophy.” teacher education quarterly, 30(3) 23-32. antonovsky, aaron. (1987). unraveling the mystery of health. san francisco: jossey-bass. appleton, nicholas.a. (1979). “a modular approach to foundations.” journal of teacher education, 25(3) 249-253. arnstine, donald g. (2002). “why should philosophers and educators speak to each other ? there are more serious problems to face and more important jobs to be done.” educational theory, 52(3) 303-313. pryor, c. r., sloan, k., and amobi, f. the journal of scholarship of teaching and learning, vol. 7, no. 1, may 2007. 95 ausubel, david. (1963). the psychology of meaningful verbal learning. new york: green and straton. berliner, david. c. (1986). “in pursuit of the expert pedagogue.” educational researcher, 15(7) 5-13. bredekamp, sue, and copple, carol. (eds.). (1997). developmentally appropriate practice in early childhood programs (rev. ed.).washington, dc: national association for the education of young children. bredo, eric. (2002). “how can philosophy of education be both viable and good?” educational theory, 52(3) 263-271. burbules, nicholas c.. (2002). “the dilemma of philosophy of education: relevance or critique.” educational theory, 52(3) 257-261. butin, dan w. (2004). “the foundations of preparing teachers: are education schools really ‘intellectually barren’ and ideological?” teachers college record online. retreived july 29, 2004 from: http//:www.tcrecord.org carbone, peter. e. jr. (1991). “the teacher as philosopher.” educational forum, 55(4) 319-331. coffey, amanda, and atkinson, paul. (1996). making sense of qualitative data. thousand oaks, ca: sage. connelly, michael f., and clandinin, jean d. (1988). teachers as curriculum planers: narratives of experience. new york: teachers college press. dewey, john (1933). how we think: a restatement of the relation of reflective thinking to the educative process. lexington, ma: d.c. heath. edel, abraham. (1972). “analytic philosophy of education.” educational theory, 22(1) 131152. ediger, marlow. (2003). “data driven decision-making.”college student journal, 37(1) 1-9. enz, billie j., freeman, donald. j., and wallin, mark b.(1996). “roles and responsibilities of the student teacher supervisor: matches and mismatches in perception.” in preparing tomorrow’s teachers: the field experience, (d.j. mcintyre & d. m. byrd, eds.), pp. 131-150. emerson, ralph waldo. (1841/1983). “self reliance.” in the essential writings of r.w. emerson. new york: penguin putnam. feinberg, walter f., and soltis, jonas f. (2004). school and society. new york: teachers college press. fen, sing-nan. (1967). “the professional and liberal education of teachers.” pryor, c. r., sloan, k., and amobi, f. the journal of scholarship of teaching and learning, vol. 7, no. 1, may 2007. 96 peabody journal of education, 45(3)158-16. gross, patrica. (1996). “grounding theory into practice.” teaching education, 8(1) 37-43. gunzenhauser, michael g. (2003). “high stakes testing and the default philosophy of education.” theory into practice, 4(1) 51-58. gutmann, amy. (1987). democratic education. princeton, nj: princeton university. guyton, edith., rainer, julie, and wright, thomas. (1997). “developing a constructivist teacher education program.” in david byrd & john mcintyre, (eds.), research on the education of our nation’s teachers, pp. 149-171. thousand oaks, ca: corwin press. hager, candice, pryor, caroline r., and bryant, john. (2004). “a comparison of four domain area standards for internship and implications for utilization in undergraduate construction education internship programs.” journal of construction education, 8(30) 157-179. haggerson, nelson l. (2002). “the mission of the mytho-poetic scholar”. in michael wolfe and pryor, caroline, (eds.), the mission of the scholar: research and practice, pp. 75-90. new york: peter lang. henry, christine m., and shea, carol a. (1986). “who’s teaching the social foundations courses?” journal of teacher education, 37(2) 10-15. hlebowitsh, peter s. (2005). designing the school curriculum. boston: allyn & bacon. huck, schuyler w. (2000). reading statistics and research. new york: longman. interstate new teacher assessment and support consortium. (1992). model standards for teacher licensing, assessment, and development: a resource for state dialogue. washington, d.c.: author. kalimo, r., and vuori, j. (1990). “work and sense of coherence: resources for competence and life satisfaction.” behaviorial medicine,16(1) 76-89. kincheloe, joe. (1999). “critical democracy and education.” in james g. henderson and kathleen r. kesson (eds.), understanding democratic curriculum leadership, pp.70-83. new york: teachers college press. kohn, alfie (1986). no contest: the case against competition. boston: houghton mifflin. korotokov, david. (1998). “the sense of coherence: making sense out of chaos.” in paul t. p. wong and prem. s. fry (eds.), the human quest for meaning: a handbook of psychological research and clinical applications, pp. 51-70. mahwah, nj: erlbaum. pryor, c. r., sloan, k., and amobi, f. the journal of scholarship of teaching and learning, vol. 7, no. 1, may 2007. 97 korthagen, fred. a.,and kessels, jos. p. (1999). “linking theory and practice: changing the pedagogy of teacher education.”educational researcher, 28(4) 4-17. leathy, robert, and corcoran, carol a. (1996). “encouraging reflective practitioners: connecting classroom to fieldwork.”journal of research and development in education, 29(2) 104-114. lustig, daniel c., rosenthal, david a., strauser, david r., and haynes, kelly. (2000). “the relationship of sense of coherence to life satisfaction for students with disabilities.” rehabilitation counseling bulletin, 43(1) 134-141. morey, ann. (2001). “the growth of for-profit higher education: implications for teacher education.” journal of teacher education. 52(4) 300-311. motzer, sandra adams, and stewart, barbara j. (1996). “sense of coherence as a predictor of quality of life in persons with coronary heart disease surviving cardiac arrest.” research in nursing and health, 19(1) 287-298. noddings, nel. (1995). philosophy of education. boulder, co: westview press. nunnally, jum. c. (1967). psychometric theory. new york: mcgraw-hill. oliva, peter. f. (2001). developing the curriculum. new york: longman. petress, ken c. (2003). “an educational philosophy guides the pedagogical process.” college student journal, 37(1) 128-135. pinar, william. f., reynolds, william. m., slattery, patrick, and taubman, paul m. (2000). understanding curriculum. new york: peter lang. pinar, william f. and grumet, madeline (1976). towards a poor curriculum. dubuque, ia: kendall-hunt. pinar, william f. (1974). “currere: toward reconceptualization.” in john jelinkek (ed.), basic problems in modern education, pp.147-171. tempe, az: arizona state university, college of education. preskill, stephen l.. (1979). “in search of a usable philosophy of education: the vision of glenn gray.”journal of educational thought, 31(3) 205-222. pryor, caroline r. (2003a, february). a comparison of preservice and inservice teachers’ philosophy of education: tests of a newly revised philosophy of education scale. paper presented at the meeting of the association of teacher educators. jacksonville, florida. pryor, caroline r. (2003b). “teaching for democratic practice: three strategies for the social studies methods course.” teacher education and practice, 10(3) 171-185. pryor, c. r., sloan, k., and amobi, f. the journal of scholarship of teaching and learning, vol. 7, no. 1, may 2007. 98 pryor, caroline r. (2004a). “creating a democratic classroom: three themes for citizen teacher reflection.” kappa delta pi the record, 40(2) 78-82. pryor, caroline r. (2004b). writing a philosophy statement: an educator’s workbook. boston: mcgraw hill. pryor, caroline r., and kuhn, jodi. (2004). “do you see what i see? bringing field experience observations into the methods course.”the teacher educator, 39(4) 17-29. pryor, caroline r., and eskirmireh, zoreh.(2004). “iranian and u.s. preservice teachers’ philosophical approaches to teaching: enhancing intercultural understandings.” current issues in comparative education, 7(1). retrieved september 16, 2005 from: www.tc.columbia.edu/ice rainer, julie, and guyton, edith. (1999). “democratic practices in teacher education and the elementary classroom.” teaching and teacher education, 15(1) 121-132. ravitch diane and thernstrom, abigail (1992). the democracy reader.new york: harperperennial. schonwetter, dieter j., sokal, laura, friesen, marcia, and taylor, k. lynn. (2002). “teaching philosophies reconsidered: a conceptual model for the development and evaluation of teaching philosophy statements.” the international journal for academic development, 7(1) 83-98. shea, christine., sola, peter, and jones, alan. (1987). “examining the crisis in the social foundations.” educational foundations, 2(1), 47-57. smith, l. glenn. (1984). lives in education. ames, ia: educational studies press. soderberg, siv, lundman, berit, and norberg, astrid. (1997). “living with fibromyalgia: sense of coherence, perception of well-being, and stress in daily life.” research in nursing and health, 20(1), 495-503. soltis, jonas f. (1986). “perspectives on philosophy of education.” journal of thought,18(2) 14-21. steiner, david. (2004). “preparing tomorrow’s teachers: an analysis of syllabi from a sample of america’s schools of education.” in frederick hess, andrew rotherham, and kate walsch, (eds.), a qualified teacher in every classroom, pp. 119-148. cambridge, ma: harvard education press. szymanski, e. m., hershensen, david. b., enright, m. s., and ettinger, j. m. (1996). “career development theories, constructs and research: implications for people with disabilities.” in edna mora szymanski and randall m. parker (eds.), work and disability: issues and strategies in career development, pp. 79-126. austin, tx: pro-ed. tanner, daniel, and tanner, laurel. (1995). curriculum development. nj: prentice-hall. pryor, c. r., sloan, k., and amobi, f. the journal of scholarship of teaching and learning, vol. 7, no. 1, may 2007. 99 the council of learned societies in education (clse) (1996). standards for academic and professional instruction in foundations of education, educational studies, and educational policy studies (2nd ed.). san francisco: author. the national council of accreditation of teacher education. (2000). ncate unit standards. washington, d.c.: author. towers, james m. (1991). “how and why social foundations of education are taught at minnesota’s private four-year colleges?” journal of social studies research, 15(1) 30-35. veenman, simon. (1984). “perceived problems of beginning teachers.” review of educational research, 54(2), 143-178. wactler, carolyn r. (1990). how student teachers make sense of teaching: the derivations of an individual’s educational philosophy. arizona state university, tempe: unpublished doctoral dissertation. wilkins-canter, elizabeth a. (1996). “providing effective cooperating teacher feedback.” in d. john. mcintyre & david m. byrd, (eds), preparing tomorrow’s teachers: the field experience, pp. 169-177. thousand oaks, ca: corwin press. witcher, ann, and travers, paul. (1999). the witcher-travers survey of educational beliefs. retrieved september 16, 2005 from: http://www.abacon.com/witcher-travers. zeichner, kenneth m. and tabachnick, b. robert (1981). are the effects of a university education “washed out” by school experience? journal of teacher education 32(3), 2-6. pryor, c. r., sloan, k., and amobi, f. the journal of scholarship of teaching and learning, vol. 7, no. 1, may 2007. 100 appendix a. definitions. the executive approach. efficiency is the focus of this approach, in which example concerns might include: time on task, test attainment, and rules that direct classroom management (feinberg & soltis, 2004). it is not unusual for executive teachers to consider student behavior and an organized classroom environment primary to effective teaching. berliner (1986) wrote that these teachers approach their teaching as managers of a business, a notion grounded in the functionalist concepts underlying a factory model of education (feinberg & soltis, 2004). the humanist approach. the humanist approach is primarily concerned with providing an environment in which the interests and abilities of each student can be fully developed (tanner & tanner, 1995). this approach suggests formulating pedagogical decisions based on recognizing various levels of student ability, and has historical support from progressivist theorists such as rousseau, froebel, dewey and others (smith, 1984). the subject specialist approach. the subject specialist approach is focused on students becoming knowledgeable within a particular domain area (e.g., mathematics) and use of pedagogical practices based on the nature of the subject. here, a teacher’s focus is the breadth and depth of the subject area with student interests and abilities less central. historically, a hierarchy has existed among the subject areas considered most important to teach (smith, 1984). the citizen teacher approach. the goal of the citizen teacher is to prepare students as active and informed participants in a democracy. three principles of democracy define this approach: liberty-freedom, justice-fairness, and equality-equal opportunity (pryor, 2003b; gutmann, 1987). using this approach, a teacher provides a foundation for analysis of the social, historical and economic roles of citizenry using activities such as discourse and communication. the explorer approach. the explorer approach emphasizes discovery of vast amounts of information, however information is considered distinctly different from the term knowledge used in the subject specialist approach. in the subject specialist approach, knowledge is valued for its collective potential; the explorer approach seeks instead to understand the rapid change in information. an explorer teacher helps students investigate the changing world through global interaction, often using multi-media as primary tool. pryor, c. r., sloan, k., and amobi, f. the journal of scholarship of teaching and learning, vol. 7, no. 1, may 2007. 101 appendix b. sample poes scoring across one dimension of teaching. the philosophy of education scale is available online at http://texascbt.tamu.edu/survey/philosophy_of_education/scale.htm or hard copy (pryor, 2004a). this form has seven rows (e.g., “classroom environment”) of large boxes. each large box has a small box and three descriptors of teaching beliefs and practice. first, start with the descriptors. rate each of the three indicators in each large box in the first row, going from left to right, using the scale below as a guide. rating numbers can be repeated. most like me 5 4 3 2 1 least like me second, rank each of the five large boxes in across each row from the one most like you (5), to the one least like you (1) using the scale above. use each ranking number only once; place this number in the small box. repeat this process for the remaining rows. third, add the small boxes (down), for each column total. sample poes dimension: lesson plans rate indicators rate indicators rate indicators rate indicators rate indicators rank approach rank approach rank approach rank approach rank approach lesson plans 4 specific objectives and standards clearly defined 4 essential elements of instruction are addressed 5 meets district guidelines, scope and sequence 1long-term, broadly structured outcome 1 thematic and integrated curriculum 1 student-centered learning 3emphasis on depth of knowledge 3 instruction extends beyond standardized testing 2 extensive resources (field trips, guest speakers) 1open-ended objectives 2 inquiry 3emphasize technological skills and information interpreting techniques 2 flexible goals based on community and citizenship needs 3 practical knowledge and life skills 5 higher-order, critical thinking and problem -solving note: to determine overall philosophical approach, sum total only the ranked small boxes down the column. 5 1 3 2 4 microsoft word v9n1meyer-8030.doc journal of the scholarship of teaching and learning, vol. 9, no. 1, january 2009, pp. 118 – 133. my life in a bag and other stories: on the road to resiliency kristine m. meyer 1 , barbara l. licklider 2 and janice a. wiersema 3 abstract: post-secondary education students will experience transitions, successes, challenges, and adversities during their college years. although they will continue their growth and development in many aspects of their lives during this time, perhaps a most critical, but often ignored, attribute for them to develop is resiliency, the ability to bounce back from difficulties. this article examines the findings from a phenomenological study of eleven students who participated in resiliency development education (rde) as first semester post secondary students. the data confirmed that, when taught through the pedagogy of storytelling, students are able to use stories of their experiences and those of others to shape not only their perceptions and choices, but also behaviors and characteristics reflected in resilient individuals. keywords: storytelling, metaphor, pedagogy, resiliency, protective factors “with weeping and with laughter, still is the story told, how well horatius kept the bridge in the brave days of old. lord macaulay, lays of ancient rome, horatius lxx the heart of the human experience is often captured in story. the way in which students use the stories of their life experiences to understand themselves and their personal ways of addressing with difficult issues can often affect how they respond to challenges and adversity. each fall college campuses experience the influx of students ready to embrace college life. however, many of these students are leaving their parental homes for the first time, often unequipped mentally and emotionally to independently address difficulties of student life, expectations of academic rigor, and the insecurities of being away from their parents, thus, negatively influencing their academic success. students who display resilient behaviors possess the ability to “bounce back” from challenges or adversity and are able to cope with the stressors inevitable to college students (benard 1993, p. 44). therefore, exposing post-secondary students to resiliency development education (rde) may be an effective way in which to positively affect students’ experiences. research has provided sound evidence that resiliency can be taught (werner & smith, 1982; benard, 1993, 2004; masten, 2001; wolin & wolin, 1993). characteristics that are 1 department of student services, north iowa area community college, 500 college drive, mason city, ia 50401, meyerkri@nicacc.edu, 641-422-4419. 2 department of educational leadership and policy studies, iowa state university, n247c lagomarcino hall, ames, ia 50011, blicklid@iastate.edu, 515-294-1276. 3 information assurance center in the department of electrical and computer engineering, iowa state university, n002 lagomarcino, ames, ia 50011, janw@iastate.edu, 515-294-4030. meyer, k., licklider, b., and wiersema, j. journal of the scholarship of teaching and learning, vol. 9, no. 1, january 2009. 119 significant of resiliency have often been labeled as “protective factors” (benard, 1993). in her study, benard (1993, 2004) identified four personal protective factors that could be found in varying degrees in people who seem resilient in the face of major life challenges. these protective factors are: social competence, problem solving skills, autonomy, and a sense of purpose. the protective factors found in resilient people emerge in a variety of ways. for example, those with social competence often possess the qualities that exude responsiveness, flexibility, empathy, caring, communication skills, and a sense of humor. socially competent people are able to develop relationships with family and friends in a variety of settings. people displaying skills in problem solving seem more able to think abstractly and reflectively while identifying possible solutions to problems – both cognitively and socially. planning, creativity, and resourcefulness come easily to a problem-solver. autonomous people have a strong sense of identity. they are independent in thought and action; they enjoy a sense of control over their environment, and are often able to separate themselves from dysfunctional family circumstances. finally, those with a sense of purpose have goals, aspirations, hopefulness, perseverance, and a sense of a bright future. resiliency education involves exploring the interpersonal as well as the contextual. teaching the concepts of resiliency requires a method or pedagogy that can span the affect in addition to the cognitive. a tool with the capacity to do both can be found in one of the oldest forms of communicating ideas and images, the art of telling stories (mello, 2001). the teller becomes a “bridge builder, a person who broadens the discourse by describing images and messages” (p. 8) bringing meaning and understanding to the subject matter. using storytelling as pedagogy enables students to examine their own self-story, find meaning in relating concepts to metaphors and folklore, and experience new understandings in mental images before applying them to life situations. supporting teaching strategies with stories enhances the process of educating students in the area of resiliency. according to mello (2001), students begin examining their own biases and conceptions when presented with stories of diverse cultural texts (p. 8). mello also posits that students create transactional experiences that allow growth in interpersonal and intrapersonal knowledge. egan (1997) determined that students derived meaning by actively engaging in the content of stories using both their emotional intelligence and cognitive ability. as pedagogy in rde, storytelling has the potential to enrich the acquisition of resilient behaviors and characteristics in students. the format and presentation, which make stories unique and innovative as strategies, are vital pedagogical tools for teaching and learning (egan, 1997). the question for this study, then, was: how do first semester freshmen involved in rde make meaning of resiliency in their lives through the pedagogy of storytelling? i. background of the study. it was during a meeting of the academic standards committee, at a midsized midwestern research university, that discussion began focusing on concerns raised when reviewing written requests by students to be reinstated after a period of dismissal from the university for lack of satisfactory academic progress. as committee members discussed the requests, they began observing a repeated pattern among students who had been dismissed. more often than not, they found that students’ responses to adverse situations in their lives were devastation and an inability to grasp perspective. the issues seemed to raise barriers and would result in poor class meyer, k., licklider, b., and wiersema, j. journal of the scholarship of teaching and learning, vol. 9, no. 1, january 2009. 120 attendance, missed assignments, poor workmanship, and often dropping out of the class. furthermore, the evidence suggested that the students had not developed any productive strategies to address life challenges. the insight into this perplexing trend prompted one of the committee members to suggest resiliency education for beginning post secondary students. as stated earlier, research is adamant that resiliency can be taught (werner and smith, 1982; benard, 1993, 2004; masten, 2001; wolin and wolin, 1993). therefore, a learning component about resiliency development for post-secondary students would contain opportunities to understand, internalize, and “try on” resiliency. the ultimate goal of rde would be to allow students to internalize the nature of resilient behaviors and characteristics as well as increase the choices of response in the face of difficult life challenges. understanding the student experiences and perceptions after exposure to resiliency development education could lead to tactical undertakings in resiliency education at the college level that have the potential to greatly impact retention. therefore, this rde project explored how to encourage students to rise above those adverse situations and work through them in a healthy, productive manner. in essence, we were curious as to how we could help students to grow into resilient young adults through intentional instruction. the project then, would encompass a curriculum that would be actualized in a classroom setting. part of the curriculum was implemented in a freshman class through a leadership and learning academy at a midwestern university. the academy offers two semester-long courses. the first course, which included the sessions on resiliency, is designed with an emphasis on: (a) learning about learning, (b) learning about self, (c) purposefully developing community, (d) deliberately practicing and refining skills to support and encourage the growth of self and others, (e) practicing metacognition, and (f) engaging in intentional mental processing. in addition to affording plenty of individual talk time, weekly two and one half-hour meetings provide opportunities for students to participate in frequent team learning. consistent with the goal of helping students manage and control their own growth and development while supporting the learning of their colleagues, the team learning opportunities centered around the science of learning and the deliberate development of community. four sessions of rde were taught to forty-seven first semester freshmen in the academy for leadership and learning, facilitated by a lead professor and four supporting faculty. the curriculum was designed as a tool to engage students in the awareness and development of resiliency. each session involved a variety of activities and learning components focused on internalizing an understanding of resiliency as well as an opportunity for a self-discovery of protective factors. there were continual checks for understanding and opportunities to reflect and share about the learning experience. while conducting the sessions, the co-facilitators and participating faculty began to notice significant changes in many of the students. it was as if they were practicing new behaviors, deeply reflecting upon what they had learned about in the sessions on resiliency, connecting past and present situations, and applying their new understandings in their lives with new awareness. for example, some of the students began sharing moments they handled differently using various tools learned in class, such as reframing and self-talk. they became more confident in expressing the protective factors they had identified in themselves and how they could use them to address challenges or adversity. as the co-facilitators and faculty listened to and observed the students in class, it became evident that there were a number of students who really had made their own meaning of resiliency and were beginning to make important connections to their lives as college students. we believed it was important to find out from the meyer, k., licklider, b., and wiersema, j. journal of the scholarship of teaching and learning, vol. 9, no. 1, january 2009. 121 students what meaning they made of the phenomenon of becoming more resilient. an email was sent to 20 of these so identified students, describing the study and requesting their participation. instead of selecting a pre-determined number of participants, the 11 students who responded with a desire to be a part of the study were chosen. making meaning of resiliency in their lives is ultimately up to the students. our goal with this study was to identify reasons for changes in students’ behaviors and to reveal important factors affecting the development of individual resiliency. ii. methodology. in light of the various factors affecting transition and adjustment of the freshman student to the college experience, it was prudent to explore how these young adults made meaning of resiliency, the ability to deal effectively with challenges, in their lives. this chapter describes the methodological framework used to conduct this phenomenological study. we begin with the theoretical framework. a. epistemology. exploring epistemology gives researchers an opportunity to probe the philosophical underpinnings that explain how we know what we know. according to jones, torres, and arminio (2006), epistemology refers to the assumptions one makes about the process of gathering knowledge. constructionism, which informs this study, is an epistemological lens which views knowledge and “all meaningful reality as such, as contingent on human practices, being constructed in and out of interaction between human beings and their world, and developed and transmitted within an essentially social context” (crotty, 2003, p. 42). meaning is constructed in relationship to something; it is not discovered, but constructed (crotty, 2003). b. theoretical perspective. the theoretical perspective that supports the methodology for this study can be found in the interpretivist philosophy. the ontology of the constructionist-interpretivist is that there are multiple valid and socially constructed realities (ponterotto & grieger, 2007). therefore, the multiple realities of the lived experience can be interpreted for meaning through the lens of perception. we can only construct meaning to what we perceive in the present, our current reality, with the knowledge we have embodied and gathered in the past. the curriculum for the resiliency development component was designed with the intent to introduce the protective factors of resiliency to the students. through the teaching process, the intent was to help students identify and enhance their existing protective factors as well as learn how to bring new strengths into being. for this to become a reality, it was necessary for the students to take the new knowledge and combine it with what they knew of themselves as they began the process of constructing new meaning in terms of resiliency. c. methodology. the methodological approach of phenomenology was appropriate for this study as we discerned the “essence” of the experience as described by the students (creswell, 2003, p. 15). phenomenology seeks to achieve a deeper understanding of the meaning of our theoretical meyer, k., licklider, b., and wiersema, j. journal of the scholarship of teaching and learning, vol. 9, no. 1, january 2009. 122 activities not only in describing the essences, but also through grasping concepts rooted in the ordinary lived experience. the phenomenologist, merleau-ponty, believed that the phenomenological philosophy is essentially the description of the “perception” of the perceived world (matthews, 2002, p. 46). merleau-ponty reiterates in primacy of perception (1964), “the perceived world is the always presupposed foundation of all rationality, all value and all existence” (p. 13). merleau-ponty believed that describing the perception was primary for phenomenology. by looking at our ordinary engagement with the world from a bit of distance, we gain clearer insight and understanding, just as we might by holding a book a little way from our eyes in order to read better (matthews, 2002, p. 35). as the students described their experience with rde, the very act of putting their perceptions into language simulated the “stepping back” which elucidated, for them, the essence of the experience. the heart of phenomenology, which informed this study, is the lived experience (merriam, 2002). it brings into relationship the conscious subject and the object (crotty, 2003), in this case it was the student and the notion of resiliency. as researchers, it is important that our focus is not on the humans nor on the human world, but rather on “the essence of the meaning of the interaction” (merriam, 2002, p. 93). merleau-ponty, conceived phenomenological philosophy as “re-learning to look at the world” (matthews, 2002, p. 46). the goal of this study was to understand the phenomenon of the students’ experiences of “re-learning to look at the world” after the encounter with rde. phenomenology was appropriate for this study as we discerned the “essence” of the experiences as described by the students (creswell, 2003, p. 15). d. methods. epoche process. the phenomenological approach to the process known as epoche is that of bracketing, or setting aside, preconceived notions, ideas, or theories concerning the present study in order to eliminate as much bias as possible (moustakas, 1995). this enables the researcher to come to the data with an open mind. at the onset of the study, it was necessary to highlight the biases that were evident in our previous experiences studying resiliency. it was, therefore, essential that we bracket our viewpoints in order to prevent the assimilation of our thoughts into that of the participants. the isolation of our previous beliefs, assumptions, and biases was pertinent to the study: • it is possible for one to learn to be resilient. • there are certain characteristics that can be found in resilient people such as intrapersonal skills, optimism, social competence, the skill of problem solving, and the ability to set goals and look forward to something in the future. • people who are not resilient are not always suicidal or otherwise maladjusted. • resiliency is essential for fulfillment in life. • resiliency is seen in response to a plethora of challenges – large and small. • resiliency is an attitude that is reflected in behaviors, feelings, and beliefs. • resiliency brings responsibility to the forefront: that of choice, action, and thought. • we can affect another’s resiliency by providing a caring environment, and having healthy expectations and opportunities to be a part of and contribute to an organization or relationship. • knowing one’s strengths contributes to one’s resiliency. meyer, k., licklider, b., and wiersema, j. journal of the scholarship of teaching and learning, vol. 9, no. 1, january 2009. 123 throughout the process of data collection, we reviewed this list in order to maintain our focus on the lived experiences of the students. participants. four sessions of resiliency education were taught in a leadership and learning academy in the midwestern university. the class was comprised of 47 freshmen students from a variety of majors: animal ecology, business, diet and exercise, elementary education, exercise science, horticulture, physical therapy, and veterinary science. qualitative research, steeped in a search for meaning necessitates selecting participants who can bring rich and meaningful data to the table. since the “idea behind qualitative research is to purposefully select participants…that will best help the researcher understand the problem and the research question” (creswell, 2003, p. 185), the participants for this study were purposefully selected. the participants selected were those students who engaged in the observed phenomenon of making meaning of resiliency in their lives after participating in rde. eleven of the 20 identified students agreed to be a part of this study. of the 11 participants in the study there were nine females and two males; all were freshmen students between 18 and 19 years of age. four of the participants were first generation college students and all were caucasian, 10 heralding from the state of iowa and one from wisconsin. there were a variety of academic undergraduate majors: three in animal ecology; two in horticulture; two in elementary education; one in exercise science and physical therapy; one in diet and exercise; one in business; and one in the pre-veterinary program. data collection. in order to gather rich and meaningful data, important to a phenomenological study, one of the principal methods used by researchers is the interview (merriam, 2002), the “heart of social research” according to esterberg (2002, p. 83). to realize the most effective interviews, the data were collected from focus groups and individual interviews that were in-depth, semi-structured, and guided with non-scripted, open-ended questions evoking authentic insight and perspective from the participants (esterberg, 2002). data collection began with two focus group interviews; one group consisted of seven participants and the other had four. two different sessions were held to accommodate the schedules of the participants. each 45-minute session was audio taped and transcribed verbatim. face-to-face and telephone individual interviews were then conducted with each of the 11 participants. in addition, data were collected from the students’ journals, other assignments, and end-of-semester written summaries of their learning. data analysis and interpretation. at the start of the analysis, using the steps described by colaizzi (1978), all transcripts and journal entries were carefully read in order to understand the essence of the students’ written and oral reflections as well as to be mindful of reoccurring topics. after a coding procedure was developed, the transcripts and journal entries were read and reread using a color-coded system to highlight significant statements, and to sort and identify potential meanings of the data. themes were then constructed and integrated to produce an exhaustive description (colaizzi, 1978) of the students’ experiences identifying and articulating the phenomenon’s fundamental structure. finally, the data were checked for validity by returning them to the participants for confirmation of thematic interpretation. iii. findings. the themes ubiquitous within this study were: (1) the efficacy of learning resiliency through the pedagogy of storytelling; (2) the value of learning in community; (3) and the transformative resiliency development of the post-secondary student. the fundamental structure meyer, k., licklider, b., and wiersema, j. journal of the scholarship of teaching and learning, vol. 9, no. 1, january 2009. 124 of becoming more resilient, then, as perceived by the participants, was a self-recognized transformative development resulting from making personal meaning through stories and experiences within a community of learners, and then intentionally applying the learning to one’s own life. this complex statement is potent with possible options to explore for students and educators alike. however, this article addresses the meaning post-secondary students derived from the exposure to storytelling as pedagogy in resiliency development education. a. storytelling as pedagogy. innovation in teaching methods is welcomed by many institutions in a day when students are increasingly conversant with highly creative methods of dispersing information via internet access (braxton, 2004; gerdes and mallinckrodt, 1994). during this study, storytelling, a technique that is as old as the world itself, was used as an innovative method of enhancing student learning. storytelling is one of the most basic ways of sharing what we know, making sense of our experiences, and gaining insight into ourselves and our relationships with others in our world (adams, allendoerfer, smith, socha, williams, and yasuhara, 2005, 2007; mello, 2001; mcadams, 1993; connelly and clandinin, 1988). the rde curriculum incorporated storytelling as part of the pedagogy in a variety of ways throughout the sessions. in order for the students to gain a deeper understanding of themselves in relationship to resiliency, it was necessary for them to explore their own stories. the activities that enhanced this understanding included: 1. “know thyself” – students made a conglomerate of words describing their attributes of character and strengths. 2. “life in a bag” – students put five things in a bag that represented something important to them or about them. 3. “self-story” – using the “know thyself,” “life in a bag,” and a story-starter (a list of memory joggers), students were to write a self-story describing five major experiences in their lives which affected who they had become. 4. insight learning personality instrument to identify their personality spectrum. 5. “my four protective factor analysis” – activity to allow opportunity to identify and analyze the presence of their personal protective factors. 6. “reframing and self-talk practice schedule” – allowed the students to monitor how they reframed situations for a better solution or used self-talk to adjust their attitude toward an issue or challenge. stories based on folklore, personal illustrations, and reference to the self-story, were used throughout these sessions to enhance the curriculum by making connections and analogies to correspond with the concepts being taught. this study examined the experiences the students had with the curriculum for development of resiliency within the pedagogical framework of storytelling. as pedagogy, using a story invites inquiry. it is fairly easy to use story within teaching lessons, information, or mental processing (collins and cooper, 1997). according to clandinin and connelly (1994), the story often stimulates questions allowing for increased clarity and insight. stories are a vehicle to transfer meaning and understanding in a safe and unthreatening environment. research conducted by livo and rietz (1986) indicated that material presented through the means of story rather than lecture held the interest of the students and made a far greater impact. by using stories in this study, we were able to present an abstract concept, such meyer, k., licklider, b., and wiersema, j. journal of the scholarship of teaching and learning, vol. 9, no. 1, january 2009. 125 as resiliency, in a concrete form. sharing stories at strategic points in the curriculum enabled us to enhance and deepen the understanding of resiliency. the data attested to the importance of using stories to augment comprehension of what resiliency is and how it can be applied and lived out in our everyday lives. this finding was initially revealed as the students experienced meaning through the writing of their self-stories. b. the self-story. using narrative to support pedagogy is a main claim in educational research for the simple reason that, “humans are storytelling organisms who, individually and socially, lead storied lives” (connelly and clandinin, 1990, p. 2). the introductory sessions for this study focused on building an awareness of self. in order to facilitate the exploration of who they are, we first wanted students to understand the stories of their lives that brought them to this point. given paper bags, each student was to return to class with “my life in a bag.” the students were given the opportunity to think about their past experiences and then to represent them using five objects or representations which they put in a paper bag (livo and rietz, 1987; pellowski, 1987). the stories around those artifacts were to tell something about themselves. the meaning that the students discovered through this experience highlighted the sessions and often propelled them to see who they were in a new light. the self-story informs as well as forms our lives (widdershoven, 1993). in this study, the self-story seemed to evoke a deeper understanding for students of not only who they are today, but also who they had been before as they reflected on the persons of their younger years. andrea found the “my life in a bag” activity to be a defining moment. she described its impact: [the activity “my life in a bag”] is so cool because it gives people the chance to really look in the mirror and see what is important to them and to some extent can help people reprioritize. allison enthusiastically relayed her experience: my favorite assignment was “my life in a bag.” i realized that there are tons of things i really enjoy, but they don’t necessarily describe me or say who i am. this assignment really made me think hard about who i am and what makes me who i am. when teaching concepts that encourage reflection, past experiences, positive and negative often resurface. for many in the class, revisiting unpleasant memories was not always easy, but often worthwhile as mary revealed in her journal: most of my stories that i wrote and didn’t write had the feeling of anger and sadness tied to them. i was put in the middle of a lot of my parents’ fights and was let down a lot. i know what it feels like and know that i will not put my own children through the same thing. situations that have gone on between my parents will always stay with me. i will never ever forget them. they have changed who i am today. a sense of appreciation for the learning or growth often replaced the resentment and tamed the anger for having to endure emotional pain. mary continued to put a positive frame around her past: meyer, k., licklider, b., and wiersema, j. journal of the scholarship of teaching and learning, vol. 9, no. 1, january 2009. 126 in a way, i’m kind of glad they occurred because i feel they made me more mature and able to handle a lot more for my age. a person who is resilient has buoyancy and adaptation in their lifestyle. i feel that i have some of these characteristics in my life. resiliency has truly taken on a new meaning for me. moving away from a painful experience involves making a reactionary choice. resiliency is often exhibited in persons who know how their emotions affect their thinking and their reactions to events in their lives (wolin and wolin, 1993). laura discovered that understanding the concepts of resiliency could shape her thinking in several ways: it was a rough semester for me, so learning about it [rde] really helped me get through everything. there was a lot, a lot of emotional that i was dealing with, and stuff, and i mean, i’m still dealing with it, but it’s [knowledge of resiliency] helped in everything. post-secondary students interact more independently with personal and academic issues than when they were in high school and living in their parental homes. using stories as a teaching strategy, the ability to understand the manner in which the brain links emotion and memory gives insight and appreciation for memories that are difficult or painful in our lives (goleman, 1995). resiliency occurs when one is able to view those experiences as fertile ground for personal growth. in her journal, andrea described how she had come to understand her strength or resiliency by looking at the hardships faced in her past: in almost every case i can trace my “strength” or “resiliency” to my past experiences. it’s unfortunate, but true, that the majority of my past experiences that have helped to develop my resiliency were bad or sad experiences. as much as my experiences stunk, it’s cool to look back on them and see that they’ve helped me grow into a better person…they’ve repeatedly taught me and shown me that life continues even during hard times, and after these hard times, it’s so important to gather yourself and essentially bounce back despite the hardship. c. illustrating through folklore. throughout the remaining sessions, storytelling was used as a way to define, identify characteristics, explain the components, and introduce tools of resiliency. a menu of stories, used to illustrate the concepts of resiliency, was presented in the form of metaphors, folklore, or personal experiences. using the oral tradition, without using visual aids, but only the listener’s imagination, the participant could visualize an actual mental scene in which behaviors could be rehearsed or “tried on.” teaching reframing and self-talk, two essential tools of resilient reactions to challenges, can effectively be tendered through stories. one story particularly meaningful to the students and an example of the power of reframing and self-talk was the spyglass written by richard paul evans (evans, 2000). the repeating phrase found in the story, ‘you have seen what might be, now go and make it so,’ seemed to resonate with the students long after the story ended, as adam wrote: today i was thinking about the “spyglass” story that was told a couple of weeks ago. it was a really inspirational story. if you look at something from a different perspective, it becomes 10 times better than it was. the quote that was repeated, “you meyer, k., licklider, b., and wiersema, j. journal of the scholarship of teaching and learning, vol. 9, no. 1, january 2009. 127 have seen what may be, now go and make it so” made me rethink different things in my life. that story meant a lot to me in how it influenced me personally because it has happened to me. i will do my best to go and make it so. thinking deepens the awareness and begins the process of change. carla found this to be true: i think the stories were crucial because it got me thinking. when i’m listening to a story i put myself in that story and it causes me to care, and it gives you a picture to associate with things. what comes to mind...for me is the bridge story, and how the two brothers were not willing to make amends. like my best friend and me…there are times where i need to be a better brother than in that story…[that] is the very clear message in those stories. as the students made meaning of resiliency through the stories shared in class, it causes one to wonder what impact storytelling could have in other disciplines such as math and science. even in hard science disciplines, such as engineering, when pedagogy of storytelling is utilized to disseminate information, feelings and attitudes can be affected to promote thinking and innovation by “shift(ing) people’s stories” (adams et al, 2005, 2007, p. 10). allison recalls a story and the impact it had on her: the stories were huge! they helped connect to real life situations. you know the toll booth one? [it taught the message], you know you just can’t let stuff like that get you down. you gotta keep moving forward. you gotta do what you want. i mean the stories had a real life connection versus, “ok, read your textbook, this is what it says.” it doesn’t make a connection, where all the stories, well, they could be real people, and they connect with other people, so… i can read my chemistry book 30 times, and i’m not gonna connect to it, but the stories are what makes the connection for me. the retention of information is a constant motivation to adjust and sharpen teaching strategies. since the nature of stories affect emotion, they garner potential to greatly affect retention of material taught in any academic discipline. students make ‘hooks,’ so to speak, because they put themselves in the content. reflecting upon the nature of the brain to retain emotional memories, when a burst of insight is gained after seeing a connection, the chances of remembering it for a long time increase dramatically (sprenger, 1999) which andrea attested to in her journal: the stories are just something that always sticks in your mind, kind of like an “aha” thing…like a role model almost, even though it’s fictional. it’s just like something that…affects people for a day, maybe for a week, maybe it affects people forever, but i think a story is something people can really hang on to. stories need to be chosen carefully, with intentional purpose when incorporating them into a curriculum. it was not happenstance that particular stories were used to teach resiliency. they were each chosen with purposeful intent, pertinent to the subject matter. students are keenly aware when course content is filler and when it is dynamically intentional according to mary: …it made us visualize the story and connect it to what we were learning. it wasn’t like, just thrown in there, ‘here’s a good story,’ or ‘story time,’ but i mean, it made me connect the things. i remember the story about the dog…and then the builder meyer, k., licklider, b., and wiersema, j. journal of the scholarship of teaching and learning, vol. 9, no. 1, january 2009. 128 of the bridge and it was when we were remembering, or trying to learn the autonomy and sense of purpose, problem solving, stuff like that, but it really made things connect, and i enjoyed the story a lot more because then i connect them, and i’d actually think about the words…and be like, ‘oh, this is the definitions of this.’ using stories as an instructional tool allows the students to not only check for their understanding, but also to measure their growth. by mere repetition from an instructor or through the mental rehearsal of the listener, the story can act as a gauge to show the students’ progress from one point in life to another. adam brought this meaning to light during the individual interview: there was one story that stuck out in particular. it was the one about the king and he had you look through the glass and then said, “you’ve seen what can be, so go and make it so…” being able to see, like, how much better i’ve become from this semester, being able to see how much better i could be, like i’ve seen how much i’ve grown this far, like how much further can i grow? i remember like, they were looking at some old garden that didn’t have anything and they looked through it and they could see green vegetables and tomatoes, and all that stuff, when i hit that it was like, it’s just like me, looking at myself in the mirror, it’s sort of the same thing. if i could be a lot better than this, i look at it and it’s like, oh yeah, i could be a lot better, and so then i’d go and do it. d. illustrating through metaphors. metaphors are figurative descriptions of a concept. when one is able to express a concept in the rhetoric of a metaphor, then true understanding has transpired. metaphors encourage one to think outside the proverbial box (adams et al, 2005, 2007). during the focus group, adam explained how it [the concept of resiliency] suddenly made sense to him: i had an “aha” moment when we first started talking about resiliency. i didn’t really, understand the meaning of resiliency, until the day when we brought out the rubber bands. it’s like, you can be stretched to your limit and be stretched there for a really long time, but you’ll come back to normal, like just the small little circle of your rubber band. but if you keep being stretched and come back, you keep stretching it out and there’s a possibility that it will break the next time that you pull it back out again, so you don’t wanna be pulled out to the max every time, about halfway maybe, and then back down. resiliency has often been defined as the ability to “bounce back,” successfully adapt to the effects of adversity, and develop a social competence even in the face of severe stress resulting from personal or environmental challenges or trauma (benard, 1991). karen summarized the basics of seeing resiliency at work in her life with this analogy: i learned what resiliency was. that was huge, just learning that it was an option for your life, to be like, ‘you don’t have to carry all that, you can learn to bounce back from stuff.’ storytelling as a pedagogical strategy promotes critical thinking and reflection (adams et al, 2005, 2007). in this way, self-stories, folklore and metaphors paved the way to introduce the concept of personal protective factors, reframing and self-talk, all tools pivotal to resiliency. meyer, k., licklider, b., and wiersema, j. journal of the scholarship of teaching and learning, vol. 9, no. 1, january 2009. 129 metaphorical thinking enabled the students to relate how strengths and protective factors impact their actions and ways of dealing with their life issues. the mind’s eye or in other words, the imagination allowed the students to envision (reframe) an alternative way of looking at a challenge or adverse situation by creating a new “story” to live by. in the same manner, using self-talk simply involved generating a dialogue that encouraged a new behavior or way of facing an issue. mary was affirmed that there could be some good found in her unhappy past by incorporating the skill of reframing to see past wrongs as ground for future growth: i guess, just looking at the situation and knowing again, that it could be a lot worse, cause in my life, like, i had a terrible childhood, so then that makes me look at my future and how i want to shape my future and what kind of job i wanna have, so i know my kids don’t have to go through the stress that i had to go through, cause of my parents and because of their mess-ups. the strategies found in teaching reframing as a tool of resilient reaction showed donna how to stop and review the situation, then adjust her course of action. reframing has taught me the importance of stopping and examining a situation. this exercise of reframing situations is making me apply this method of thinking. i tend to be negative or look at the negative in situations. reframing has been teaching me to slow down and think things through along with re-examining a situation. reframing this situation doesn’t make the problem go away but it helps me to think clearly and remain calm during this situation. metaphorically speaking, perhaps the students in essence learned to “re-story” their lives in order to successfully navigate the challenges faced in their everyday. using story as a pedagogical tool to present the various concepts embedded in rde, the students made meaning by allowing the stories (be it their self-stories or illustrative stories) to make deep and lasting connections to the information. they used the stories as a place to practice what they learned about resiliency and then apply it to their everyday lives. using stories to connect abstract concepts to students’ lives allowed the curriculum to become more meaningful to the students, and, thus, they were able to perceive resiliency in their lives by connecting the past, present, and future experiences. merleau-ponty (1948, trans. 2004), the french phenomenologist, claimed that our perceptual experience should be deducted with a reasoning or knowledge structure: it is our ‘bodily’ intentionality which brings the possibility of meaning into our experience by ensuring that its content, the things presented in experience, are surrounded with references to the past and future, to other places and other things, to human possibilities and situations (p. 10). iv. discussion. in reflecting upon the reaction of the students to their personal growth and enhanced understanding of self through rde it is without question that using story as a teaching strategy was paramount to their ability to learn the concepts of resiliency. students’ enthusiastic identification with the messages of the stories attested to its importance and value of this pedagogy. they supported the research that stories are vehicles that bring us to a better understanding of our lives (kilpatrick, 1993). meyer, k., licklider, b., and wiersema, j. journal of the scholarship of teaching and learning, vol. 9, no. 1, january 2009. 130 what would happen if major concepts from all disciplines could be paired with stories or metaphors that could illuminate meaning and understanding for students in the classroom? what would happen if educators focused on ‘re-storying’ lectures to include illustrations borrowed from folklore or personal encounters? how much more information could effectively be retained by students if we adjust our teaching strategies to include the art of storytelling? stories encapsulate clues by which educators, listening to the stories of the students, can use to know what is real and important to them and thus know what questions to ask that will enhance their learning (collins & cooper, 1997). karen gave voice to the importance of the story as a tool of instruction: the stories told really made an impact on how we learned the protective factors. it made the lesson more engaging and interesting. it made me realize that everything, story or situation, has a way of turning out for the best. i just have to look for it. by looking for the key factor in situations, i’ll be able to find the underlying meaning in almost everything. i hope to continue using the tools i have learned while in this class. i hope to continue to be resilient. the story is invaluable as a tool be it used as a metaphor, which connects abstract concepts to concrete understanding, folklore that allows one to immerse in mental practice, or the self-story which encourages exploration and analysis of one’s identity. this evidence suggests that educators should seek ways to incorporate stories as a method to enhance student learning. andrea framed it well from the perspective of a learner: i can almost 100% guarantee myself that five, ten years from now i won’t remember many details or concepts from my classes such as biology or chemistry, but there is no doubt in my mind that i will regularly use the concepts and ideas discussed throughout this course. the implications of this notion suggest that when applied in the field across a variety of disciplines, retention of information increases. what then would happen if instructors in the fields of math, science, or english, for example, offered a story, as a teaching tool, illustrating the desired concept to be taught? according the data from this study, students would construct meaning with the concept by way of putting themselves into the content of the story, thus creating a hook to which they can then connect knowledge with understanding. directed to the concept of resiliency, it would appear there is a way to prevent students from being devastated by crisis, challenges, or setbacks as was often observed by academic standards committee members who initiated the original project. the potential exists within rde that through the thoughtful use of stories, students may learn to address the adversity they experience with resilient behaviors and attitudes. the students in this study were able to make personal meaning and apply it to their lives through their understanding of the protective factors, as well as their abilities to use the tools of reframing and self-talk to confront and address the issues they faced. critical to this understanding was the story, whether it was metaphor, folklore, or self-story, that provided the insight and virtual practice to face future obstacles. v. conclusion. the stories of experience hold power in the process of shaping not only our perception, but also our character and values. with a keen understanding of that power, white (1982) meyer, k., licklider, b., and wiersema, j. journal of the scholarship of teaching and learning, vol. 9, no. 1, january 2009. 131 related, “what is imprinted deeply on our minds in our youth shapes who we are and what we shall become. the stories, the dreams we live by, are vital for our growth” (p. 22). if the use of story encompasses such a magnitude of possibilities to affect the connections students make with resiliency development, why would we, as educators, not enthusiastically embrace this pedagogical concept with which to help our students learn more deeply? references adams, r., allendoerfer, c., smith, t. r., socha, d., williams, d., & yasuhara, k. (2005). storytelling in engineering education. paper presented at the frontiers in education conference, indianapolis, in. benard, b. (1993). fostering resiliency in kids: protective factors in the family, school, and community. san francisco: far west laboratory for educational research and development. (eric document reproduction service no. ed335781). benard, b. (2004). resiliency: what we have learned. san francisco, ca: wested. braxton, j. m. (2004). five strategies to improve student retention. naspa leadership exchange (winter), 18-21. colaizzi, p. f. (1978). psychological research as the phenomenologist views it. in r. valle & m. king (eds.), existential-phenomenological alternatives for psychology (pp. 48-70). new york: oxford university press. collins, r., & cooper, p. j. (1997). the power of story: teaching through storytelling (second ed.). scottsdale, az: gorsuch scarisbrick. clandinin, d. j., & connelly, f. m. (1994). personal experience methods. in n. k. denzin & y. s. lincoln (eds.), handbook of qualitative research. thousand oaks, ca: sage publications. connelly, f. m., & clandinin, d. j. (1988). teachers as curriculum planner: narratives of experience. new york: teachers college press. connelly, f. m., & clandinin, d. j. (1990). stories of experience and narrative inquiry. educational researcher 19(5), 2-14. creswell, j.w. (2003). research design: qualitative, quantitative, and mixed methods approaches. thousand oaks, ca: sage publications, inc. crotty, m. (2003). the foundations of social research: meaning and perspective in the research process. thousand oaks, ca: sage publications, inc. egan, k. (1997). the educated mind: how cognitive tools shape our understanding. chicago, il: chicago university press. meyer, k., licklider, b., and wiersema, j. journal of the scholarship of teaching and learning, vol. 9, no. 1, january 2009. 132 esterberg, k. g. (2002). qualitative methods in social research. new york, ny: mcgraw hill. evans, r. p. (2000). the spyglass. new york, ny: simon & schuster books for young readers. gerdes, h., & mallinckrodt, b. (1994). emotional, social, and academic adjustment of college students: a longitudinal study of retention. journal of counseling & development, 72(january/february), 281-288. goleman, d. (1995). emotional intelligence: why it can matter more than iq. new york, ny: bantam. jones, s. r., vasti torres, & arminio, j. (2006). negotiating the complexities of qualitative research in higher education: fundamental elements and issues. new york, ny: routledge, taylor & francis group. kegan, r. (1982). the evolving self: problem and process in human development. cambridge, ma: harvard university press. kilpatrick, w. (1993). the moral power of good stories. american educator, summer, p. 24-25. livo, n. j. & rietz, s. a. (1987). storytelling activities. littleton, co: libraries unlimited, in. mcadams, d. p. (1993). stories we live by: personal myths and the making of the self. new york, ny: william morrow and company, inc. macaulay, t. b. (1952). macaulay: prose and poetry. cambridge, ma: harvard university press. masten, a. s. (2001). ordinary magic: resilience process in development. american psychologist, 56, (3), 227-238. matthews, e. (2002). the philosophy of merleau-ponty. montreal & kingston: mcgill-queen's university press. mello, r. (2001). the power of storytelling: how oral narrative influences children's relationships in classrooms. international journal of education and the arts, 2(1), 1-14. merleau-ponty, m. (1948, trans. 2004). the world of perception (o. davis, trans.). new york city, ny: routledge. merleau-ponty, m. (1964). the primacy of perception and other essays. evanston, il: northwestern university press. merriam, s. b. (2002). qualitative research in practice: examples for discussion and analysis. san francisco, ca: jossey-bass. moustakas, c. (1995). being-in, being-for, being-with. northvale, nj: jason aronson, inc. meyer, k., licklider, b., and wiersema, j. journal of the scholarship of teaching and learning, vol. 9, no. 1, january 2009. 133 pellowski, a. (1987). the family storytelling handbook: how to use stories, anecdotes, rhymes, handkerchiefs, paper, and other objects to enrich your family traditions. new york, ny: macmillan publishing company. ponterotto, j. g., & grieger, i. (2007). effectively communicating qualified research. the counseling psychologist, 35(3), 404-430. pulley, m. l., & wakefield, m. (2001). building resiliency: how to thrive in times of change. greensboro, nc: center for creative leadership. sprenger, m (1999). learning and memory: the brain in action. alexandria, va: association for supervision and curriculum development. werner, e. e., & smith, r. s. (1982). vulnerable but invincible: a longitudinal study of resilient children and youth. new york, ny: mcgraw hill. white, w. r. (1982). speaking in stories: resources for christian storytellers. minneapolis, mn: augsburg publishing house. widdershoven, g. a. m. (1993). the story of life: hermeneutic perspectives on the relationship between narrative and life history. in r. josselson & a. lieblich (eds.), the narrative study of lives (vol. 1, pp. 1-20). thousand oaks, ca: sage publications, inc. wolin, s.j., & wolin, s. (1993). the resilient self: how survivors of troubled families rise above adversity. new york, ny: villard. 503 service temporarily unavailable 503 service temporarily unavailable nginx/1.14.1 503 service temporarily unavailable 503 service temporarily unavailable nginx/1.14.1 503 service temporarily unavailable 503 service temporarily unavailable nginx/1.14.1 journal of the scholarship of teaching and learning, vol. 12, no.2, june 2012, pp. 39-64. differences in procrastination and motivation between undergraduate and graduate students li cao1 abstract: procrastination became increasingly prevalent among students in recent years. however, little research was found that directly compares academic procrastination across different academic grade levels.  the  present study used a self-regulated learning perspective to compare procrastination types and associated motivation between undergraduate and graduate students. sixty-six undergraduate and sixty-eight graduate students responded to a packet of questionnaires concerning their experience in an educational psychology class. the results show that students’ beliefs about the usefulness of procrastination were a better predictor of academic procrastination than self-efficacy beliefs and achievement goal orientations. student age was related to procrastination types. among the undergraduate procrastinators, the younger students were more likely to engage in active procrastination while the older students tended to engage in passive procrastination. implications and future research directions are discussed. keywords: procrastination, motivation, self-regulated learning, college students i. introduction. despite considerable research describing negative consequences, procrastination has become increasingly prevalent among university students in recent years (harriort & ferrari, 1996; knaus, 2000; steel, 2007). procrastination refers to the lack or absence of self-regulated performance and the behavioral tendency to postpone what is necessary to reach a goal (knaus, 2000). procrastination has long been viewed as a self-handicapping behavior that leads to wasted time, increased stress, and poor academic performance (özer, 2011; solomon & rothblum, 1984; tice & baumeister, 1997; wang & englander, 2010). research demonstrates that academic procrastination impacts both undergraduate and graduate students. over 70% of undergraduate students admitted to procrastinating on their academic tasks (ellis & knaus, 1977; schouwenburg, 1995), while more than 50% of them procrastinated consistently and problematically (day, mensink, & o’sullivan, 2000; ferrari, o’callaghan, & newbegin, 2005). most recently, klassen, et al. (2010) reported that about 58% of their undergraduate participants “report[ed] spending three hours or more per day in procrastination” (p. 372). solomon and rothblum (1984) found that undergraduate students procrastinated more often when writing term papers (46%) than when reading weekly assignments (30%) and studying for examinations (28%); and that (self-reported) fear of failure and task aversiveness were the two main reasons why undergraduate students procrastinated. research shows that undergraduate student procrastination is related to gender, laziness, and difficulty in making decisions (özer, demir, & ferrari, 2009; schouwenbury, 2004), perfectionism and control 1 department of educational innovation university of west georgia, 1601 maple street carrollton, ga 30118, phone (678)-8396118, fax (678)-839-6153, e-mail: lcao@westga.edu cao, l. journal of the scholarship of teaching and learning, vol. 12, no.2, june 2012. www.iupui.edu/~josotl 40 (burns, dittmann, nguyen, & mitchelson, 2000), and the ability to resolve role conflict between school and interpersonal relationships (senécal, julien, & guay, 2003). studies consistently show positive correlations between procrastination and undesirable behaviors or affective outcomes, such as failure to complete assignments, lower grades, low self-esteem, and higher stress (ferrari, 2001; schraw, wadkins, & olafson, 2007; tice & baumeister, 1997). academic procrastination is also a severe problem for graduate students (collins & veal, 2004; jiao, daros-voseles, collins, & onwuegbuzie, 2011; onwuegbuzie & jiao, 2000). disturbingly, onwuegbuzie (2004) found that graduate students tended to procrastinate more than undergraduate students. in  graduate  students,  procrastination  was  associated  with  (self-­‐ reported)  fear of failure, task aversiveness, reading ability, self-efficacy (collins, onwuegbuzie, & jiao, 2008), and various types of academic-related anxiety (onwuegbuzie,   2004;   onwuegbuzie  &  collins,  2001;  onwuegbuzie  &  jiao,  2000).  procrastination  has  a  negative   impact  on  graduate  students’  academic  achievement  (onwuegbuzie,  2000)  and  grade  point   averages  (prohaska, morrill, atiles, & perez, 2000). research also shows that undergraduate students perceive their procrastination tendencies are a barrier to academic success in college (fritzsche, rapp & hickson, 2003; kachgal et al., 2001). similarly, between   65   and   75%   of   graduate   students   wanted   to   decrease   their   procrastination (onwuegbuzie, 2004). despite students’ motivation and extensive research efforts to curtail this debilitating habit, academic procrastination has become increasingly prevalent, which suggests that procrastination is not entirely understood, and more research is needed (kachgal, hansen, & nutter, 2001; steel, 2007). as seen, an extensive body of research has examined the prevalence, reasons, and consequences of academic procrastination in undergraduate and graduate students. surprisingly, no  study  has  directly  compared  procrastination  in  undergraduate  and  graduate  students,   except   onwuegbuzie (2004) and özer (2011). onwuegbuzie (2004) reported that graduate students demonstrated an even greater tendency to procrastinate on academic tasks (3.5 times in keeping up with weekly reading assignments and 2.28 times in studying for examinations) than undergraduate students (onwuegbuzie, 2004). however, onwuegbuzie’s (2004) findings were based comparison of the graduate student data he collected recently with the undergraduate student data that solomon and rothblum (1984) observed two decades ago. onwuegbuzie’s (2004) approach to data collection and analysis raised a concern that the prevalence of procrastination among the current undergraduate students might be underestimated, since frequency of procrastination among the undergraduate students has increased in the past two decades (harriort & ferrari, 1996; knaus, 2000, steel, 2007). contrary to onwuegbuzie’s (2004) findings, özer (2011) found that undergraduate   students   claimed   to   procrastinate   more   than   graduate   students   on   studying   for   exams,   writing   term   papers,   and   reading   weekly   assignments. the inconsistent findings of onwuegbuzie (2004) and özer (2011) suggest that more research is needed to study similarities and differences of procrastination in undergraduate  and  graduate  students.   in addition to the methodological concern, the present study expanded the earlier focus on the nature, antecedents, etiology, and consequences of academic procrastination (knaus, 2000; sommer, 1990; steel, 2007). more recently, this research has shifted its focus from treating academic procrastination as a self-defeating personality flaw (ferrari, 1991; lay, 1990; milgram, dangour, & raviv, 1992; schouwenburg, 2004) to viewing academic procrastination as a complex phenomenon with cognitive, affective, and behavioral components (rothblum, solomon, & murakami, 1986; schraw et al., 2007; wolters, 2003). cao, l. journal of the scholarship of teaching and learning, vol. 12, no.2, june 2012. www.iupui.edu/~josotl 41 as a result of this conceptual shift, recent research stressed that motivational and cognitive factors must be considered together to understand academic procrastination (howell & buro, 2009; muszynski & akamatsu, 1991; steel, 2007). for instance, lee (2005) reported that intrinsic motivation had significant unique effects on procrastination. brownlow and reasinger (2000) found that low extrinsic motivation, together with perfectionism, external locus of control, and attribution style contributed to the tendency of delaying school tasks. howell and buro (2009; howell & watson, 2007) investigated how academic procrastination was correlated with beliefs, ability, achievement goals, and learning strategies. senécal et al., (1995) examined the extent that academic motivation predicted academic procrastination and they concluded that procrastination is a motivational problem that involves more than poor time management skills or trait laziness. these results demonstrate that ascertaining student motivation associated with academic procrastination would contribute to a better understanding of academic procrastination and ultimately lead to effective interventions to reduce its negative impact on student learning. however, no research study has directly examined similarities and differences in motivation of academic procrastination between undergraduate and graduate students. the present study addressed this gap by comparing procrastination and motivation of undergraduate and graduate students simultaneously. to facilitate the comparison, two different types of procrastinators were distinguished: passive procrastinators and active procrastinators. academic procrastination recent research noted that not all forms of procrastination lead to negative consequences and examined the adaptive values associated with procrastination (bernstein, 1998; ferrari, 1991; 1994). this research shows that procrastination is related to intrinsic motivation (senécal et al., 1995). students reported that course materials become less boring, more interesting, and more engaging when they procrastinate (schraw et al., 2007). other benefits of procrastination include freeing up time for planning and other activities, more concentrated effort, a greater sense of challenge, and peak experience immediately prior to exams (knaus, 2000; lay, edwards, parker, & endler, 1998; schraw et al., 2007). furthermore, procrastination does not necessarily affect the quality of performance. for instance, solomon and rothblum (1984) found that there was no relationship between students’ procrastination scores and their course grades. similarly, ferrari (1992) reported that procrastination scores were positively related to academic behavior delays but unrelated to exam scores. pychyl, morin, and salmon (2000) concluded “our results do not support the findings of previous research in this regard. there was no significant difference in exam performance between those students scoring high versus low on procrastination, despite the differences in the amount of time studied and onset of studying” (p. 147). these results suggest that procrastinators may also include those who choose to delay a task for the adaptive values of procrastination. in line with this alternative view, chu and choi (2005) distinguished passive procrastinators and active procrastinators. passive procrastinators were those who did not intend to procrastinate, but they often ended up postponing tasks because of their inability to make decisions quickly and to thereby act on them quickly. active procrastinators were significantly different from passive procrastinators described in the traditional sense (knaus, 2000; senécal et al., 1995; steel, 2007). active procrastinators procrastinated because they preferred pressure and often used procrastination as a deliberate self-motivating strategy in order to be adequately cao, l. journal of the scholarship of teaching and learning, vol. 12, no.2, june 2012. www.iupui.edu/~josotl 42 motivated (ferrari, johnson, & mcgown, 1995). because of their intention to accomplish the task and their ability to meet deadlines and produce satisfactory outcomes, the active procrastinators were believed to possess characteristics similar to non-procrastinators in managing their learning (chu & choi, 2005; choi & moran, 2009). the concept of active procrastination was included in the present study in order to examine the possible differences in procrastination and motivation between undergraduate and graduate students. more specifically, inclusion of active procrastination allowed the present study to examine whether active procrastination is associated with adaptive motivation factors, and whether active procrastinators actually performed better than passive procrastinators and nonprocrastinators. self-regulated learning perspective the present study used a self-regulated learning perspective (pintrich, 2000; zimmerman, 2008) to examine how procrastination is related to motivation in undergraduate and graduate students. self-regulated learning is described as an ‘‘active, constructive process whereby learners set goals for their learning and then attempt to monitor, regulate, and control their cognition, motivation, and behavior, guided and constrained by their goals and the contextual features in the environment’’ (pintrich, 2000, p. 453). the self-regulated learning perspective was selected because it focuses on motivational, cognitive, and metacognitive processes of student learning (pintrich, 2000; wolters, 2003; zimmerman & schunk, 2001). guided by this framework, the present study examined whether active and passive procrastinators possess distinctive characteristics in self-efficacy, metacognitive beliefs, and achievement goal, as well as test performance in undergraduate and graduate students. self-efficacy. self-efficacy refers to students’ judgment of their capability to accomplish tasks and succeed in activities (bandura, 1986, 1997). bandura (1986) was the first to introduce the association between procrastination and self-efficacy beliefs. he posited that students possess the capabilities to regulate their thoughts and actions by reflecting on the outcomes of their learning process. however, students who were skeptical of their ability to exercise control over their behavior tend to undermine their own efforts to deal effectively with situations that challenge their capabilities (bandura, 1986). existing research supports bandura’s (1986) position that self-efficacy plays an important role in task initiation and persistence (pintrich, 2000; schraw et al., 2007; schunk & pajares, 2005). an inverse relationship was found between self-efficacy belief and academic procrastination among college students (ferrari, parker, & ware, 1992; tuchman, 1991; wolters, 2003). for instance, tan et al., (2008) reported that self-efficacy for self-regulated learning was negatively correlated with procrastination. high self-efficacy for self-regulated learning also predicted students' expectations of doing well; low self-efficacy for self-regulated learning predicted students' expectations of not doing well academically. similarly, seo (2008) found that self-efficacy fully mediated the relationship between self-oriented perfectionism and academic procrastination, and that students with high self-oriented perfectionism procrastinated less than others. furthermore, chu and choi (2005) found that self-efficacy was correlated negatively with passive procrastination, but positively with active procrastination, and that passive procrastinators had significantly lower self-efficacy than the active procrastinators. exploring differences in the relationship between procrastination types and self-efficacy among undergraduate and graduate students would clarify how student judgment of academic capabilities influenced the tendency to procrastinate at different levels. cao, l. journal of the scholarship of teaching and learning, vol. 12, no.2, june 2012. www.iupui.edu/~josotl 43 metacognitive beliefs. recent research also shows that metacognitive beliefs play a role in procrastination (fernie & spada, 2008). metacognitive beliefs refer to the information individuals hold about their own cognition and internal states, as well as the coping strategies they activate in problematic situations (wells, 2000; wells & matthews, 1994, 1996). from a metacognitive standpoint, procrastinators are thought to delay or postpone action primarily because they doubt their own ability to complete a task, and they fear possible negative consequences of failing to adequately complete a task (shoham-saloman, avner & neeman, 1989). current theory has identified positive and negative metacognitive beliefs about procrastination (fernie & spada, 2008; spada, hiou, & nikcevic, 2006). positive metacognitive beliefs concern primarily the usefulness of procrastination in improving cognitive performance. they may include beliefs such as “procrastination helps creative thinking” or “when i procrastinate, i am unconsciously mulling over difficult decisions.” such beliefs may predispose students to delay task initiation as a form of coping. negative metacognitive beliefs concern primarily the uncontrollability of procrastination. they may include beliefs such as “procrastination makes me feel down” or “when i procrastinate, i waste a lot of time thinking about what i am avoiding” (fernie, spada, nikcevic, georgiou & moneta, 2009). such beliefs may perpetuate procrastination through predisposing students to intrusive thoughts and feelings which simultaneously consumes their cognitive resources necessary for concentration and controlling over thinking and coping (fernie et al., 2009). specific positive and negative metacognitive beliefs about procrastination were found in chronic procrastinators in the general population (spada, hiou, & nikcevic, 2006). for instance, spada,  hiou, and nikcevic (2006)  found that metacognitive beliefs about cognitive confidence (“my memory can mislead me at times”) predicted behavioral procrastination, and that positive metacognitive beliefs about worry (“worry can help me solve problems”) predicted decisional procrastination. they postulated that individuals who hold negative beliefs about their cognitive efficiency may doubt their task performance capabilities. the latter are likely to adversely impact motivation as well as task initiation and persistence, leading to behavioral procrastination. similarly, fernie et al., (2009) found that positive metacognitive beliefs about procrastination were positively correlated with decisional procrastination. negative metacognitive beliefs were positively correlated with both decisional and behavioral procrastination in undergraduate students. however, the influence of metacognitive beliefs about procrastination on students’ behaviors and motivation has not been studied in graduate students, and no study has directly compared undergraduate and graduate students’ metacognitive beliefs about procrastination. achievement goal orientation. the final motivational variable the present study examined was achievement goal orientations. achievement goal orientations represent the different purposes or reasons for students to engage in achievement situations (ames, 1984; pintrich, 2000). these purposes direct student cognition and behavior across a range of academic tasks or learning situations, and determine how they approach and engage in learning activities (ames, 1984). according to elliot and mcgregor’s (2001) (2 × 2) achievement goal framework, a mastery-approach goal applies to the students who focus on improving ability, or thoroughly understanding new information. a mastery-avoidance goal applies to the students who strive to avoid failing to learn what there is to learn (elliot & harackiewicz, 1996; pintrich, 2000). a performance-approach goal applies to the students who focus on doing better than their peers, or proving their self-worth to other people (ames & archer, 1988; dweck, 1991; moller & elliot, 2006). a performance-avoidance goal applies to the students who strive to avoid demonstrating a lack of competence with a particular topic (mcgregor, & elliot, 2002; midgley, kaplan & cao, l. journal of the scholarship of teaching and learning, vol. 12, no.2, june 2012. www.iupui.edu/~josotl 44 middleton, 2001). students with performance-avoidance goal orientations are also concerned about how they compare with others. however, these students focus on avoiding the demonstration of their lack of ability, or preventing the perception that they are not competent with a particular topic or skill (mcgregor, & elliot, 2002; midgley et al., 2001). in addition, work-avoidance goal orientation was included in the present study (elliot, 1999; maehr, 1983; nicholls, patashnick, & nolen, 1985). work-avoidance goal orientation applies to students who strive to minimize their effort for academic tasks, prefer the tasks that can be completed quickly and easily, or prefer not to work too hard. students with workavoidance goals tend to exhibit maladaptive motivation, cognitive and metacognitive strategies, and poor academic outcomes (howell & watson, 2007; meece & holt, 1993; middleton & midgley, 1997; wolters, 2003). work-avoidance goal was included in the present study because it provided an index to detect students’ motivational beliefs and behavior patterns of trying to get away with putting as little effort as possible into academic tasks (wolters, 2003). current research of achievement goal orientations supports the view that procrastination is one specific self-handicapping behavior (ferrari, 1992, 1994; ferrari & tice, 2000; ommundsen, 2001; rhodewalt, 1994; wolters, 2004). for instance, the mastery-approach goal was found to be related positively to higher levels of self-efficacy and help-seeking strategies (pintrich, 2000; schraw et al., 2007), but negatively to self-handicapping (midgley, arunkamar, & urdan, 1996; pintrich, 2000) and procrastination in undergraduate students (howell & watson, 2007; wolters, 2003, 2004). similarly, midgley and urdan (1995) found that self-handicapping was predicted negatively by a mastery goal orientation, but positively by performance-avoidance orientation. other research shows that students may procrastinate more and have higher test anxiety under conditions that foster a mastery-avoidance orientation (elliot & mcgregor, 2001; howell & buro, 2009; howell & watson, 2007), a performance-avoidance orientation (mcgregor & elliot, 2002), or work-avoidance orientation (blunt & pychyl, 1998; clark & hill, 1994; ferrari, 1991; ferrari & tice, 2000; wolters, 2003). contrary to the popular view of procrastination as a dysfunctional self-handicapping behavior, chu and choi (2005) argued that active procrastination is a self-regulatory behavior that some procrastinators intentionally engage in for adaptive values and positive outcomes. they described active procrastinators as possessing desirable characteristics similar to nonprocrastinators who maintain positive motivation toward the tasks and intend to learn and perform well in class. nevertheless, chu and choi (2005) did not include achievement goal orientations in their study, and no research has examined the difference of achievement goal orientations between undergraduate and graduate students. to address this gap, the present study adopted a more comprehensive framework (i.e., elliot & mcgregor’s (2 x 2) model, plus workavoidance goal orientations, maehr, 1983) to examine how achievement goal orientations relate to different types of procrastination (chu & choi, 2005) in undergraduate and graduate students. the present study recently, research on procrastination started to examine academic procrastination from the selfregulated learning perspective (schraw et al., 2007; senécal et al., 1995; wolters, 2003, 2004). this research distinguished different procrastination types and examined motivation factors associated with passive and active procrastination (e.g., chu & choi, 2005; schraw et al., 2007). however, the existing research was limited mostly to a single educational level. the present study contributed to the literature by using a cross-sectional design to compare undergraduate cao, l. journal of the scholarship of teaching and learning, vol. 12, no.2, june 2012. www.iupui.edu/~josotl 45 and graduate students’ procrastination types and the associated motivation variables in one subject-matter area. controlling the subject-matter area allowed the present study to exclude the influence of different subject-matter disciplines on students’ motivation and behaviors related to procrastination, and therefore would enhance validity of the study. based on the self-regulated learning perspective, self-efficacy, metacognitive beliefs, and achievement goals were examined in the present study, because these motivational variables were expected to be predictors of procrastination. more importantly, because they are malleable student characteristics, future interventions can be designed to work on these variables (banudra, 1997; pintrich, 2000; rakes & dunn, 2010; wolters, 2003). for instance, if self-efficacy, metacognitive beliefs, and achievement goal are found to be predictive of procrastination, courses can be designed to take pre-emptive action against academic procrastination by promoting student academic confidence, increasing guidance for self-regulation, and facilitating learning goal orientation. understanding how different types of academic procrastination relate to these motivational factors in undergraduate and graduate students would allow faculty and staff to make concerted efforts to more effectively tackle this prevalent problem. specifically, the present study addressed three research questions: (1) how procrastination types were associated with motivation for undergraduate and graduate students? (2) which motivational factors predicted different types of procrastination for undergraduate and graduate students? (3) what were the differences in motivation among the different types of procrastinators between undergraduate and graduate students? ii. method. a. participants. participants of the study included sixty-six undergraduate students and sixty-eight graduate students enrolled in two educational psychology classes in the college of education at a fouryear university in the southeastern u.s. the same instructor taught both classes for undergraduate and graduate students, thereby minimizing the threat to internal validity due to instructor differences. standard institutional review board procedures were followed to ensure the privacy and anonymity of the participants. of the 66 undergraduate students, 82% (54) were female and 18% (12) male. forty-three (80%) participants identified themselves as caucasian/white, ten (15%) as black, and three (5%) as other. they majored in early childhood (55%), middle grades (24%), secondary (4%), special education (7%), and other majors (10%). their age ranged from 20 to 59 (m=27.21, sd=9.28), suggesting that the sample included a considerable number of nontraditional students. of the 68 graduate students, 84% (57) were female and 16% (11) male. forty-three (64%) identified themselves as white, eighteen (27%) as black, and five (9%) as hispanic and other. they majored in counseling (65%) and other education majors (35%; e.g., early childhood, art education, social studies, etc.). they ranged in age from 22 to 56 (m=32.12, sd=9.04). inclusion of older non-traditional students was expected in the graduate sample. b. measurement and procedure. participants were invited to respond to a survey packet during the last class. the packet included the following measurement instruments. an educational psychology self-efficacy inventory cao, l. journal of the scholarship of teaching and learning, vol. 12, no.2, june 2012. www.iupui.edu/~josotl 46 consisting of eight items answered on a 5-point likert scale. participants were asked to indicate their level of agreement on each statement ranging from 1 (nothing like me) to 5 (a great deal like me). this self-developed questionnaire followed bandura’s (1986, 1997) guideline of selfefficacy scales and has been shown to be internally reliable in previous studies (e.g., nietfeld, cao, & osborne, 2006). sample items included “i am sure that i can learn educational psychology” (cronbach α=.79 for the total; .92 for the graduate, and .68 for the undergraduate, hence after). metacognitive beliefs about procrastination questionnaire (fernie et al., 2009) consisted of two-factors of eight items each measuring metacognitive beliefs about procrastination. the first factor (cronbach α=.81;.74/.86) represented positive metacognitive beliefs about procrastination (e.g., procrastination allows creativity to occur more naturally), while the second factor (cronbach α=.80;.78/.82) represented negative beliefs about procrastination (e.g., procrastination increases my worry). participants were asked to express their level of agreement with the statement on a likert scale ranging from 1 (not at all true) to 7 (very true). achievement goal orientations questionnaire consisted of 16 items on a 7-point likert scale (cronbach alpha=.79;.79/.77). for each item, the participants read a short statement and then chose a number from 1 to 7 to indicate how strongly they agree (7) or disagree (1) with the statement. the questionnaire included 12 items (elliot & mcgregor, 2001) that measured the masteryand performance-approach vs. masteryand performance-avoidance goal orientations, plus four items measuring the work-avoidance goal orientation (wolters, 2003). a sample item of mastery-approach goal orientation read, “i want to learn as much as possible from this class.” a sample mastery-avoidance goal orientation item included, “i worry that i may not learn all that i possibly could in this class.” a sample performance-approach goal orientation item is, “my goal in this class is to get a better grade than most of the other students.” a sample performanceavoidance goal orientation item included, “i just want to avoid doing poorly in this class.” a sample work-avoidance goal orientation item read, “i like the class work best that i can finish quickly.” academic procrastination. tuckman’s (1991) 16-item procrastination scale (cronbach α=.87;.90/.83) was used to measure “the tendency to waste time, delay, and intentionally put off something that should be done” (p. 479). participants were asked to indicate agreement on a likert scale ranging from 1 (not at all true to me) to 7 (very true to me) on a statement (e.g., “i needlessly delay finishing jobs, even when they’re important.”) of passive procrastination. active procrastination. choi and moran’s (2009) 16-item scale was used to identify active procrastinators (cronbach α=.83;.73/.86). this 7-point likert scale measures four defining characteristics of active procrastinators: (a) preference for pressure (e.g., “i tend to work better under pressure”), (b) intentional procrastination (e.g., “i intentionally put off work to maximize my motivation”), (c) ability to meet deadlines (e.g., “since i often start working on things at the last moment, i have trouble finishing assigned tasks most of the time” [reverse coded]), and (d) outcome satisfaction (e.g., “i feel that putting work off until the last minute does not do me any good” [reverse coded]). a composite score of these four subscales was used to assess the overall tendency toward active procrastination. iii. results. pearson correlation procedures were used to address the first research question: how procrastination types were associated with motivation for undergraduate and graduate students? cao, l. journal of the scholarship of teaching and learning, vol. 12, no.2, june 2012. www.iupui.edu/~josotl 47 no significant correlation was found between academic procrastination and active procrastination either for undergraduate or graduate students, suggesting that the academic procrastination scale and the active procrastination scale measured different constructs. as table 1 shows, for undergraduate students, academic procrastination is positively correlated with positive metacognitive beliefs about procrastination (r=.56, p<0.001), performance-avoidance goal orientation (r=.29, p<0.019), and work-avoidance goal orientation (r=.35, p<0.004); but negatively correlated with test performance (r=-.26, p<0.038) and age (r=.25, p<0.044). active procrastination is positively correlated with positive metacognitive beliefs about procrastination (r=.29, p<0.019); but negatively correlated with negative metacognitive beliefs about procrastination (r=-.51, p<0.001) and mastery-approach goal orientation (r=-.34, p<0.005). undergraduate students’ educational psychology self-efficacy beliefs are positively correlated with test performance (r=.46, p<0.001) and mastery-approach goal orientation (r=.31, p<0.012); but negatively correlated with mastery-avoidance goal orientation (r=-.42, p<0.001), performance-avoidance goal (r=-.27, p<0.027), and work-avoidance goal (r=-.25, p<0.044). their positive metacognitive beliefs about procrastination are positively correlated with performance-avoidance (r=.33, p<0.006) and work-avoidance goal orientation (r=.41, p<0.001); but negatively correlated with test performance (r=-.36, p<0.003), age (r=-.37, p<0.003), and mastery-approach goal orientation (r=-.39, p<0.001). for graduate students, academic procrastination is positively correlated with positive metacognitive beliefs about procrastination (r=.72, p<0.001), performance-avoidance goal orientation (r=.39, p<0.001), and work-avoidance goal orientation (r=.52, p<0.001); but negatively correlated with mastery-approach goal orientation (r=-.32, p<0.008). active procrastination is positively correlated with educational psychology self-efficacy (r=.37, p<0.002). their positive metacognitive beliefs about procrastination are positively correlated with mastery-avoidance goal orientation (r=.27, p<0.028), performance-avoidance (r=.42, p<0.001), and work-avoidance goal orientation (r=.49, p<0.001); but negatively correlated with age (r=-.35, p<0.004) and mastery-approach goal orientation (r=-.25, p<0.042). their negative metacognitive beliefs about procrastination are positively correlated with mastery-approach goal orientation (r=.27, p<0.024). a three-step hierarchical regression analysis was used to address the second question: which motivational factors predicted different types of procrastination for undergraduate and graduate students? the hierarchical approach was selected over a forced entry or stepwise method, because this approach allowed selection of predictors for the theoretical reasons to examine the added influence of different motivational variables on procrastination (field, 2009). before the regression analysis was conducted, normality of the dataset was examined using methods described by tabachnick and fidell (2001). specifically, the skewness and kurtosis scores of the dependent variables of the regression models [i.e., the total scores of the academic procrastination scale (tuckman, 1991) and the active procrastination scale (choi & moran, 2009)] were examined for both undergraduate and graduate students. none of the skewness and the kurtosis scores exceed 2.5 times of their corresponding standard errors (morgan, leech, gloechner, & barrett, 2011), suggesting the dataset normality was not violated. as table 2 shows, positive metacognitive beliefs about procrastination were the sole predictor of academic procrastination for both undergraduate students in step one (β=.56, t(63)=5.19, p<.001), step two (β=.54, t(62)=4.92, p<.001), and step three (β=.50, t(57)=3.75, p<.001); and graduate students in step one (β=.73, t(65)=8.16, p<.001), step two (β=.72, cao, l. journal of the scholarship of teaching and learning, vol. 12, no.2, june 2012. www.iupui.edu/~josotl 48 t(64)=8.03, p<.001), and step three (β=.60, t(59)=5.53, p<.001). the model explained 32% of the variance in academic procrastination score for undergraduate students and 51% for graduate students. in addition, table 2 shows that for undergraduate students active procrastination was predicted by negative metacognitive beliefs about procrastination in step one (β=-.47, t(63)=-4.25, p<.005), step two (β=-.47, t(62)=-4.26, p<.001), and step three (β=-.45, t(57)=-3.89, p<.001), plus mastery-approach goal orientations in step three (β=-.32, t(57)=-.2.31, p<.025). for graduate students, active procrastination was predicted by positive metacognitive beliefs about procrastination (β=.26, t(64)=2.30, p<.025) and educational psychology self-efficacy (β=.39, t(64)=3.48, p<.001) in step two; but only by educational psychology self-efficacy (β=.36, t(59)=3.16, p<.002) in step three. the model explained 29% of the variance in active procrastination score for undergraduate students and 6% for graduate students. in order to address the third research question about the differences in motivation among different types of procrastinators between undergraduate and graduate students, a two-step process (chu & choi, 2005) was used to categorize the participants into three subgroups for undergraduate and graduate students. in the first step, participants’ responses on tuckman’s (1991) academic procrastination scale were used to distinguish procrastinators from nonprocrastinators among undergraduate students. the undergraduate participants who scored less than the median score (3.00) on the tuckman scale were grouped as non-procrastinators and those who scored equal or greater than 3.00 were grouped as procrastinators. among the 66 undergraduate participants, 30 were categorized as non-procrastinators and 36 were categorized as procrastinators. in the second step, participants’ responses on choi and moran’s (2009) active procrastination scale were used to distinguish passive procrastinators from active procrastinators. among the 36 undergraduate procrastinators, those who scored less than the median score (3.75) on the active procrastination scale were grouped as passive procrastinators (n=16) and those who scored equal or greater than 3.75 were grouped as active procrastinators (n=20). the same procedure was used to distinguish the procrastinator groups for the graduate students. among the 68 graduate students, 33 were identified as non-procrastinators, 15 as passive procrastinators, and 20 as active procrastinators. analyses of covariate (ancova) procedures were used to examine differences of the major variables among non-procrastinators, passive procrastinators, and active procrastinators separately for undergraduate and graduate students. because the undergraduate group included nontraditional students (m=27.21, sd=9.28) and a significant age difference was found among the three procrastination groups (f(2,62)=9.08, p=.004; η²=.13), student age was used as a covariate to control the age effect on procrastination and motivation for the undergraduate group. for the undergraduate students, the ancova results revealed a significant omnibus effect among the three procrastination groups on metacognitive beliefs, educational psychology selfefficacy, achievement goals, and test performance (wilk’s λ=.54, f(2,62)=2.15, p=.008, η²=.26). as table 3 shows, a significant difference was found among the three procrastination groups in positive metacognitive beliefs about procrastination (f(2,62)=9.18, p=.001; η²=.23); negative metacognitive beliefs about procrastination (f(2,62)=5,64, p=.006; η²=.15); mastery-avoidance goal orientation (f(2,62)=3.50, p=.036; η²=.10); and work-avoidance goal orientation (f(2,62)=4.19, p=.020; η²=.12). the bonferroni procedures were used to further examine differences among the three groups. the pair-wise comparisons show that both active procrastinators (group 3, m=4.02, p =.001) and passive procrastinators (group 3, m=3.43, p =.040) reported a significantly higher level of positive metacognitive beliefs about procrastination than the non-procrastinators (group cao, l. journal of the scholarship of teaching and learning, vol. 12, no.2, june 2012. www.iupui.edu/~josotl 49 table 1. correlations among the major variables among undergraduate and graduate students. 1 2 3 4 5 6 7 8 9 10 11 12 1 test performance -.33** -.26* .06 .46** -.36** .10 .17 -.14 .04 -.34** -.17 2 age .24 --.25* -.13 .06 -.37** -.03 .24 -.15 -.22 -.37** -.09 3 academic procrastination -.16 -.17 -.19 -.19 .56 ** -.16 -.16 .08 .08 .29* .35** 4 active procrastination -.13 -.02 .14 -.02 .29* -.51** -.34** -.20 .08 -.04 .11 5 ed. psychology selfefficacy .02 .11 -.16 .37 ** --.22 .08 .31* -.42** .20 -.27* -.25* 6 positive beliefs about procrastination -.18 -.35 ** .72** .24 -.08 --.24 -.39** .09 .12 .33** .41** 7 negative beliefs about procrastination .15 .24 -.13 -.10 -.07 -.23 -.15 .21 -.07 .14 .00 8 mastery approach .12 .34** -.32** .11 .14 -.25* .27* -.19 .27* -.04 -.22 9 mastery avoidance -.33** -.08 .17 .15 -.03 .27* -.02 .19 --.00 .39** .17 10 performance approach .14 -.24 .24 .22 -.00 .20 -.13 -.16 .03 -.19 .28* 11 performance avoidance -.16 -.29* .39** .08 -.13 .42** -.17 -.20 .17 .23 -.33** 12 work avoidance -.09 -.29* .52** .21 -.06 .49** -.17 -.52** .00 .49** .42** - note: ** =significant at 0.01 level; * =significant at the 0.05 level (2-tailed). correlational coefficients above the diagonal line represent undergraduate students (n=66) and those below the diagonal line represent graduate students (n=68). cao, l. journal of the scholarship of teaching and learning, vol. 12, no.2, june 2012. www.iupui.edu/~josotl 50 table 2. summary of hierarchical regression analyses predicting academic and active procrastination. academic procrastination active procrastination model b std. error β b std. error β undrg grad undrg grad undrg grad undrg grad undrg grad undrg grad positive metacog. beliefs about procrast. .62 .60 .12 .07 .56** .73** .14 .21 .08 .12 .18 .22 step 1 negative metacog. beliefs about procrast. -.02 .03 .11 .07 -.02 .04 -.33 -.05 .08 .12 -.47** -.05 positive metacog. beliefs about procrasti. .60 .59 .12 .07 .54** .72** .15 .25 .09 .11 .20 .26* negative metacog. beliefs about procrast. -.02 .03 .11 .07 -.02 .03 -.33 -.01 .08 .11 -.47** -.01 step 2 educational psychology self-efficacy -.10 -.19 .16 .16 -.07 -.10 .11 .82 .11 .24 .10 .39** positive metacog. beliefs about procrasti. .56 .49 .15 .09 .50** .60** .06 .17 .10 .13 .07 .18 negative metacog. beliefs about procrasti. -.06 .06 .12 .08 -.06 .07 -.32 -.05 .08 .11 -.45** -.05 educational psychology self-efficacy -.08 -.15 .20 .16 -.06 -.08 .17 .76 .14 .24 .17 .36** mastery-approach .12 -.08 .12 .09 .13 -.09 -.19 .23 .08 .14 -.32* .23 mastery-avoidance -.04 .01 .10 .06 -.06 .02 .01 .05 .07 .08 .01 .07 performance-approach -.05 .01 .09 .05 -.07 .02 .04 .08 .06 .08 .09 .12 performance-avoidance .08 .03 .10 .05 .11 .05 .0 -.02 .07 .08 .01 -.04 step 3 work-avoidance .14 .12 .12 .10 .16 .16 .01 .19 .08 .15 .02 .21 note: ** p<.001; * p<.05. left column for undergraduate students (n=66): for academic procrastination, δr²=.32 (p<.001) for step 1; δr²=.00 (p<.001) for step 2; δr²=.03 (p<.001) for step 3. for active procrastination, δr²=.29 (p<.001) for step 1; δr²=.01 (p<.001) for step 2. δr²=.07 (p<.001) for step 3. right column for graduate students (n=68): for academic procrastination, δr²=.51 (p<.001) for step 1; δr²=.01 (p<.001) for step 2; δr²=.05 (p<.001) for step 3. for active procrastination, δr²=.06 (p=.15) for step 1; δr²=.15 (p<.002) for step 2; δr²=.08 (p<.008) for step 3. cao, l. journal of the scholarship of teaching and learning, vol. 12, no.2, june 2012. www.iupui.edu/~josotl 51 table 3. mean, sd, and ancova results of test performance, metacognitive beliefs about procrastination, self-efficacy, and achievement goal orientations of non-procrastinators, passive procrastinators, and active procrastinator with age as covariate. undergraduate student group 1 (n=30) group 2 (n=16) group 3 (n=20) total (n=66) graduate student (n=33) (n=15) (n=20) (n=68) f p η² age 28.07(10.69) 29.56 (11.69) 23.10(4.48) 27.21(9.28) 9.08 .00** .13 32.73 (9.12) 31.93 (10.56) 31.25 (8.01) 32.12 (9.04) 3.49 .07 .05 test scores 86.73 (8.36) 83.00 (8.07) 82.90 (8.55) 84.67 (8.44) 1.39 .26 .04 88.97 (5.60) 87.33 (8.05) 85.95 (6.46) 87.72 (6.49) 1.23 .30 .04 positive metacognitive beliefs about procrastination 2.78 (.83) 3.43 (.90) 4.02 (.98) 3.31 (1.03) 9.18 .00** .23 2.33 (.815) 3.63 (1.33) 3.78 (.87) 3.04 (1.18) 18.87 .00** .37 negative metacognitive beliefs about procrastination 4.72 (1.03) 5.02 (.69) 3.95 (1.24) 4.56 (1.10) 5.64 .00** .15 4.62 (1.16) 4.40 (1.38) 4.53 (1.20) 4.54 (1.18) .15 .86 .00 ed. psychology selfefficacy 3.74 (.69) 3.52 (.72) 3.58 (.91) 3.64 (.76) .49 .62 .02 3.87 (.53) 3.61 (.39) 3.66 (.61) 3.75 (.53) 1.59 .21 .05 mastery-approach 5.39 (1.11) 5.63 (1.11) 4.63 (1.48) 5.22 (1.28) 2.54 .09 .08 5.65 (1.02) 4.71 (1.04) 5.33 (1.19) 5.35 (1.12) 4.06 .02* .11 mastery-avoidance 4.67 (1.51) 5.27 (1.33) 4.13 (1.91) 4.65 (1.64) 3.50 .04* .10 4.30 (1.64) 4.62 (1.63) 4.95 (1.51) 4.56 (1.60) .96 .39 .03 performance-approach 3.39 (1.58) 3.48 (1.73) 3.93 (1.60) 3.58 (1.61) .38 .69 .01 2.73 (1.75) 2.93 (1.87) 3.70 (1.77) 3.06 (1.81) 1.72 .19 .05 performance-avoidance 4.66 (1.69) 5.40 (1.21) 5.12 (1.33) 4.97 (1.49) 1.71 .19 .05 3.88 (1.77) 5.00 (1.52) 5.32 (1.78) 4.55 (1.82) 4.83 .01* .13 work-avoidance 3.53 (1.25) 4.06 (.72) 4.60 (1.47) 3.98 (1.28) 4.19 .02* .12 2.63 (1.13) 3.47 (1.09) 3.94 (1.18) 3.20 (1.26) 8.68 .00** .21 note: *=p<.05, **=p<.001. df=(2, 62) for undergraduate students and df=(2, 64) for graduate students. group 1=non-procrastinators; group 2=passive procrastinators; group 3=active procrastinators. age was used as covariate in the ancova, but reported here for group comparison. cao, l. journal of the scholarship of teaching and learning, vol. 12, no.2, june 2012. www.iupui.edu/~josotl 52 1, m=2.78). however, active procrastinators (group 3, m=3.95) reported a significantly lower level of negative metacognitive beliefs about procrastination than passive procrastinators (group 2, m=5.02, p =.007) and the non-procrastinators (group 1, m=4.72, p =.028). furthermore, a significant difference was also found in the mastery-avoidance goal orientation between passive procrastinators (group 2, m=5.27) and active procrastinators (group 3, m=4.13, p =.032); and in the work-avoidance goal orientation between non-procrastinators (group 1, m=3.53) and active procrastinators (group 3, m=4.60, p =.017). no significant difference in test performance was found among the three procrastinator groups in the undergraduate students. for the graduate students, the anova results revealed a significant omnibus effect among the three procrastination groups on metacognitive beliefs, self-efficacy, achievement goals, and test performance (wilk’s λ=.46, f(2,62)=3.00, p=.001, η²=.33). as table 3 shows, a significant difference was found among the three procrastination groups in positive metacognitive beliefs about procrastination (f(2,64)=18.87, p=.001; η²=.37); mastery-approach goal orientation (f(2,64)=4.06, p=.022; η²=.11); performance-avoidance goal orientation (f(2,64)=4.83, p=.011; η²=.13); and work-avoidance goal orientation (f(2,64)=8.68, p=.001; η²=.21). again, the bonferroni analyses show that both active procrastinators (group 3, m=3.78, p =.001) and passive procrastinators (group 2, m=3.63, p =.001) reported a significantly higher level of positive metacognitive beliefs about procrastination than the non-procrastinators (group 1, m=2.33). in addition, the non-procrastinators (group 1, m=5.65) reported a significantly higher level of the mastery-approach goal orientation than passive procrastinators (group 2, m=4.71, p =.018). however, the non-procrastinators reported a significantly lower level of the performance-avoidance goal orientation (group 1, m=3.88, p =.016) and work-avoidance goal orientation (group 1, m=2.63, p =.001) than active procrastinators (group 3, m=5.32, m=3.94, respectively). no significant difference in test performance was found among the three procrastinator groups in the graduate students. iv. discussion and conclusion. the present study used a self-regulated learning perspective to compare undergraduate and graduate students’ procrastination types and associated motivation. the purpose was to better understand similarities and differences of procrastination behaviors and associated motivation in undergraduate and graduate students. the results contribute to research on procrastination and self-regulated learning and inform interventions addressing procrastination. results to the first research question on the relationships between procrastination types and motivation revealed three points of similarity of undergraduate and graduate students. the first similarity concerns the tendency and reason of procrastination. the results show that academic procrastination was more likely to occur in those who had stronger beliefs that procrastination was beneficial and would improve cognitive performance in both undergraduate and graduate students. the second similarity relates to the strength of the correlation between academic procrastination and students’ positive beliefs about procrastination. for both undergraduate and graduate students, the correlation between positive metacognitive beliefs about procrastination and academic procrastination was the strongest among all the relations. together, these findings suggest that students’ positive metacognitive beliefs about the adaptive values of procrastination play a more important role in propagating academic procrastination than other motivation variables, such as self-efficacy and achievement goal orientations. cao, l. journal of the scholarship of teaching and learning, vol. 12, no.2, june 2012. www.iupui.edu/~josotl 53 the third similarity points to the purpose of procrastination. the results show that procrastinators in undergraduate and graduate students had a higher tendency to avoid performing worse than their peers and to minimize their efforts for academic tasks (blunt & pychyl, 1998; clark & hill, 1994; ferrari, 1991; ferrari & tice, 2000; wolters, 2003). these findings demonstrate efficacy of the self-regulated learning perspective in the study of procrastination, and show that procrastination is a motivational problem that involves more than poor time management skills or trait laziness (özer, 2011; senécal et al., 1995). more importantly, these findings suggest that motivational and cognitive factors must be considered together to understand academic procrastination (howell & buro, 2009; muszynski & akamatsu, 1991; steel, 2007). in particular, metacognitive beliefs about procrastination, performanceavoidance goal, and work-avoidance goal need to be addressed to help both undergraduate and graduate students battle against academic procrastination. also, interesting differences were found on the negative correlates with academic procrastination between undergraduate and graduate students. first, the results show that age was negatively related to academic procrastination for undergraduate students, but not for graduate students. this finding suggests that younger undergraduate students were more likely to procrastinate than their older counterparts who were mostly nontraditional students. this finding is consistent with the previous research that procrastination tendency reaches a peak for persons in their middle-to-late 20s and declines until approximately age 60 (ferrari, johnson, & mcgown, 1995). this finding also implies the possibility that students may grow out of the procrastination problem as they become more experienced in school and more mature in life. consequently, procrastination can be approached as a developmental problem in undergraduate students, as well as a flaw in personality trait (özer, demir, & ferrari, 2009; jiao et al., 2011; schouwenbury, 2004; steel, 2007). second, a negative correlation was found between academic procrastination and masteryapproach goal orientation in the graduate students, but not in undergraduate students. it was a little surprising that no significant relation was found between academic procrastination and mastery-approach goal orientation in undergraduate students. however, the negative correlation found between academic procrastination and mastery-approach goal orientation in graduate students was expected. this finding suggests that procrastination was less likely to occur for the graduate students who seek to improve their knowledge and learn all there is to learn. the inverse relationship between academic procrastination and the mastery-approach goal is consistent with the previous research that a negative correlation exists between academic procrastination and a general mastery orientation (schraw et al., 2007). as the previous results show, students who procrastinated were less likely to adopt the learning goal and make the effort to learn everything there is to learn (howell & buro, 2009; wolters, 2003), but more likely to adopt avoidance goal orientations (howell & watson, 2007). the present data presented mixed results regarding the relationships between test performance and procrastination. on the one hand, the present result supported the previous finding that undergraduate students who reported high on procrastination score achieved lower on test performance (brinthaupt & shin, 2001; jiao et al., 2011; tice & baumeister, 1997; wang & englander, 2010). these results demonstrated that procrastination has a negative effect on test performance. on the other hand, the present data show that there is no significant difference among the three different procrastination groups in both undergraduate and graduate participants, despite their differences in the motional variables, e.g., metacognitive beliefs about procrastination and achievement goals discussed above. these results were consistent with the cao, l. journal of the scholarship of teaching and learning, vol. 12, no.2, june 2012. www.iupui.edu/~josotl 54 previous findings that procrastination scores were positively related to academic behavior delays but unrelated to exam scores (ferrari, 1992; solomon & rothblum, 1984). these mixed results suggest that a complex relationship between procrastination and academic performance. one possible explanation of the lack of influence of procrastination on test performance was the small sample size in each procrastination group in the present study, even though each group satisfied the minimum requirement (table 3) for the parametrical data analysis procedure such as anova and regression. another possible reason might be that the deleterious consequences of procrastination on performance are cumulative (ferrari et al., 1995) which might be better captured by measures of academic performance over time such as grade point average (gpa). the discrepancy noted between the present results and previous research does indicate that further research is necessary to understand at what point procrastination begins to affect performance (pychyl, morin, & salmon, 2000). similarly, differences were found in the correlations between active procrastination and motivation factors in undergraduate students and graduate students. for the undergraduate students, active procrastinators tended to be those who believed more in the usefulness of procrastination, had less concerns about uncontrollability of procrastination, and possessed lower mastery-approach goals. in contrast, for the graduate students, active procrastination tended to be those who were more confident about their ability to learn the class content. the correlates of active procrastination mostly concurred with the motivational factors identified in the existing research such as metacognitive beliefs and achievement goal orientations for undergraduate students (howell & buro, 2009; özer, 2011; schraw et al., 2007; steel, 2007; wolters, 2003). however, the association of active procrastination with self-efficacy suggests student beliefs of their ability to learn the class content was a unique motive for graduate students to engage in active procrastination. the positive correlation between student self-efficacy and active procrastination found in the present study is consistent with chu and choi’s (2005) observation. this result was also confirmed by the regression analysis showing self-efficacy as the sole predictor of active procrastination. these results suggest that graduate students tended to procrastinate when they felt more confident with their abilities to accomplish academic tasks. according to chu and choi (2005), this is because active procrastinators were confident in their abilities to meet deadlines and complete the tasks under time pressure, so they intentionally postponed academic tasks and directed their attention toward more urgent issues at hand. however, these results are inconsistent with prior observations that students who were confident about their abilities to do well tended to start their academic work in a more timely manner (bandura, 1986; steel, 2007; wolters, 2003). these conflicting results suggest that observations in the existing research are far from conclusive in regards to the relationships between self-efficacy and procrastination. nevertheless, the present results show that different motivational factors need to be considered to understand active procrastination in undergraduate and graduate students. in addition to the procrastination types, the present study examined associated motivational variables. again, mixed results were found on students’ positive metacognitive beliefs about procrastination. first, similarities were found in the undergraduate and graduate students who believed more about the usefulness of procrastination. these students reported a higher tendency to engage in academic procrastination. they tended to be younger in age within their group. they also tended to adopt lower mastery-approach goals but higher performanceavoidance goals and work-avoidance goals. at the same time, differences were found between undergraduate and graduate students regarding beliefs about the usefulness of procrastination. cao, l. journal of the scholarship of teaching and learning, vol. 12, no.2, june 2012. www.iupui.edu/~josotl 55 for the undergraduate students, those who held a stronger belief that procrastination was beneficial tended to have a higher active procrastination and lower performance on tests. for the graduate students, those who reported a stronger belief about the usefulness of procrastination tended to have a higher level of master-avoidance goal orientations. these students tended to try everything they can to avoid failure to learn all the materials, which may explain the reasons why procrastination occur to these students (brownlow & reasinger, 2000; jiao et al., 2011; onwuegbuzie, 2000). these results demonstrate that students’ positive metacognitive beliefs about procrastination were associated with maladaptive motivational and cognitive factors. reducing the positive metacognitive beliefs about procrastination would help both undergraduate and graduate students deal with procrastination. similarly, differences were found regarding negative metacognitive beliefs about procrastination between undergraduate and graduate students. among undergraduate students, those who were more concerned about the uncontrollability of procrastination were less likely to engage in active procrastination. in contrast, among the graduate students, those who were more concerned about the uncontrollability of procrastination tended to adopt a stronger masterapproach goal orientation. these findings suggest that reinforcing the negative metacognitive beliefs may help undergraduate students to reduce active procrastination and graduate students to adopt the mastery-approach goal which is most desirable to promote learning (elliot & mcgregor, 2001; fernie & spada, 2008; howell & watson, 2007; wolters, 2003). results to the second research question largely confirmed the findings of the first research question. positive metacognitive beliefs about procrastination were the sole predictor of academic procrastination for both undergraduate and graduate students, even when self-efficacy and achievement goal orientations were considered. these findings suggest that metacognitive beliefs play a more important role in academic procrastination than self-efficacy and academic achievement goal orientations in undergraduate and graduate students. therefore, an attempt to help students overcome academic procrastination may be more effective by focusing on students’ beliefs of the usefulness of their procrastination. similar to the correlation results above, the regression results on active procrastination varied between undergraduate and graduate students. the results show that undergraduate students tended to engage in active procrastination when they were less concerned about the uncontrollability of procrastination and less oriented toward learning in class. these findings are inconsistent with chu and choi’s (2005) characterization of active procrastination. chu and choi (2005; choi & moran, 2009) posited that active procrastinators intentionally delayed academic tasks because they preferred time pressure, and they possess the confidence and ability to meet deadlines. however, the function of master-approach goal orientation as a negative predictor of active procrastination clearly shows that active procrastination is associated with maladaptive motivation value, and that the purpose of students engaging in active procrastination is not to learn and develop their competences. apparently, more research is needed to examine the notion of active procrastination and address the question: is active procrastination associated with desirable cognitive and motivational characteristics in undergraduate students? the regression results on active procrastination show that educational psychology selfefficacy is a significant positive predictor to active procrastination in graduate students. this finding is consistent with chu and choi’s (2005) observation of the positive correlation between self-efficacy and active procrastination. it suggests that students may intentionally delay academic tasks when they have strong beliefs about their abilities to learn the class materials. this finding demonstrates that active procrastination is associated with self-efficacy, which is cao, l. journal of the scholarship of teaching and learning, vol. 12, no.2, june 2012. www.iupui.edu/~josotl 56 often viewed as a desirable motivation variable (bandura, 1986) in graduate students. evidently, more research is needed to sort out procrastination among the high self-efficacy graduate students. one way to achieve this purpose is to conduct multivariate studies of procrastination that include ability and motivation. as bandura (1997) suggested, students’ self-efficacy beliefs have a significant impact on their task initiation, self-regulatory efforts, and academic performance when adequate levels of ability and motivation exist. this position suggests that the relationship of self-efficacy with task initiation, efforts, and academic performance is not straightforward, but mediated by a certain level of ability and motivation. again, results to the third research question revealed similarities and differences among the three procrastinator groups in undergraduate and graduate students. the group comparisons show that the passive procrastinators and active procrastinators in undergraduate and graduate students reported a significantly higher level of beliefs about the usefulness of procrastination and work-avoidance goal orientation than non-procrastinators. these results suggest that active procrastinators and passive procrastinators are similar in believing procrastination is useful. however, their intent to engage in procrastination is to get away with putting as little efforts as possible in achievement tasks (elliot, 1999; maehr, 1983; nicholls, patashnick, & nolen, 1985). these findings are consistent with previous research (schraw et al., 2007; wolters, 2003) that procrastination is an irrational delay, or avoidance, of academic tasks and a failure of selfregulation of the learning process (senécal et al., 1995; steel, 2007). the group comparisons also reveal differences among the three procrastinator groups between undergraduate and graduate students. among the undergraduate students, the active procrastinators were the youngest in age of the three procrastination groups, and they were significantly younger than the passive procrastinators. these findings suggest that among the undergraduate procrastinators, the younger students tended to engage in active procrastination while the older students tended to engage in passive procrastination. also, active procrastinators reported the least concerns about the uncontrollability of procrastination among the three procrastinator groups; and their concerns were significantly lower than those of the nonprocrastinators and passive procrastinators. furthermore, active procrastinators reported a significantly lower level of mastery-avoidance goal orientation than passive procrastinators. these findings are consistent with the results to the first and second research question discussed above. they suggest that the reasons undergraduate active procrastinators procrastinate relate to their minimal concern with the negative consequences of procrastination and failure to learn all of the class materials. in addition, these results support chu and choi’s (2005) differentiation between active and passive procrastinators. in this case, active procrastinators are different from passive procrastinators in negative metacognitive beliefs about procrastination and masteryavoidance goal orientation. while a lower level of negative metacognitive beliefs about procrastination is consistent with active procrastinators’ intentional delay of academic tasks (chu & choi, 2005), the influence of master-avoidance goal orientations in active procrastination has not yet been adequately examined (elliot & mcgregor, 2001; howell & buro, 2009; howell & watson, 2007). further research in this area will facilitate greater understanding of the nature of procrastination, achievement goal orientation, and self-regulated learning (pintrich, 2000; wolters, 2003). two differences stood out among the three procrastination groups in graduate students. non-procrastinators reported a significantly higher level of mastery-approach goal orientations than passive procrastinators, but a significantly lower level of performance-avoidance goal orientations than active procrastinators. while the finding concerning the mastery-approach goal cao, l. journal of the scholarship of teaching and learning, vol. 12, no.2, june 2012. www.iupui.edu/~josotl 57 confirmed the negative correlation of the mastery-approach orientation with self-handicapping (midgley & urdan, 1995; midgley, arunkamar, & urdan, 1996; pintrich, 2000) and procrastination (howell & watson, 2007; wolters, 2003), the finding about the performanceavoidance goal orientation is inconsistent with the research that active procrastination was associated with adaptive values of procrastination (chu & choi, 2005). similar to the results for the undergraduate students, these results also challenged chu and choi’s (2005) description that active procrastinators are more similar to non-procrastinators than to passive procrastinators, even though active procrastinators procrastinate to the same degree as passive procrastinators. more research is called to look into the inconsistent results between the present study and chu and choi’s (2005) work in order to better understand the nature of active procrastination. for instance, the future research could use quantitative and qualitative designs to examine adaptive and maladaptive characteristics of active and passive procrastinators in the behavioral, motivational, and affective domains. one way to investigate the nature of active procrastination is to identify the procrastinators who are successful in managing their learning process and achieving superior academic performances; and then examine differences in the beliefs, affects, and behaviors of these successful procrastinators as compared to unsuccessful procrastinators and non-procrastinators. future research could also examine to what extent students’ ability and motivation would be adequate so that self-efficacy enables them to exercise some control over their thoughts, feelings, and actions. at the same time, this research could also indentify to what extent, and under what conditions, students’ ability and motivation would become inadequate so that their self-efficacy leads to underestimation of difficulty of a task while simultaneously overestimating the positive benefits of procrastination (schraw et al., 2007). this line of research would advance research of procrastination and self-regulated learning. practically, results of this research would help design interventions to help graduate students avoid overconfidence of their ability and consequently failing to self-regulate their learning (pintrich, 2000; senécal et al., 1995; steel, 2007; wolters, 2003). in summary, the findings of the present study extend the research on procrastination by providing a more in-depth look at procrastination types and the associated motivation among undergraduate and graduate students simultaneously in one subject area. the present results suggest that students’ beliefs about the usefulness of procrastination play a more important role in propagating academic procrastination than other motivation variables for both undergraduate and graduate students. in contrast, different motivational factors, including metacognitive beliefs, self-efficacy, and achievement goal orientations, were involved in active procrastination for undergraduate and graduate students. in addition, student age was related to procrastination types particularly in undergraduate students. among the undergraduate procrastinators, the younger students were more likely to be active procrastinators, while the older students tended to be passive procrastinators. these results confirmed the traditional view that procrastination is related to undesirable factors that hinder learning (day et al., 2000; ferrari, 2001; jiao et al., knaus, 2000; lay, 1990; steel, 2007); but also offered mixed support to the notion that active procrastination is associated with adaptive values of procrastination (chu & choi, 2005, choi & moran, 2009) and motivational factors conductive to learning (wolters, 2003, 2004). clearly, more evidence is needed to demonstrate that procrastination is not a result of students’ systematic underestimation of the difficulty of the task while simultaneously overestimating the positive benefits of procrastination (schraw et al., 2007). cao, l. journal of the scholarship of teaching and learning, vol. 12, no.2, june 2012. www.iupui.edu/~josotl 58 the present results demonstrated that the self-regulated leaning perspective was useful in studying a complex phenomenon like procrastination. however, the present results should be interpreted with caution. the present study was limited to a relatively small sample observed in one subject area for a short period of time, and the cross-sectional design precluded causal inferences. studies with larger samples across different subject areas, and tasks over time will expand the research on procrastination, motivation, and self-regulated learning. in particular, further research is needed to investigate the notion of active procrastination for a better understanding of the nature of procrastination. also, more studies are needed to examine whether self-efficacy functions as a motivational factor that encourages students to procrastinate, or as a deterrent that discourages them to procrastinate in academic situations. the present study used a self-reported measure of procrastination. future research might employ observation of actual procrastination behavior as an additional, confirmatory measure of student procrastination. the incorporation of such data would strengthen the results of future investigations of procrastination, motivation, and self-regulatory behaviors. despite the above limitations, the present results illustrate the importance of examining the relationships between procrastination, motivation, and self-regulated learning in the research of procrastination. they also suggest implications for educational practice. in particular, interventions designed to curtail academic procrastination among undergraduate students might be more effective if they focus on decreasing students’ positive metacognitive beliefs about procrastination, and if they pair the younger students with non-traditional students. the present results also raised questions about the role of procrastination in the college classroom. one such question concerns whether teachers and students should be more accepting of procrastination, or even attempt to promote the “safe” active procrastination (choi & moran, 2009). although the present results are preliminary in nature, they clearly suggest that different variables need to be considered in future research and interventions to reduce procrastination in undergraduate and graduate students. references ames, c. (1984). competitive, cooperative, and individualistic goal structures: a motivational analysis. in r. ames & c. ames (eds.), research on motivation in education (vol. 1, pp. 177207). new york: academic press. ames, c., & archer, j. (1988). achievement goals in the classroom: student learning strategies and achievement motivation. journal of educational psychology, 18, 409-414. bandura, a. (1986). social foundations of thoughts and action: a social cognitive theory. englewood cliffs, nj: prentice-hall. bandura, a. (1997). self-efficacy: the exercise of control. new york: cambridge university press. bernstein, p. (1998). against the gods: the remarkable story of risk. new york: wiley. cao, l. journal of the scholarship of teaching and learning, vol. 12, no.2, june 2012. www.iupui.edu/~josotl 59 blunt, a., & pychyl, t. a. (1998). volitional action and inaction in the lives of undergraduate students: state orientation, procrastination and proneness to boredom. personality individual difference, 24, 837–846. brinthaupt, t. m., & shin, c. m., (2001), the relationship of academic cramming to flow experience. college student journal, 35(3), 457-272. brownlow, s., & reasinger, r. (2000). putting off until tomorrow what is better done today: academic procrastination as a function of motivation toward college work. journal of social behavior & personality, 15(5), 15-34. burns, l., dittmann, k., nguyen, n., & mitchelson, j. (2000). academic procrastination, perfectionism, and control: associations with vigilant and avoidant coping. journal of social behavior & personality, 15(5), 35-46. choi, j. n., & moran, s. v. (2009). why not procrastinate? development and validation of a new active procrastination scale. the journal of social psychology, 149(2), 195-211. chu., a. h. c., & choi, j. n. (2005). rethinking procrastination: positive effects of “active” procrastination behavior on attitude and performance. the journal of social psychology, 145(3), 245-264. collins, k., onwuegbuzie, a., & jiao, q. g. (2008). reading ability as a predictor of academic procrastination among african american graduate students. reading psychology, 29(6), 493507. collins, k. m. t., & veal, r. e. (2004). off-campus adult learners’ levels of library anxiety as a predictor of attitudes toward the internet. library & information science research, 26(1), 5-14. day, v., mensink, d., & o'sullivan, m. (2000). patterns of academic procrastination. journal of college reading and learning, 30, 120-134. dweck, c. (1991). self-theories and goals: their role in motivation, personality, and development. in r. a. dienstbier (ed.), nebraska symposium on motivation (vol. 38, pp. 195273). lincoln: university of nebraska press. elliot, a. j. (1999). approach and avoidance motivation and achievement goals. educational psychologist, 34, 169-189. elliot, a. j., & church, m. a. (1997). a hierarchical model of approach and avoidance achievement motivation. journal of personality and social psychology, 72, 218-232. elliot, a. j., & harachiewicz, j. m. (1996). approach and avoidance achievement goals and intrinsic motivation: a meditational analysis. journal of personality and social psychology, 70, 461-475. cao, l. journal of the scholarship of teaching and learning, vol. 12, no.2, june 2012. www.iupui.edu/~josotl 60 elliot, a. j., & mcgregor, h. a. (2001). a 2 x 2 achievement goal framework. journal of educational psychology, 80(3), 501-519. ellis, a., & knaus, w. j. (1977). overcoming procrastination. new york: signet. fernie, b. a., & spada, m. m. (2008). metacognitions about procrastination: a preliminary investigation. behavioural and cognitive psychotherapy, 36, 359-364. fernie, b. a., spada, m. m., nikcevic, a. v., georgiou, g. a., & moneta, g. b. (2009). metacognitive beliefs about procrastination: development and concurrent validity of a selfreport questionnaire. journal of cognitive psychotherapy, 23(4), 283-293. ferrari, j. r. (1991). self-handicapping by procrastinators: protecting self-esteem, social esteem, or both? journal of research in personality, 25, 245-261. ferrari, j. r. (1992). procrastinators and perfect behavior: an exploratory factor analysis of self presentation, self-awareness, and self-handicapping components. journal of research in personality, 26, 75-84. ferrari, j. r. (1994). dysfunctional procrastination and its relationship with self-esteem, interpersonal dependency, and self-defeating behaviors. personality and individual differences, 17, 673–679. ferrari, j. r. (2001). procrastination as self-regulation failure of performance: effects of cognitive load, self-awareness, and time limits on “working best under pressure.” european journal of personality, 15, 391–406. ferrari, j. r., johnson, j. l., & mcgown, w. g. (1995). procrastination and task avoidance: theory, research, and treatment. new york: plenum press. ferrari, j. r., o’callaghan, j., & newbegin, i. (2005). prevalence of procrastination in the united states, united kingdom, and australia: arousal and avoidance delays among adults. north american journal of psychology, 7, 2-6. ferrari, j. r., & tice, d. m. (2000). procrastination as a self-handicap for men and women: a task-avoidance strategy in a laboratory setting. journal of research in psychology, 34, 73–83. ferrari, j. r., parker, j. t., & ware, c. b. (1992). academic procrastination: personality correlates with myers-briggs types, self-efficacy, and academic locus of control. journal of social behavior and personality, 7, 595-602. field, a. (2009). discovering statistics using spss (3rd ed.). los angeles: sage. fritzsche, b. rapp, b. y., & hickson, k. c. (2003). individual differences in academic procrastination tendency and writing success. personality and individual differences, 35 15491558. cao, l. journal of the scholarship of teaching and learning, vol. 12, no.2, june 2012. www.iupui.edu/~josotl 61 harriott, j., & ferrari, j. (1996). prevalence of procrastination among samples of adults. psychological reports, 78, 611–616. howell, a. j., & buro, k. (2009). implicit beliefs, achievement goals, and procrastination: a meditational analysis. learning and individual differences, 19, 151-154. howell, a. j., & watson, d. c. (2007). procrastination: associations with achievement goal orientation and learning strategies. personal and individual differences, 43, 167-178. jiao, q. g., daros-voseles, d. a., collins, k. m. t., & onwuegbuzie, a. j. (2011). academic procrastination and the performance of graduate-level cooperative groups in research methods courses. journal of the scholarship of teaching and learning, 11(1), 119-138. klassen, r. m., ang, r. p., chong, w. h., krawchuk, l. l., huan, v. s., wong, i. y. f., & yeo, l. s. (2010). academic procrastination in two settings: motivation correlates, behavioral patterns, and negative impact of procrastination in canada and singapore. applied psychology: an international review, 59(3), 361-379. knaus, w. j. (2000). procrastination, blame, and change. journal of social behavior and personality, 15, 153–166. lay, c. h. (1990). working to schedule on personal projects: an assessment of personal-project characteristics and trait procrastination. journal of social behavior and personality, 5, 91-103. lay, c. h., edwards, j. m., parker, j. d., & endler, n. s. (1989). an assessment of appraisal, anxiety, coping, and procrastination during an examination period. european journal of personality, 3, 195–208. lee, e. (2005). the relationship of motivation and flow experience to academic procrastination in university students. journal of genetic psychology, 166(1), 5-14. maehr, m. l. (1983). on doing well in science: why johnny no longer excels, why sarah never did. in s. paris, g. olson, & h. stevenson, (eds.), learning and motivation in the classroom (pp. 179-120). hillsdale, nj: erlbaum. mcgregor, h. a., & elliot, a. j. (2002). achievement goals as predictors of achievementrelevant processes prior to task engagement. journal of educational psychology, 94, 381–395. meece, j., & holt, k. (1993). a pattern analysis of students’ achievement goals. journal of educational psychology, 85, 582-590. middleton, m., & midgley, c. (1997). avoiding the demonstration of lack of ability: an underexplored aspect of goal theory. journal of educational psychology, 89, 710-718. cao, l. journal of the scholarship of teaching and learning, vol. 12, no.2, june 2012. www.iupui.edu/~josotl 62 midgley, c., arunkumar, r., & urdan, t. (1996). if i don’t do well tomorrow, there’s a reason: predictors of adolescents’ use of academic self-handicapping behavior. journal of educational psychology, 88, 423-434. midgley, c., kaplan, a., & middleton, m. (2001). performance-approach goals: good for what, for whom, under what circumstances, and at what cost? journal of educational psychology, 93, 77-86. midgley, c., & urdan, t. (1995). predictors of middle school students’ use of self-handicapping strategies. journal of early adolescence, 15, 389-411. milgram, n., dangour, w., & raviv, a. (1992). situational and personal determinants of academic procrastination. journal of general psychology, 11, 123-133. moller, a. c., & & elliot, a. j. (2006). the 2 x 2 achievement goal framework: an overview of empirical research. in a. v. mittel (ed.), focus on educational psychology (pp. 307-326). hauppauge, ny: nova science. morgan, g.a., leech, n.l., gloechner, g.w., & barrett, k.c. (2011). spss for introductory and intermediate statistics (4th ed.). new york: routledge. muszynski, s., and akamatsu, t. (1991). delay in completion of doctoral dissertations in clinical psychology. professional psychology: research and practice, 22, 119-123. nicholls, j.g., patashnick, m., & nolen, s.b, (1985). adolescents’ theories of education. journal of educational psychology, 77, 683-692. nietfeld, j. l., cao, l., & osborne, j. w. (2006). the effect of distributed monitoring exercises and feedback on performance and monitoring accuracy. metacognition and learning, 2, 159179. ommundsen, y. (2001). self-handicapping strategies in physical education classes: the influence of implicit theories of the nature of ability and achievement goal orientations. psychology of sport and exercise, 2, 139-156. onwuegbuzie, a. (2000). academic procrastinators and perfectionistic tendencies among graduate students. journal of social behavior and personality, 15, 103-109. onwuegbuzie, a. (2004). academic procrastination and statistics anxiety. assessment & evaluation in higher education, 29(1), 3-19. onwuegbuzie, a. j., & collins, k. m. t. (2001). writing apprehension and academic procrastination among graduate students. perceptual and motor skills, 92, 560-562. onwuegbuzie, a., & jiao, q. g. (2000). i’ll go to the library tomorrow: the role of procrastination in library anxiety. college and research library, 61(1), 45-54. cao, l. journal of the scholarship of teaching and learning, vol. 12, no.2, june 2012. www.iupui.edu/~josotl 63 özer, b. u. (2011). a cross sectional study on procrastination: who procrastinate more? 2011 international conference on education, research and innovation, ipedr, 18, 34-37. özer, b., demir, a., & ferrari, j. (2009). exploring academic procrastination among turkish students: possible gender differences in prevalence and reasons. journal of social psychology, 149(2), 241-257. pintrich, p. r. (2000). the role of goal orientation in self-regulated learning. in m. boekaerts, p. r. pintrich, & m. zeidner (eds.), handbook of self-regulation. san diego: academic press. prohaska, v., morrill, p., atiles, i., & perez, a. (2000). academic procrastination by nontraditional students. journal of social behavior and personality, 15, 125–134. pychyl, t. a., morin, r. w., & salmon, b. r. (2000). procrastination and the planning fallacy: an examination of the study habits of university students. journal of social behavior and personality, 15(5), 135-150. rhodewalt, f. (1994). conceptions of ability, achievement goals, and individual differences in self-handicapping behavior: on the application of implicit theories. journal of personality, 62, 67–85. rothblum, e. d., solomon, l. j., and murakami, j. (1986). affective, cognitive, and behavioral differences between high and low procrastinators. journal of counseling psychology, 33, 387394. schouwenburg, h. c. (2004). academic procrastination: theoretical notions, measurement, and research. in h. c. schouwenburg, c. h. lay, t. a. pychyl, & j. r. ferrari (eds.), counseling the procrastinator in academic settings (pp. 3–17). washington, dc: american psychological association. schraw, g., wadkins, t., & olafson, l. (2007). doing the things we do: a grounded theory of academic procrastination. journal of educational psychology, 99(1), 12-25. senécal, c., julien, e., & guay, f. (2003). role conflict and academic procrastination: a selfdetermination perspective. european journal of social psychology, 33, 135–145. senécal, c., koestner, r., & vallerand, r. j. (1995). self-regulation and academic procrastination. journal of social psychology, 135(5), 607-619. seo, e. (2008). self-efficacy as a mediator in the relationship between self-oriented perfectionism and academic procrastination. social behavior & personality: an international journal, 36(6), 753-764. cao, l. journal of the scholarship of teaching and learning, vol. 12, no.2, june 2012. www.iupui.edu/~josotl 64 shoham-saloman, v., avner, r., & neeman, r. (1989). you’re changed if you do and changed if you don’t: mechanisms underlying paradoxical interventions. journal of consulting and clinical psychology, 57, 590–598. sommer, w. g. (1990). procrastination and cramming: how adept students ace the system. journal of american college health, 39, 5-10.????? solomon, l. j., & rothblum, e. d. (1984). procrastination assessment scalestudents (pass). in j. fischer & k. corcoran (eds.), measures for clinical practice (pp. 446-452). new york: the free press. spada, m. m., hiou, k., & nikcevic, a. v. (2006). metacognitions, emotions, and procrastination. journal of cognitive psychotherapy, 20, 319-326. steel, p. (2007). the nature of procrastination: a meta-analytic and theoretical review of quintessential self-regulatory failure. psychological bulletin, 133, 65-94. tabachnick, b.g., & fidell, l.s. (2001). using multivariate statistics, (4th ed.). boston: pearson. tan, c., ang, r., klassen, r., lay see, y., wong, i., huan, v., et al. (2008). correlates of academic procrastination and students' grade goals. current psychology, 27(2), 135-144. tice, d., & baumeister, r. f. (1997). longitudinal study of procrastination, performance, stress, and health: the cost and benefits of dawdling. psychological science, 8, 454–458. wang, z., & englander, f. (2010). a cross-disciplinary perspective on explaining student performance in introductory statistics -what is the relative impact of procrastination? college student journal, 44(2), 458-471. wells, a. (2000). emotional disorders and metacognition: innovative cognitive therapy. chichester, uk: wiley. wells, a., & matthews, g. (1994). attention and emotion: a clinical perspective. hover, uk: erlbaum. wells, a., & matthews, g. (1996). modelling cognition in emotional disorder: the s-ref model. behaviour research and therapy, 34, 881-888. wolters, c. a. (2003). understanding procrastination from a self-regulated learning perspective. journal of educational psychology, 95(1), 179-187. wolters, c. a. (2004). advancing achievement goal theory: using goal structures and goal orientations to predict students’ motivation, cognition, and achievement. journal of educational psychology, 96(2), 236-25. v12n2cao v12n2cao.2 v12n2cao.3 microsoft word hughesfinal.doc journal of scholarship and teaching, vol. 6, no. 1, august 2006, pp 110 117. bridging the theory-practice divide: a creative approach to effective teacher preparation jacqueline a. hughes, ph.d.1 abstract: teacher educators need to remain current regarding the challenges that prospective teachers are going to face in their classrooms. one way to maintain this currency is for teacher educators periodically to spend some time in the k-12 classroom testing the theories they teach. this paper will discuss the benefits both teacher educators and prospective teachers will derive from engaging in such an activity. i. introduction preparing prospective teachers for the realities of today’s classrooms is a complex and challenging undertaking for teacher educators. this complexity and challenge is a result of the changing nature of the classroom. schools today face an increasing number of language learners, the mainstreaming of special population students, and, working with a standards driven curriculum, all of which present new challenges for the teacher as they attempt to meet their students educational needs. as a result of this “new classroom environment” and the educational needs they present teacher educators must now seek different approaches to prepare prospective teachers to meet these needs because the traditional (e.g. coursework independent of fieldwork) approaches to teacher preparation are no longer effective in equipping teachers to address these issues. it has been my observation that some teacher educators are so far removed from the k-12 environment that what they teach sometimes does not reflect the realities their students face. additionally, there is the belief that “learning to teach is a two-step process of knowledge acquisition and application or transfer” (feiman-nemser & remillard, 1996, p. 79). the latter view infers a mutually exclusive relationship between the teacher educator, the prospective teacher and the classroom. in this approach to teacher preparation, the teacher educator provides the knowledge and the prospective teacher applies it. however, the teaching of theories or knowledge to prospective teachers and expecting that they will effectively apply them is an inadequate approach (stuart & thurlow, 2000;wideen, mayer-smith & moon, 1998; adams, shea, liston & deever, 1998) to teacher preparation. the assumption lying herein is that prospective teachers not only acquired the knowledge and theories in their program but the wherewithal to apply it in their classrooms. this, of course, might be true if the process of learning to teach were linear rather than dynamic; free of extraneous influences and circumstance rather than a complex mélange of variables this thought is best captured by britzman (1991): “learning to teach is not a mere matter of applying decontextualized skills or of mirroring predetermined images: it is a time when one’s past, present and future are set in dynamic tension. learning to teachlike teaching itselfis always the process of becoming: a time of formation and transformation, of scrutiny into 1 assistant professor, school of education, california state university, bakersfield, (661) 665-6807, jhughes4@csub.edu hughes, j.a. journal of scholarship and teaching, vol. 6, no. 1, august 2006. 111 what one is doing and who one can become…learning to teach is a social process of negotiation rather than an individual problem of behavior.” since preparation of teacher candidates is, at best, a complicated process, teacher educators must consider adopting new practices: as such, for teacher educators to better prepare prospective teachers three things must occur: examination of their teaching practices and “the process of learning to teach” (szabo, scott & yellin, 2002, p.1); utilization of field work to aid prospective teachers in their process of meaningful reflection and construction of practical knowledge (perry & power, 2004); and, finally, inculcating prospective teachers’ understanding of the relationship between theory and practice (szabo, scott & yellin, 2002). it is the consistent interconnection and persistent engagement in the above facets of learning-to-teach that show promise of more effectively preparing teachers. in this paper, i will discuss a field-based approach i implemented to examine my own teaching practices in linking theory with practice and regaining currency in the real world of public school. i approached this project with the tentative optimism that my experiences would assist me in preparing prospective teachers for their “multiple roles and [the] contextual complexities of life in schools” (knowles & cole, 1996, p. 648). ii. a theoretical framework the teaching of theory must be (or should be) inextricably linked to its application (brunner, 1997). when students are exposed to theoretical concepts for the first time, they must be introduced to these ideas in a manner to which they can best relate (brunner, 1997). brunner (1977) refers to this “as grasping the structure of a subject.” he further argues that “teaching specific topics or skills without making clear their context in the broader fundamental structure of a field of knowledge is uneconomical in several deep senses” (pg. 31), in that: “such teaching makes it exceedingly difficult for a student to generalize from what he has learned to what he will encounter later…the best way to create interest in a subject is to render it worth knowing, which means to make the knowledge gained usable in one’s thinking beyond the situation in which the learning have occurred. third, knowledge one has acquired without sufficient structure to tie it together is knowledge that is likely to be forgotten. an unconnected set of facts has a pitiably short half-life in memory.” similarly, theories cannot be taught in a vacuum; prospective teachers must understand the relationship between the ideas they are taught and the applications they will encounter. one way to develop this skill is to arrange for this connection to be made in the context of their “lived” realities. such learning-in-context will provide prospective teachers with the opportunity of questioning what they do and think (brookfield, 1995). it is during this process of inquiry, thinking about their practice, that teacher transformation occurs. another valuable theoretical approach which informs this process is the constructivist approach to learning, which derives its name and its power from the belief that knowledge is best constructed when the learner actively interacts with the environment and, hence, constructs meaning from that experience (hausfather, 2001). similarly, hall-quest asserts in the editorial foreword to dewey’s experience and education, that “sound educational experience involves, above all, continuity and interaction between the learner and what is learned” (dewey, 1938, hughes, j.a. journal of scholarship and teaching, vol. 6, no. 1, august 2006. 112 p.10). the end result of this nexus is the teacher’s ability to transfer teacher-knowledge to effective practice. in sum, the theories that drive this project are those of constructivism, the theory of integration, and the rigorous application of critical reflective thinking. iii. the role of fieldwork in teacher preparation field experiences are significant means through which to develop prospective teachers’ understanding of the why, what, and how of teaching and learning. however, simple placement of student teachers in the field does not automatically result in a valuable experience for the teacher candidate (zeichner, 1990). after all, not “all experiences are genuinely or equally educative” (dewey, 1938, p. 25). dewey (1938) asserts that “it is not enough to insist upon the necessity of experience, nor even of activity in experience [emphasis mine]. everything depends upon the quality of the experience which is had” (dewey, 1938, pg. 27). although, there is little doubt among teacher educators about the role of fieldwork in preparing better teachers, “there is persistent concern that such experiences do not reach their full potential value” (bowman & mccormick, 2000, p.256). several circumstances may account for this: traditional structures of student teaching (zeichner, 2002), which are often developed out of “convenience or tradition” (guyton & mcintyre, 1990, p 517) rather than innovative practices; limited resources to carry out field work (goodlad, 1990; darling hammond, 1999); the individualized nature of fieldwork (goodlad, 1994); the quality of the field placement (laboskey & richert, 2002); and a traditional approach to university supervision (bowman & mccormick, 2000). empirical evidence and current thinking suggest that many time-honored and time-worn field experience practices, such as those referenced earlier, need to be either refurbished or abandoned all together. one viable solution to teacher-educator lack of currency lies in the periodic return of teacher-educators to the public school environment with the intent of gaining practical experience. a return to the living laboratory of the k-12 classroom will allow teacher educators to test the theories and concepts they teach as well as to examine their own teaching practices while making pertinent and necessary revisions and adjustments in their practice. iv. approach from several years of informal conversations with teacher candidates during office hours, class discussions, brief encounters with them in the hallways, and reading their observation journals, i have concluded that for teacher preparation to be effective it must take place in the context in which it occurs, the school environment. over the course of several semesters, i utilized various approaches to field work, such as focused observation activities with required critical reflective inquiry of their observation and case study development and analysis of issues of interest to the prospective teacher. although these exploratory attempts at “teaching in context” yielded some satisfying results, i became increasingly restless with my methodology. searching for something new and innovative, i assumed the role of a 9th grade algebra-i teacher, sharing teaching responsibilities with the teacher of record. this pilot project was conducted at a high school located in a predominantly hispanic community in southern california. the high school population consisted of approximately 95% hispanic with asians, caucasians, and african americans constituting the remaining 5%. i selected this school because of its cultural location (a high percentage of minority students, hughes, j.a. journal of scholarship and teaching, vol. 6, no. 1, august 2006. 113 mexican and south-american, of low socio-economic status) as well as the current commitment between the university and the school district.2 four students fully participated in this pilot project. because the majority of my students were themselves classroom teachers (on emergency permit) or holding jobs outside of the field of education, the remainder of my college class members participated in a somewhat more limited way. the project full-participants observed my classroom experience for a minimum of two class periods twice weekly. in addition, a thirty to forty minute critical and reflective debriefing was conducted immediately after the observation. if time constraints prevented this from happening, the requisite discussion session took place in my university office later that same day. during these discussion sessions, participants were encouraged to speak freely about my teaching strategies and my interaction with the algebra students, as well as the general classroom atmosphere; they provided insights into the events of the period, explored various perspectives on the relationship between classroom practice and education theory, highlighted various concepts already covered in class discussions at the university, interpreted and analyzed particular incidents and offered suggestions for solutions; further, they discussed with me the reasons i had handled an issue or situation in the way that i had. in addition to these discussions, all participants maintained an observation journal in which they reflected on the various concerns and issues they encountered, some entries of which are noted below: it is important to note here that the 9th graders received a combination of before and/or after school tutoring; tutoring from the student teachers; exam review activities; and, homework to practice algebraic concepts. a. student #1 first reflective entry: how do i adjust piaget’s cognitive theories to meet the different learning stylesin my classroom? it does not seem as if it applies. especially, when i have to interpret the curriculum standards in order to teach it to my students who have such grave disparities in their learning levels. second reflective entry: i also find that learning the concept takes so much time and i don’t have the luxury to study it in a way that would help me put it into practice due to the dayto-day demands of my classroom environment. i know that the things that i learn in this course are important and informs my teaching in some way. but, i don’t see it, yet. i know that students are at different stages cognitively, intellectually, socially, and so forth. that’s evident! i see it demonstrated in my students everyday. so i don’t need a theorist to tell me this. my problem is how these seemingly relevant concepts, theories, help me motive, j--e. how does it help me teach my academically diverse students complex curriculum standards while being cognizant of their overall development – social, emotional, behavioral? third reflective entry: i find myself not wanting to learn these concepts because i don’t know yet howto readily apply it to my class. i hear my peers speak of how they have applied cooperative learning in their classroom and how successful it was. but, i am afraid i don’t share those experiences. dr. hughes – struggled to illustrate this concept in class last night and there were no videos except from her high school class regarding this issue. so i was really stuck. 2 human subjects clearance was not needed for this research. hughes, j.a. journal of scholarship and teaching, vol. 6, no. 1, august 2006. 114 b. student #2 reflective entry: i think i got it! the theory of motivation. i so want my students to want to learn. then it occurred to me that (as i watched dr. hughes struggle with how best to motivate a----a and some of the other low motivated students) that she missed something important – what interest them). she talked about it in class but she didn’t do it. i don’t know why! i will ask her later. reflective entry – a week later: this week dr. hughes planned to review for the upcoming exam. s----n and i have worked hard all weekend putting together a jeopardy algebra game. we weren’t sure if it would work but dr. hughes let us try anyway. it worked well! i was so relieved. the students were excited and very much into the game. i thought it was because the method was fun, innovative and so forth. i was worried about no real reward (e.g. no homework, etc.) except for the points they received for the correct answers in each category. much to our surprise (dr. hughes, too). the students were really into the activity. they didn’t seem to mind that all they would have at the end of the period was just points for first, second, or third place. i am not sure if the students learned much in preparation for the exam. but, they indicated that they liked the game, that it was fun. dr. hughes informed us the next week that a majority of the students had passed the exam… we had hope that the game would motivate the students to go home and study as a result of the positive feelings of giving the right answer and being praised by their team-mates and teacher. these sessions (as well as several days of my teaching) were videotaped; these tapes were then used in my university class as a teaching tool to enhance the curriculum and to demonstrate best or worst practices. this procedure provided a rich foundation for discussion by both full and limited participants. the limited participants benefited from watching the video of my teaching as well as listening to the discussions between me and their student colleagues. the retrospective discussions allowed all students the opportunity to form sound and valid arguments, to make explicit their practical knowledge, and to apply this new knowledge to current educational issues. throughout the entire experience, i provided extensive feedback to guide interpretations and encourage critical reflective inquiry. it is important to note here that the 9th graders received a combination of before and/or after school tutoring; tutoring from the student teachers; exam review activities; and, homework to practice algebraic concepts. v. reflections i found from my experience that when both the teacher-educator and prospective teachers are actively involved in both the college class and the k-12 environment, the ability for everyone to teach and learn simultaneously was enhanced. furthermore, this process made it possible for me and my prospective teachers to identify and examine the convergence of theory and practice. the goal of exposing students to the process of teaching and learning through such organized field experience was met. i fervently believe that a process similar to that which i have described has the potential to augment the quality and merits of the field observation requirement. as such it contributes immeasurably to the overall ability of the students to teach effectively in the k-12 environment. a summative entry from my own journal follows: hughes, j.a. journal of scholarship and teaching, vol. 6, no. 1, august 2006. 115 what i learned as a teacher educator. as an educator i learned that it wasn’t just about teaching the theories of motivation, piaget’s theories of cognitive development, or vygostky’s zone of proximal development in my educational psychology course. rather it was about teaching motivation as it came to life in my algebra class instead of in abstract form. i was challenged with how best to motivate my 9th grade students. i tried no homework if they completed their worksheet, etc. but, i was faced with the fact that most of my students did not understand the concepts. so with the help of the collaborating teacher we divided the class into several smaller groups and assigned them work that met their skill and comprehension level. this approach worked very well. the next semester i used the experience to exemplify how piaget’s theory of cognitive development might inform high school teaching. my college students remarked how very beneficial this was. of course, i did not do away with teaching the stages of the theory of piaget’s cognitive development, etc. what this experience did for me was to transform how i teach and how i constructed my course curriculum. theoretical concepts were paired with some real life classroom experience i had encountered. my text selection was based on how well the authors presented complex theoretical concepts (that is their approach to illustrating these complex theories in the hope of increasing comprehension, and possible later transfer of this understanding). as a result, i looked for the level of language used to introduce complex concepts, case studies, activities that would facilitate understanding and skill acquisition. vi. suggestions for change the serious nature of the process of preparing students to meet the demands of their profession requires a critical examination of the professional requirements. one such requirement is the familiarity with the changing nature of the profession. meeting this requirement will likely provide insights on how to interpret and integrate professional standards and expectations into program course work and requirements. educators can meet this requirement in one of two ways. the first is, a return to the environment of professional practice for a semester or a year-long reintroduction to the field. the educator may serve as a social worker, counselor, or teacher performing duties similar to what their current students would do once in the field. the second approach to gaining familiarity with the current practices in one’s field is to allow former students to return to the college classroom to share their lived experiences and current practices as it relates to theory. the above approaches can be combined or employed separately. i prefer the approach wherein the educator returns to the field. implementation of one or both of the above approaches or a variation will no doubt demonstrate how serious we are in bridging the divide between what our students are learning in their college classrooms and what they do in their jobs. as a result of my experiences, i am now even more convinced that faculty currency is critical to the development of well-grounded students who can effectively respond to the demands and challenges of their profession. hughes, j.a. journal of scholarship and teaching, vol. 6, no. 1, august 2006. 116 references adams, n., shea, c., liston, d., & deever, b. (1998). learning to teach: a critical approach to field experiences. new jersey: lawrence erlbaum. baker, b. (1997). “anthropology and teacher preparation: some possibilities and precautions.” queensland journal of educational research, 13(2), 41-58. bullough, r. v., jr. young, j., erickson, l., birrell, j.r., clark, d.c.’ egan, m.w., berrie, c. f., hales, v. & smith, g. c., (2002). “rethinking field experiences partnership teaching versus single-placement teaching.” journal of teacher education, 53(1), 68-80. bowman, c., & mccormick, s. (2000). “comparison of peer coaching versus traditional supervision effects.” journal of educational research, 93(4), 256-261. britzman, d. (1991). practice makes practice. albany, ny: state university of new york press. brookfield, s. d. (1995). becoming a critically reflective teacher. san francisco: jossey-bass. brunner, j. (1977). the process of education. massachusetts: harvard university press darling-hammond, l. (1999). “educating teachers for the next century: rethinking practice and policy.” in g. griffin (ed.), the education of teachers. chicago: university of chicago press. dewey, j. (1933). how we think. new york: houghton mifflin. dewey, j. (1938). experience and education. new york: macmillan. feiman-nemser, s. & remillard, j. (1996). “perspectives on learning to teach.” in f. murray (ed.), the teacher educator’s handbook (pp.63-91). san francisco: jossey-bass. goodlad, j. (1990). teachers for our nation’s schools. san francisco: jossey-bass. goodlad, j. (1994). educational renewal better teachers, better schools. san francisco: josseybass. guyton, e., & mcintyre, d. j. (1990). “student teaching and school experiences.” in w.r. houston (ed.), handbook of research on teacher education. new york: macmillan. hausfather, s. (2001). where’s the content? “the role of content in constructivist teacher education.” educational horizons, 80(1), 15-19. knowles, j. & cole, a. (1996). “developing practice through field experiences. in f. murray (ed.), the teacher educator’s handbook (pp. 648-688) san francisco: jossey-bass. laboskey, v.k., & richert, a. e. (2002).” identifying good student teaching placements: a programmatic perspective.” teacher education quarterly, 29(2), 7-34. hughes, j.a. journal of scholarship and teaching, vol. 6, no. 1, august 2006. 117 perry, c. m. & power, b. m. (2004). “finding the truths in teacher preparation field experiences.” teacher education quarterly, 31(2), 125 –136. stuart, c., & thurlow, d. (2000). “making it their own: preservice teachers’ experiences, beliefs and classroom practices.” journal of teaching education, 51(2), 113-121. wideen, m., mayer-smith, j. & moon, b. (1998). “a critical analysis of the research on learning to teach: making the case for an ecological perspective on inquiry.” review of educational research, 68(2), 130-178. zeichner, k. (1990). “changing directions in the practicum: looking ahead to the 1990s.” journal of education for teaching, 16(2), 105-132. zeichner, k. (2002). “beyond traditional structures of student teaching.” teacher education quarterly, 29(2), 59-64. 503 service temporarily unavailable 503 service temporarily unavailable nginx/1.14.1 v11n3lewis journal of the scholarship of teaching and learning, vol. 11, no. 3, august 2011, pp. 86 – 106.   competence assessment integrating reflective practice in a professional psychology program deborah lewis1, tom virden, philinda smith hutchings, ruchi bhargava abstract: the midwestern university clinical psychology program – glendale campus (mwu) created a comprehensive assessment method in psychology (camp) comprised of 35 different “tasks” of authentic work products representing a variety of assessment techniques based on pedagogical theory. each task assesses one or more components of one of the program’s five identified competence areas. tasks are submitted at multiple points, increasing in complexity, during the student’s tenure in graduate school. camp includes an innovative qualifying exam (qe) which formally evaluates a student’s ability to self-reflect and to accurately self-assess. in the qe, students engage in a process of reflection and substantive dialogue with a panel of two faculty members about their camp work products and describe their development, understanding of the context, and purpose of their education. the goal of the camp was not only to achieve and measure student competence, but to create an environment where students and faculty participate in ongoing reflection and even, aspire to artistry. mwu sought expert feedback to establish construct validity of the camp and qe processes in the form of a survey. the authors believe the method has relevance for graduate training in many disciplines, particularly those leading to professional practice degrees. keywords: reflection, self-regulation, self-assessment, competence, assessment one hundred years ago, john dewey published “how we think.” in this book dewey describes reflection as including, “a) a state of perplexity, hesitation, doubt; and b) an act of searching or investigation directed toward bringing to light further facts which serve to corroborate or to nullify the suggested belief.” (dewey, 1910. p.9) the authors of the current paper began a new doctoral program in professional psychology in 2007. we set out to find superlative methods for developing and evaluating students in their journey to become professionals. we extensively reviewed information on competence assessment and educational development in our field and reviewed educational theory outside our discipline. we found the work of john dewey (1910), the work on self-reflection by donald schön (1987) and the 1985 book “reflection: turning experience into learning,” edited by david boud, rosemary keogh and david walker, particularly helpful. we came to the conclusion that reflection is the necessary element for all growth and development. in educational programs, imparting knowledge and developing skills are necessary, but not sufficient, for what we wanted to accomplish. we wanted to create an educational environment that enabled students, not only to develop minimum competence in identified areas necessary for our profession, but to aim for what schön (1987) dubbed artistry. schön explained there are some people in every profession who become truly outstanding                                                                                                                           1  clinical psychology, college of health sciences, midwestern university, 19289 n. 59th avenue, glendale, az 85308, dlewis@midwestern.edu lewis, d., virden, t., hutchings, p. s., and bhargava, r.   journal of the scholarship of teaching and learning, vol. 11, no. 3, august 2011. www.iupui.edu/~josotl 87 practitioners. these practitioners are not described as having more professional knowledge than peers, but are described as wise, talented, and intuitive, more aptly, artists. we wanted to help our students, and ourselves, seek wisdom and maturity in a context where reflection is integrated, planned and evaluated. this paper is an articulation of the process undertaken by authors and a compilation of conclusions. we believe it has relevance for graduate programs in many content areas, particularly programs that lead to professional practice. finding better ways to assess student competence is a focus of many professional programs, but innovation can be disconcerting. the appearance of legitimacy and tradition is important for programs trying to satisfy accrediting bodies, even when the familiar methods have not been proven to accurately assess competence or even acceptably measure an outcome. established organizations, including academic institutions, often change slowly, even amid dissatisfaction with existing procedures (shelleyann & dixon, 2009). faculty may hesitate to veer from customary paths, because initiating creative methods often startles a system and may have unintended consequences. planned change requires faculty to reflect on competencies needed by students and by themselves, the nature of learning and education, and logistics of implementation. these processes require insight on the part of faculty, an open recognition of limits, and a commitment to create a milieu that supports ongoing development. schön (1987) proposed that a hierarchy exists in the prestige of knowledge in professional schools. those who teach basic scientific knowledge have the most prestige, decreasing for those who teach applied science and further diminishing for those who teach the technical or clinical skills of day-to-day practice. this prestige imbalance poses challenges for professional programs who seek to train practitioners. schön (1987) argued against the assumption that the acquisition of more basic knowledge is what leads to competence. schön (1987) explained that the prevailing relationship between professional knowledge and practice competence (which still exists today) needs to be turned upside down. the high bar for professional competence is artistry, and we need to carefully examine the path to achieve it. schön (1987) argued artistry is teachable and not just for the lucky. schön (1987) described an artist as an outstanding professional, who, faced with an unusual circumstance, novel situation or ambiguous area of practice, goes beyond the basic knowledge, technical and applied skills, and learned values and becomes a creative, innovative problem solver. this ability is based on knowledge, skills, attitude/values and experience, but also moves beyond them. artistry requires ability, when faced with something unexpected or not-yetlearned, to think about what one is doing as one is doing it (reflection-in-action), and to be able to create solutions when there is no clear right answer. the idea of reaching beyond the knowledge, skills and attitudes/values typically taught may be a long term goal, but the first step is for a program to create necessary frameworks for the development of artistry and commit to the pursuit. this paper describes an attempt to implement the best methods of assessment of professional competence and to integrate planned self-reflection. the authors chose to assess competence in knowledge, technical skill, attitude, ability to reflect and the development of artistry by focusing on assessment using multiple types of knowing and learning. the authors present a viable method of comprehensive assessment of competence, relying on pertinent pedagogical theory, and based on literature that includes: outcome measurement, evaluation of competence and portfolios comprised of authentic assessment. the authors present a qualifying examination process, proposing evaluation of a student’s ability to lewis, d., virden, t., hutchings, p. s., and bhargava, r.   journal of the scholarship of teaching and learning, vol. 11, no. 3, august 2011. www.iupui.edu/~josotl 88 self-reflect and self-regulate as the capstone event for advancement to doctoral candidacy. this is combined into an innovative overall approach to competence assessment in a psychology doctoral program. the authors have dubbed the method the comprehensive assessment method in psychology (camp). the following is a case study of a system of outcome measurement and evaluation of self-reflective practice. camp is a compilation of thirty-five tasks performed during the span of the graduate program (see appendix i for brief description), and includes a series of three self-reflection/selfregulation events. the second of these events is the qualifying examination (qe), used to determine advancement to doctoral candidacy (see appendix ii for brief descriptions). each of the camp tasks is directly linked to at least one of the program’s five competency goals. the tasks become more complex and integrated as the student progresses through the program, and tasks become linked to multiple potential competencies. the authors wanted to establish construct validity on the camp and qe, and prepared a survey of both, enlisting experts to review and comment, and incorporated the feedback. the authors have additional plans to compile outcomes on the camp and qe as students graduate and the program matures. i. types of learning. acquisition of knowledge can be described as: declarative (verbal learning), procedural (skill learning), conceptual (concept attainment), analogical (one-trial learning) and/or logical (problem solving) (farnham-diggory, 1994). farnham-diggory (1994) also describes three types of instructional paradigms that distinguish novice from expert level (defining “expert” as the standard or level of competence a program has set): behavior, development, and apprenticeship. in the behavior paradigm, novices become experts by accumulating some factor (e.g. speed, knowledge). in the developmental paradigm, novices and experts are distinguished based on the complexity of their personal theories, and their experience. in the apprenticeship model, novices become experts via acculturation into the world of the expert. farnham-diggory (1994) stated that these models are mutually exclusive. declarative knowledge can be memorized or reproduced, but not necessarily applied to situations. multiple choice exams are a reliable way to assess this accumulation of facts. procedural knowledge is the ability to apply information to situations; knowledge that can be demonstrated, but is not well-suited to multiple choice examinations. examination of procedural knowledge often occurs via work samples, either an unstructured portfolio collection or intensive exams of procedures (e.g. a medical school student’s ability to perform a history and physical on a standardized patient an actor trained to respond with a specific diagnosis). the program’s initial purpose was to create competent professionals; for student’s to achieve a minimum standard across all competence areas. to achieve this first step, the authors adopted the concept of acculturation described by farnham-diggory (1994) in the apprenticeship model. the authors believe that most graduate students, particularly in programs leading to professional practice, need to integrate their acquisition of declarative knowledge by applying it. students need to demonstrate procedural knowledge and use it effectively. competent professionals need conceptual knowledge to be able to fit new learning quickly into acquired cognitive schemes and logical knowledge to understand what is connected to what and what leads to what. lewis, d., virden, t., hutchings, p. s., and bhargava, r.   journal of the scholarship of teaching and learning, vol. 11, no. 3, august 2011. www.iupui.edu/~josotl 89 in the graduate education process, development can be described in knowledge, skills and/or attitudes or values (ksa) (kenkel & peterson, 2009). the more measures we use to evaluate a student’s array of ksa, the more assurance of validity we obtain in our estimation of competence, and the more reliability we can assume in our measures of multiple types of knowing and learning and gauged competence. the acquisition of learning process can be a measure of self-reflection, if specifically evaluated. the learning procedures may incorporate novel, ambiguous or indeterminate situations in order for students to have the situations necessary to develop artistry. the camp is an array of measurement procedures sampling student development across time and compiling evaluated work products. it was designed after reviewing literature on assessment methods and incorporating an array of types of learning. the result is that camp incorporates a variety of tasks that utilize different techniques for measurement. some camp tasks are traditional portfolio pieces allowing the student to choose a “best work” in a particular area. most camp tasks are faculty guided projects that assess knowledge, skill and/or attitude (ksa) in the competence area or areas being measured. the qualifying exam assesses a student’s ability to evaluate his or her own development and performance in each of the programs competency areas by engaging in a planned, evaluated dialogue about his or her assessment of completed camp work products. schön’s (1987) work is theoretically broad. one of his practical suggestions about the journey for artistry is to have students engage in what he called the reflective practicum. the reflective practicum is based on an apprenticeship model, and provides students opportunities to reflect upon their application of knowledge into practice. pearson and smith (1985) discuss the importance of engaging students in dialogue and “debriefing” about practical experience to increase the opportunity for learning. the concept of practicum is well established in many professional disciplines, including the psychology program where the authors teach. during the supervision process, students are often prompted to reflect upon their application of knowledge to practice, developing conceptualizations to explain the patient’s development of symptoms and integrating their knowledge of theory with the outcome of their practice (fouad et al, 2009; stoltenberg, mcneil, & delworth, 1998). the authors needed to establish a logical and pragmatic process of collecting student work samples to support the reflection. toward this end the authors researched portfolio projects. the concept of portfolio, though redefined and expanded for the camp, served as a platform for the system. with an attempt to correct for criticisms brought to light in the literature the authors reviewed portfolio systems comprised of authentic assessment. in the next section of this paper we review portfolio methods. this is followed by a discussion of types of learning, comprehensive and qualifying exams, and reflective practice that make up the theoretical framework for the camp and qe events. ii. portfolios. portfolios include a collection of authentic assessment pieces, actual work products that estimate what students will be required to produce in a profession. the process of choosing one’s “best work” is an act of self-reflection and sheds light on the student’s understanding of professional standards. a portfolio provides an opportunity for educators to objectively evaluate both the task and the ability of the student to self-assess. collecting authentic assessment pieces is a sensible lewis, d., virden, t., hutchings, p. s., and bhargava, r.   journal of the scholarship of teaching and learning, vol. 11, no. 3, august 2011. www.iupui.edu/~josotl 90 approach for evaluation of competence and self-reflection. it also is a useful tool for remediation and advisement (lombardi, 2008). there are several critiques of portfolios (mcguire, lay, & peters, 2009; lombardi, 2008; tisani, 2008; cook-benjamin, 2003) including that portfolios; • are a student’s compilation of their “best work” and may not be representative of their typical abilities, • are summative, a compilation of many projects over a long period of time, and lose the ability to evaluate individual outcomes because feedback is only at the end, not ongoing. reflective practice requires continuous feedback to students, • are large, complex and lack structure. reflective thinking needs to occur in an orderly fashion, • lack of structure and guidance for students leads to decreased compliance due to the overwhelming nature of the project, • create logistical problems such as compilation, storage and electronic technology, • require a large amount of work for students, faculty and staff, • employ vague scoring systems. in order to capture and utilize the best of portfolio projects while addressing the criticisms, the following procedures were implemented to address each of those concerns; • camp utilizes both student-chosen and faculty guided tasks/ pieces, • camp is broken down into 35 tasks compiled during the student’s entire tenure in the program to provide structure and continuous, specific feedback, • specific explanations of the practical importance of each task are provided to students. each task’s rubric clarifies how it leads to competence in a stated professional goal. this understanding of a task’s importance increases compliance and motivation (marzano & kendall, 2007), • the entire camp is broken down into a developmentally appropriate sequence. the tasks become more complex and integrated over time, • a holistic scoring method is used consistently across all tasks and a staged rubric approach is used, allowing for on-going and increasingly complex feedback to students that engages them in cooperative learning (see appendix iii), • self-reflective and self-regulation events are incorporated to allow students to demonstrate a commitment to lifelong learning and ownership of one’s development. self-reflection and self-assessment events occur at major points in the program; the qualifying examination (qe) and pre-qe and post-qe tasks, • the qe requires students to review and choose their best work and their least effective work compiled over their tenure in the program to promote self-reflection and to help them identify specific goals for their development. iii. integrating authentic assessment and self-reflection. a. process and products. there is a new emphasis in education on “process” as well as “product” (lombardi, 2008). examples of educational products are; papers, exams or professional reports by students. process lewis, d., virden, t., hutchings, p. s., and bhargava, r.   journal of the scholarship of teaching and learning, vol. 11, no. 3, august 2011. www.iupui.edu/~josotl 91 refers to understanding the context, course and means of one’s education. main (1985) discusses the importance of learning how to learn; and emphasizes reflection on the process of learning. the series of camp tasks are “products” and the series of three self-reflection/selfregulation events, (qe, pre-qe, post-qe), are “process” events. during the process events, students review camp “products,” and examine the context of their education and their own development. during the qe events, the ability to accurately self-reflect, self-assess and selfregulate are specifically evaluated. these novel and indeterminate process events were developed to require problem solving, reflection and to utilization of multiple types of learning and knowledge. this embodiment of ongoing reflection and process is a foundation for artistry. b. taxonomy for education. marzano and kendall (2007) propose a revised taxonomy for education. they describe six levels of learning processes, each more integrated and complex: 1) retrieval, 2) comprehension, 3) analysis, 4) knowledge utilization, 5) metacognitive, and 6) self-system. the first four levels of learning processes are more familiar. retrieval is transferring knowledge to conscious awareness. comprehension is translating knowledge into a form for memory storage. in analysis, we elaborate on the knowledge as comprehended, and in knowledge utilization, the individuals employs the knowledge they wish to complete a task (marzona & kendall, 2007). these first four types of processes are relied upon in most academic coursework and in many authentic assessment pieces. the last two levels of learning processes are more complex, and require reflective practice. according to marzona and kendall (2007), in metacognitive processes, students establish a goal and create a plan for the goal, monitor execution of the goal, determine their knowledge mastery, or in the case of this professional graduate program: the extent of their ksa, and the extent to which they are accurate in their assessment of the development of their ksa in the program. in self-system processes, students identify how important the ksa are to them. then students identify beliefs about their ability to improve competence or understand the ksa and the reasoning underlying this perception (thinking about their thinking), identify emotional responses to the ksa and reasons for these responses, identify their overall level of motivation to improve competence or understand ksas and reasons for this level of motivation (marzano & kendall, 2007). in addition, practice-oriented professions involve a type of thinking erlandson and beach (2008) described as situational thinking (thinking that concerns situational practice) that defines being a professional. while examining motivation, marzano and kendall (2007) explain that most motivated students perceive the acquisition of ksas as important, have the necessary ability, power and resources to increase their competence, and have a positive emotional response to the acquisition of ksas. c. reflective practice. saltiel (2007) believes that reflective practice is a crucial component in education, but states it is often employed uncritically and without appreciation of its limitations. methods to incorporate self-reflection into an evaluated educatory experience are not well established. lewis, d., virden, t., hutchings, p. s., and bhargava, r.   journal of the scholarship of teaching and learning, vol. 11, no. 3, august 2011. www.iupui.edu/~josotl 92 according to saltiel (2007), the concept of reflective practice has been so enthusiastically embraced in education it is, “little short of sacrilege to question it.” (pp.2). saltiel (2007) supports schön’s (1987) theory that reflective practice is the counterpoint to the technical rationality of basic science evidence-based practice. saltiel (2007) believes that technical rationality has been more readily adopted by practitioners and educators because evidence-based practice seeks “order and certainty in a procedural world.” reflective practice engages with ambiguity and with actual experiences of practitioners, emphasizing skill and artistry (saltiel, 2007). for professional education, both need to be developed in parallel and given equal weight. dewey published his influential ideas about education and training thought in 1910 in how we think. however, erlandson and beach (2008) trace modern query on reflective practice to schön’s publication of the reflective practitioner in 1983. schön’s paradigm of reflective practice and the concept of reflection-in-action have been central in the education literature and are particularly important in the education of reflective practitioners (erlandson & beach, 2008). boud, keogh, and walker (1985) define reflective process as a conscious process by which teachers and learners organize learning activities. the model describes the outcome of reflection, “which may be a personal synthesis or integration of appropriation of knowledge, the validation of personal knowledge, a new affective state, or the decision to engage in a further activity” (p.20). the concept of collecting educational products (that allow maximum understanding of the competence to be achieved) and then having conscious reflection as a specific learning activity fits with boud, keogh, and walker’s (1985) model to increase the conscious reflective process throughout the educational experience and maximize learning. candy, harri-augstein, and thomas (1985) discuss the importance of examining our own learning in a systematic manner. they posit that as learners, we need to examine our learning to understand our own assumptions and constructs, and to precisely identify our learning strategies. all students have strengths and weaknesses. competencies in the areas identified as important by a program need to be assessed regularly and thresholds for achievement set. areas of strength need to be nurtured and areas of difficulty remediated, if possible. the authors agreed with the premise that learners who accurately identify their strengths and weaknesses were much more likely to be motivated students, become competent professionals, seek development in areas in need of growth, strengthen areas of talent, be cooperative learners during graduate training and work within their areas of competence, seeking guidance when necessary. in short, we are developing self-regulated learners (zimmerman, 2008) within the broader goal of artistry. self-regulation is an expansive concept that includes self-reflective practice. according to zimmerman (2008), self-regulated learners assess their own behavior in terms of their goals and are able to adequately reflect on their development. this process enhances student satisfaction, motivation to improve, optimism, and the likelihood of becoming a life-long learner. self-regulation involves knowledge acquisition, self-awareness, self-motivation, behavioral skill to implement knowledge appropriately, and the ability to self-correct when necessary. selfregulation is complex and not innate, but it can be learned. according to zimmerman (2004; 2008), the components of a self-regulating student are evident when he/she can; set specific goals, adopt strategies for attaining goals, monitor progress, restructure his/her context (social and environmental) to make it compatible with goals, manage time, evaluate methods, attribute causation to results, and flexibly adapt future methods to improve development of competence. as with all abilities, people vary widely in their talent for and experience with selfreflection. reflection comes very easily to some, with some effort for most, and is exceedingly difficult for others, as evident in borderline personality disorder (bennett, pollock, & ryle lewis, d., virden, t., hutchings, p. s., and bhargava, r.   journal of the scholarship of teaching and learning, vol. 11, no. 3, august 2011. www.iupui.edu/~josotl 93 2005). each profession determines the standards and qualities required to practice. educational programs and practice regulatory boards serve as gatekeepers of that profession. educational programs have the challenging job of determining admissions and competence standards. academic failure from lack of knowledge is relatively easy to identify. skills tend to require more time and consideration to evaluate than knowledge, but the most difficult processes to evaluate have been attitude and ability to reflect. articulating specific professional standards and competencies that deal with values (e.g. cultural understanding or ethical reasoning) can be a hard task, and therefore more difficult to identify as a professional success or failure. dimaggio, vanheule, lysaker, carcione, and nicolò (2009) state that self-reflection is key in healthy human adaptation. they suggest there are four forms of deficits in self-reflection; difficulty in sense of ownership of one’s own thoughts or actions, lack of emotional awareness, difficulty distinguishing between fantasy and reality, and trouble integrating a range of different views of oneself and others. students, particularly students seeking a professional practice degree, who have a deficit in ability to self-reflect, who cannot evaluate their own performance, who are incapable of benefiting from the milieu, or who cannot incorporate supervisor feedback, will not be able to appropriately mature and develop. d. comprehensive and qualifying examinations. schafer and giblin (2008) explain that doctoral comprehensive examinations have changed dramatically in recent decades, and vary greatly from program to program. schafer and giblin (2008) state that comprehensive exams are assumed to have implicit objectives to: evaluate mastery and integration of knowledge, measure skills, serve a gatekeeping function, and serve as a rite of passage. however, schafer and giblin (2008) found little systematic discussion concerning the proper role, objectives and approaches for doctoral comprehensive exam processes in various disciplines. they found an increasing level of flexibility in comprehensive exam structures and considerable variation in timing, format and administration. they conclude that this variability may be a healthy indicator that programs are tying exams to their unique objectives. there are few established comprehensive exams measuring or incorporating reflective practice. v. description of the camp and qe project. camp is comprised of 35 fundamental assignments or categories of assignment (e.g. supervisor evaluations). each submission lists the competence area(s) it is intended to demonstrate. the objective of the camp and qe was to create a method that is: comprehensive (measuring a wide range of competencies identified for professional practice); developmental (was sampled during several points of training and allowed for early remediation); theoretically sound and grounded in educational principles, a teaching tool that promoted awareness, reflection and artistry; created data useful for outcome measurement; reflective of the goals and philosophy of the program; utilized current best practices in evaluation; and creative, flexible and practical. camp requires: a sampling method based on the program’s identified competence areas; a practical, efficient method of compilation and tracking; rubrics for each applicable task; and a scoring system. the authors established a theoretical relationship between each competence area and assessment method, as suggested by klenowski, (2002) and leigh, et. al. (2007). the authors were able to integrate thirteen, and part of a fourteenth, of the fifteen categories of lewis, d., virden, t., hutchings, p. s., and bhargava, r.   journal of the scholarship of teaching and learning, vol. 11, no. 3, august 2011. www.iupui.edu/~josotl 94 assessment method described in the “competency assessment toolkit” for professional psychology (kaslow, et. al., 2009) (see appendix iv for description). the program delineated five competency areas; 1) research evaluation/foundations of psychological science, 2) professionalism (which includes ethics, diversity and advocacy), 3) diagnostics/assessment, 4) intervention, and 5) relationship and communication (interpersonal skills and professional writing). in addition to these core competence areas, the program has a healthcare emphasis where students are taught how to apply the five core areas in interdisciplinary healthcare settings (see appendix v for description). camp is a compilation of work products (e.g. reports, videotapes, projects, and activities) sampled throughout a student’s program to demonstrate development in each of the competency areas. these products are prepared in courses, field training or as reflection tasks (e.g. service project). camp includes; an extensive portfolio of required submissions in specific areas (faculty guided submissions), student choice submissions in specific areas (e.g. favorite literature review paper), the qualifying examination (qe) which is an oral defense by the student of their professional strengths and weaknesses using camp submissions as a guide (selfreflective practice and self-regulation) and an analysis of their understanding of the program and professional training model, the doctoral scholarly project, and clinical training materials (e.g. practicum supervisor evaluations). these types of assessment methods prompt faculty to focus on content, process, and context rather than grades to evaluate student development. mwu chose a six-point holistic scoring system similar to that reviewed in elbow (2003) for all camp submissions. the system is based on expected developmental level. this method changes the tradition of using single numbers to rank complex performances. a score of “3” is the anchor for developmentally expected level. the range for each submission is a “1: novice/beginning level” (where the student demonstrates notable difficulty in the developmentally expected ksas evaluated) to “6: sophisticated/advanced level (where the student demonstrates well above developmentally expected level).” using such a scale typically increases inter-rater reliability (elbow, 2003) (see appendix iii for description). the best assessment methods are teaching tools in themselves (kösters & ritzen, 2003). camp introduces students to competencies in the profession, aids in self-reflection, helps in designing and documenting a student’s individual program, assists students in making choices and setting goals, and in managing their learning (self-regulation). student reflection on the competencies is useful if the competencies are emphasized and transparent throughout the curriculum. the developmental nature of the camp gives students time to recognize the sequential, cumulative, and interactive nature of competencies in the larger context of their education and profession. traditional forms of assessment (e.g. discrete multiple choice competency exams) do not integrate development explicitly into the evaluative process nor tell us how students improve and develop (hessler & kuntz, 2003). camp provides an opportunity for faculty to monitor student development through actual work products and for students to monitor their own development. camp incorporates guided submissions (tasks that are specifically required), and “portfolio” pieces where students choose their favorite work in designated categories. the camp submissions begin with basic abilities early in the program, and ends with advanced and integrated abilities. this allows faculty to determine areas of student strength and challenge early in a student’s program to focuses on the development of specific professional competencies, and to monitor program outcomes. lewis, d., virden, t., hutchings, p. s., and bhargava, r.   journal of the scholarship of teaching and learning, vol. 11, no. 3, august 2011. www.iupui.edu/~josotl 95 each submission must be reviewed, accepted and is a prerequisite for the next field training, course or submission. if a student has a developmental weakness, it is immediately addressed, and the student can re-submit the camp assignment after remediation. multiple attempts are allowed, though more than one remediation attempt may significantly delay a student’s program and may even prevent a student from completing in the required time period for graduation. the qe was designed for faculty to evaluate a student’s self-reflection and self-regulation capacity. students review the goals, objectives and competencies of the program, their camp materials, and reveal their self-assessment. they are not evaluated on camp tasks (already evaluated products), but rather on the accuracy of the observation of their own developmental needs and progress (process) considering program goals. does the student believe they are as good, or as in need of assistance, as the faculty believe them to be in each competence area? the program’s goal was to develop habits of reflective practice and self-regulation. the qe takes place after the second year of full-time study and field training. a “pre-qe” event occurs at the end of the first year and; helps teach program expectations, broader theoretical goals (e.g. self-reflection), and serves a self-regulatory function to help maintain student motivation. it is a self-reflection paper based on specific questions (e.g. what makes your most effective or least effective camp submission?), and is evaluated on the 6-point rating scale. a “post-qe” task occurs at the end of the third year, before students embark on a year of full-time field training. during this project, students develop goals for their final year and entry into professional practice, continuing the reflective process and moving closer to areas where they may have developed are developing or could develop artistry. they are given the questions to answer (e.g. what is your plan for continued development after graduation and into practice?), and it is evaluated on the 6-point holistic rating scale. the qe (along with pre-qe and post-qe events) are integral parts of camp. the qe determines whether students advance to doctoral candidacy. several types of qualifying exam procedures were researched, including traditional multiple-choice knowledge based exams, and skill-based exams. the mwu faculty created the qe events to add self-reflective practice, selfassessment and self-regulation as an integral assessment component. the mwu faculty believe that competence assessment measures for the traditional types of knowledge and skill already existed in the curriculum. the literature reviewed explained the importance of self-reflective practice and attitude, but few publications revealed formats for evaluating these abilities. in the qe manual and preparation materials, students are given background materials on the context of their education. mwu created the qe to be an event where the students engage deeply with their own development, competence literature, history of the profession, and information on the program model. the qe events assess these abilities by requiring students to formally present themselves to a panel of faculty and engage in substantive dialogue. students are given the questions in advance. then, using their camp materials as a guide, students assess their strengths and weaknesses and create developmental plans across each of the program’s competence areas (demonstrating accurate self-assessment, active participation in their learning process and practice for life-long learning). students have spent time in field training before the qe and incorporate their professional experiences (including experience in reflective practicum) as well. the qe is designed to specifically practice and assess conscious reflection, develop self-regulation and provide framework for a trajectory toward artistry, with a novel educational exercise that lewis, d., virden, t., hutchings, p. s., and bhargava, r.   journal of the scholarship of teaching and learning, vol. 11, no. 3, august 2011. www.iupui.edu/~josotl 96 requires introspection and creative solutions for achieving future goals. students are holistically scored on knowledge of the context of their training, ability to self-reflect, accuracy in selfassessment, professional attitude and developmental planning skill. survey. in an effort to measure the construct validity of the method, mwu surveyed experts in doctoral psychology education and training, and experienced in program accreditation. mwu sought to establish construct validity of the camp and qe. a survey was created and external experts were identified (see appendix vi for description of survey results). all experts responding to the survey indicated that they believed that a student passing the qe would sufficiently demonstrate self-reflective practices at a level appropriate for advancement to doctoral candidacy (m = 1.80, sd = 0.447) on a 4 points scale from 1 (strongly agree) to 4 (strongly disagree). the majority (80%) felt that a student passing the qe would understand the competence areas required in professional psychology (m = 2.0, sd = 1.225), and most (60%) would consider adopting the qe or a similar assessment in their own programs if practical (m = 2.40, sd = 1.140). four comments provided with regard to the qe recommended greater depth in measuring the competencies, reflected concerns with regard to measuring self-reflection, and regarded the rubric as a strength of the qe. viii. discussion. as educators and trainers of graduate students in a program leading to a professional practice degree, the authors sought better ways to achieve excellent outcomes, even of ambiguous abilities, such as reflective practice and artistry. the authors suggest that the act of reflection is required for all development; ethical values and actions, multi-cultural awareness and attitude, critical thinking, and decision-making. this entire project became a way to articulate the need for planned and integrated reflection in graduate professional training by everyone involved, students, faculty, staff and administrators to create an environment that strives for excellence. recent publications (kenkel & peterson, 2009; fouad, et. al., 2009) help define competencies of knowledge, skill, and attitudes in psychology graduate education and discuss their importance. these concepts apply broadly to professional practice education. the next logical step is to create methods of evaluating those competencies and creating practical outcome measures, (kaslow, et. al., 2009). one such measure is the camp, which draws upon combinations of types of student learning and evaluations of competence assessment. the qe events are innovative and engage students in planned self-reflective practice, self-assessment and articulation of their developmental plan as a collaborative learner and budding professional (self-regulation). in addition, the qe events evaluate attitude. attitude has been more difficult to specifically measure than knowledge or skill. the qe events institute a self-regulatory process integrating both work products and educational process of development. mwu will continue to evaluate the camp as a means to measure student learning and program outcomes and the qe as reflective practice. we plan to incorporate many assessment methods including having a blind, outside rater review the post-qe reflection exercise, collect internship evaluations and review the feedback for student’s self-reflective competence, and include questions on the efficacy of the camp and qe in our alumni survey. as declared by roberts, borden, christiansen, and lopez (2005), a culture-shift toward assessment of competence and away from counting hours of practice requires development of innovative and comprehensive assessment methods. this culture-shift is occurring in many areas of graduate study and is particularly interesting in programs that lead to professional practice lewis, d., virden, t., hutchings, p. s., and bhargava, r.   journal of the scholarship of teaching and learning, vol. 11, no. 3, august 2011. www.iupui.edu/~josotl 97 degrees. the old adage, “more is not always better” comes to mind when reviewing this culture shift, as does the idea that acquisition of knowledge, technical skills and values can lay the foundation of striving for artistry. schön’s (1987) idea that we are seeking to train reflective practitioners is relevant, and forces us to look for innovation. schön’s (1987) brainstorm that educators can create novel and indeterminate situations for students to develop artistry, is still inspirational, and helps us think beyond the measurement of minimum competence thresholds. the authors have made some of the changes suggested by the expert reviewers, seeking to clarify definitions, reduce overlap, and refine evaluation methods. further research on the link between camp outcomes and actual practice success is suggested. further development of practical methods of compilation, electronic submission, storage and maintenance of information are necessary. the authors will continue to reflect on the utility of the camp and qe, in their efforts to train artists of professional psychology. the authors learned several lessons to offer as advice for programs considering adopting a similar method. first, focusing on outcome assessment is a paradigm shift. when we initially described the camp method to faculty outside the program, a common reaction was that it would be “too much additional work.” the camp is not an addition to work on top of what is in a program. it is the spine of our program. it demonstrates achieved entry-level competence in the knowledge, skills, and attitudes of the profession. everything feeds into competence assessment, including coursework and field training feedback. this is a change from judging a student’s competence based on courses completed, grade point average, and field training hours completed. the approach is much more individualized and based on what the student can demonstrate about what he or she knows, what he or she can do, and how he or she thinks. implementation requires a change of philosophy. second, logistics are important. we learned that each camp task needs a “home.” a specific instructor is tied to each task, often in a course or seminar, and is responsible for assuring that students turn it in on time, and that resubmissions, if needed, are completed. additional outcome measures will be collected to determine if the camp and qe are effective. the authors have prepared an alumni survey with questions on efficacy when our first cohort graduates. we also plan to elicit feedback from supervisors of students at their psychology pre-doctoral internship. internship is a full-time field training experience as part of a national match and supervisors are not associated with the program. we are planning to have blind, outside raters review the third year self-reflection paper (post-qe). the authors believe the move to competence assessment and inclusion of self-reflective practice and accuracy in self-assessment are the future of education. this is particularly true in professional practice programs. the camp and qe are practical examples of the philosophical shift. it is our hope to improve the education and training program for our students, contribute to the field of competency assessment, and to develop a learning community that supports aiming for artistry. lewis, d., virden, t., hutchings, p. s., and bhargava, r.   journal of the scholarship of teaching and learning, vol. 11, no. 3, august 2011. www.iupui.edu/~josotl 98 appendix 1. camp contributions & brief description. key: *ihce= integrated healthcare emphasis, 1=research evaluation & foundations of psychological science, 2=professionalism (ethics, diversity), 3= diagnostics & assessment, 4= intervention, 5=relationship & communication (see appendix v for description) in order due: year 1 competencie s evaluated submission title (abbreviated) 1-1 2 board of psychologist examiner’s paper 1 1-2 3 intelligence test administration (i.e. wais or wisc) on role-play subject 1-3 1 analysis of a psychometric test 1-4 3 intelligence test scoring (i.e. wais or wisc) responses supplied 1-5 3, 5 intelligence test interpretation & write-up (wais or wisc) 1-6 3, 5 objective personality test interpretation & write-up 1-7 3, 5 projective personality test interpretation & write-up 1-8 2,3 & 5 intake video sample & document: mse, diagnosis, diversity, ethics, writing quality, treatment plan, recommendations, rapport, self-critique 1-9 2,4 &5 rapport video sample with explanation of a conceptualization including diversity, orientation and self-critique 1-10 2 & 5 collegiality & professional practice (demonstrating supervisee preprtn.) 1-11 5 presentation of choice 1 from any class 1-12 5 pre-qeself-reflection end of year one after reviewing the camp. 1-13 1,5 literature review paper 1 from any class year 2 competencie s submissions 2-1 2, 3, 5 integrated test interpretation (i.e. write interpretation of a testssample data) 2-2 ihce* integrated behavioral healthcare assignment 2-3 2, 5 board of psychologist examiner’s paper 2 2-4 2,5 qualifying examination 2-5 all practicum evaluations 2-6 1, 5 literature review paper 2: from any class demonstrating scholarship (evidence base to inform an area of practice) 2-7 2, 3, 5 integrated assessment report (administer, score and interpret several tests with a client and write-up a report) 2-8 2, 3 & 5 intake tape of actual client (with document explaining treatment plan and recommendations, multicultural competence, issues of rapport, ethics and self-critique) 2-9 5 presentation of choice 2: from any class year 3 competencie s submissions 3-0 2 aapi application 3-1 ihce & 5 adv. integrated healthcare assignment: completes an integrated healthcare assignment 3-2 1 research project, 4 options: proposed psp project or acting as research assistant or having a poster or presentation at conference or master’s prjt. 3-3 all practicum evaluations lewis, d., virden, t., hutchings, p. s., and bhargava, r.   journal of the scholarship of teaching and learning, vol. 11, no. 3, august 2011. www.iupui.edu/~josotl 99 3-4 5 consultation proposal 3-5 5 relationship project (options: mentoring a student, acting as a teaching assistant or consultant to another program) 3-6 1, 5 literature review paper 3 from any class demonstrating evidence base for an area of clinical practice. 3-7 2 service/advocacy project: participation in community service, university service or committee work, advocacy effort 3-8 all intervention tape -actual client (document explaining choice of orientation, multicultural competence, issues of relationship, ethics and self-critique) 3-9 5,2 presentation of choice 3: given to audience outside the program (e.g. another department, the clinic, community site with program faculty in attendance. 3-10 5 post-qe,student reflection:after reviewing camp, describing development since the qualifying examination, goals, and self-care plan for internship year 4 competencie s submissions 4-1 1,5 (ihce, if ap) practitioner-scholar project 4-2 all (ihce, if ap) internship evaluations faculty will review student’s camp & progress through the program (annual review of students). students who have passed all courses, qe, complete camp, and training requirements are ready for graduation. additional documents include: annual student review feedback, field training log reports, training plans, remediation plans (if applicable). appendix 2. qualifying examination manual excerpts. description: the qe is an opportunity for students to demonstrate their skills in analysis and synthesis of information, self-evaluation and reflective thinking, self-direction in their own learning, professional identity, growth and commitment to that growth, creativity, ownership of their own work, and understanding of strengths and areas in need of development. students will be graded based on holistic methods and rubric distributed prior to the examination. in the areas of self-reflection and selfassessment, a doctoral candidate in the mwu program is able to: (sample area) accurately self-assess their competence in the domains and integrate that selfassessment to create plans for growth; able to accurately assess their own strengths and weaknesses; identify learning objectives & collaborates in the planning of their development. structure of the examination: the qualifying examination is an oral defense by the student in the presence of a minimum of two core faculty members of the department chosen by the department chair. the examination is scheduled for a total of 90 minutes. the student will create a powerpoint presentation on each question for half of the time allotted for the question, leaving the remaining half of the estimated time allotment for follow-up questions by faculty. please bring camp notebook to the exam. sample qualifying examination question: explain your present level of development in each of the five areas and the healthcare emphasis delineated in the developmental sequence of the program. you will present yourself for each competence area and then describe your plan for continued developmental through graduation. lewis, d., virden, t., hutchings, p. s., and bhargava, r.   journal of the scholarship of teaching and learning, vol. 11, no. 3, august 2011. www.iupui.edu/~josotl 100 sample scoring rubric for scores: 1, 3 & 6 on: self-reflection & accuracy of self-assessment 1: (beginning/novice level): demonstrates significant difficulty or defensiveness in evaluating own performance; not able to identify appropriate strengths or areas for further development and/or misidentifies more than one area or is inaccurate in more than one area; demonstrates little or no appreciation for self-awareness or dedication to self-development based upon personal evaluation; demonstrates significant difficulty in identifying how perceptions & assumptions have changed during his/her development & how this may impact future professional work & attitude. diversity awareness has gaps, or student is intolerant of interpersonal differences. 3: (developmentally expected level): demonstrates willingness to evaluate own performance, identifies 2 or 3 strengths & areas for further development; and is fairly accurate in self-assessment in all areas; demonstrates an appreciation for self-awareness and dedication to self-development based upon personal evaluation; able to identify how perceptions & assumptions have changed during his/her development & how this may impact future professional work & attitude toward diversity. 6: (sophisticated/advanced level): demonstrates ease in evaluation of own performance and shows significant insight into many areas of strength and areas for further development; shows clear appreciation for self-awareness and dedication to self-development and is able to self-correct without significant feedback; diversity awareness is sophisticated. appendix 3. holistic scoring descriptionsix (6) point scale. the scoring method is consistent across all comprehensive assessment in psychology (camp) submissions. this method avoids using single numbers to rank complex performances along a single dimension. all camp ratings are on a “6” point scale. a score of “3” is considered a minimum level for an acceptable camp submission. 1= beginning/novice level: student demonstrates difficulty in the expected knowledge, skills or attitude being evaluated. 2= basic level: student demonstrates below developmental expectation in some area of knowledge, skills and/or attitude being evaluated. 3= developmentally expected level: student demonstrates developmentally expected level in knowledge, skill and/or attitude being evaluated. 4= advanced basic level: student demonstrates above developmental expectations in some area of knowledge, skills or attitude being evaluated. 5= proficient advanced level: student demonstrates above developmental expectations on all knowledge, skills and attitudes being evaluated. 6= sophisticated advanced level: students demonstrates sophistication in knowledge, skills and attitudes being evaluated. lewis, d., virden, t., hutchings, p. s., and bhargava, r. journal of the scholarship of teaching and learning, vol. 11, no. 3, august 2011. www.iupui.edu/~josotl 101 appendix 4. competency assessment toolkit fifteen categories of assessment method (kaslow, et. al., 2009). the authors were able to integrate thirteen (and part of a fourteenth) of the fifteen categories of assessment method described in the “competency assessment toolkit” for professional psychology. the following were integrated as camp tasks, and implementation information is provided in parens, if illustrative: performance reviews (ratings by field training supervisors), case presentation reviews (in practicum seminars), competency evaluation rating forms (each camp submission), consumer surveys (at the mwu in-house clinic), live or recorded performance ratings (camp submissions, 2-8, 3-8), objective structured clinical examinations (camp submission 2-1 using “standardized patients,” trained actors to act as clients for students), portfolios, record reviews, simulations/role plays (camp submissions 1-8., 1-9), self-assessment (qe), standardized client interviews (at the mwu in-house clinic), structured oral exams and written exams (qe). client outcome data was not being collected at mwu in a format useful for student evaluation, and only part of a 360-degree evaluation was being implemented (peer review was a missing component). appendix 5. brief description of competence areas. 1. research and evaluation/foundations of psychological science: this competence includes the areas of research and evaluation, test construction, statistics, scholarship, and scientific mindedness. this competence rests on the assessor’s foundation of knowledge, skills, and professional attitudes in the areas of tests and measurement, statistics, qualitative methods, and experimental design. this competence also encompasses knowledge of the history of scientific psychology and its clinical applications, including the areas of physiological psychology, neuropsychology, psychopharmacology, cognitive and affective bases of behavior, history and systems of psychology, and social psychology. 2. professionalism: this competence includes the areas of ethics, diversity (defined broadly), self-care, awareness, self-reflection, practice management, collegiality, professional problem solving, a commitment to lifelong learning, critical thinking which underlies all subject matter and professional behavior. 3. diagnostics & assessment: the diagnostics and assessment competence rests on the foundation of knowledge, skills, and professional attitudes in the areas of human development, psychopathology and psychometric assessment. the diagnostics and assessment competence requires an ability to acquire and synthesize multiple sources of data into a comprehensive, cohesive and clearly articulated communication form. 4. intervention: this competence requires students to demonstrate an ability to intervene with clients from an articulated theoretical perspective. intervention is broadly defined to include a variety of activities that promote or sustain well-being or provide remedial or preventative services. intervention populations are broadly defined (e.g. individuals, groups, couples, families, communities). students demonstrate knowledge, skills and attitudes congruent with evidence-based practice rationales and can articulate them. 5. relationship & communication: the relationship competence requires a demonstration of interpersonal skills and effective written and oral communication. ability to consult and collaborate with others, interdisciplinary teams and members of agencies and organizations is considered part of relationship skills. evidence of ability to teach/present and manage at a lewis, d., virden, t., hutchings, p. s., and bhargava, r. journal of the scholarship of teaching and learning, vol. 11, no. 3, august 2011. www.iupui.edu/~josotl 102 developmentally appropriate level is also included. supervisory ability (including the ability to be supervised) is part of this competence. health care emphasis: the mwu clinical psychology program emphasizes a broad and general training in psychology. in addition, the program has an emphasis of psychological practice in integrated health care settings. appendix 6. construct validity survey method. a. participants. participants were selected on the basis of expertise in education and training in psychology. all participants had expertise in psychology program accreditation (e.g. served on the commission on accreditation of the american psychological association (apa), currently serve as consultants to psychology programs for apa accreditation or were a program director of a doctoral psychology program). eight expert participants were invited and five (62%) agreed to participate and completed the study. all participants were provided with materials concerning the camp and qe, including manuals and grading rubrics. participants were then contacted via email and provided with a link to an online survey evaluating the camp and qe. b. instrument and procedure. the camp and qe evaluation survey was a brief, 10-item questionnaire that consisted of 8 rating scale items and 2 open-ended queries. respondents used a 4-point scale anchored at strongly agree (1), agree (2), disagree (3), and strongly disagree (4) to rate the relative applicability of statements such as “a student who passed all camp submissions has adequately displayed the knowledge, skills, and attitude required to successfully begin a psychology internship” and “a student who passed the qualifying examination sufficiently demonstrates self-reflective practice at a level appropriate for advancement to doctoral candidacy.” two open-ended queries were free-response items in which the respondent was asked to provide any comments with regard to the assessment of the camp and qe. the survey was placed on the internet via limesurvey platform and made available through a url on the fourth author’s server. after clicking on the electronically provided link, respondents were directed by their browsers to the survey site, where they completed the items in 4 screens, including informed consent and debriefing pages. responses to each item were required in order for respondents to continue to the survey’s next section or to submit results. data was collected automatically and securely on the server and exported into a spreadsheet. c. data analysis. due to low sample size, the responses are presented solely via descriptive statistics. frequency and percentages of responses regarding opinions of the experts surveyed are illustrated in the appendices. open-ended responses were evaluated by two independent judges who classified comments into one of eight categories: competencies, faculty training, health care emphasis, objectives, rubrics, ratings and scoring, reliability, and remediation and repeated attempts. ebel’s estimated reliability coefficient (1951) indicated consistent interrater reliability (rx = 0.84) as did pearson’s correlation coefficient (r = 0.737, p = 0.004). vii. results. a. camp responses. lewis, d., virden, t., hutchings, p. s., and bhargava, r. journal of the scholarship of teaching and learning, vol. 11, no. 3, august 2011. www.iupui.edu/~josotl 103 the majority of experts surveyed (80%) reported they believed that a student who passed all camp submissions adequately displayed knowledge, skills, and attitude required to successfully begin an internship in psychology (m = 1.50, sd = 0.89) and that camp adequately samples the competency areas in professional psychology as defined by the program (m = 1.80, sd = 0.76). all respondents indicated that the camp is capable of documenting developmental achievement of competence (m = 1.40, sd = 0.72) and is developmental and graded in complexity (m = 1.40, sd = 0.609). most of the responding experts (60%) noted that they would adopt the camp or a similar method in their own programs if feasible (m = 2.00, sd = 0.884). the most comments were classified in the competencies (5), rating/scoring (4), and rubric categories (3). comments made regarding competencies generally recommended expanding the competencies to include diagnosis and supervision in the field. it was also noted that competencies in the camp appeared unclear and overlapped. some comments reflected a belief that camp competencies did not have enough emphasis on the regulatory guidelines for programs seeking accreditation by the apa. rating and scoring-related comments indicated confusion as to the utility of the scales and how outcomes are evaluated. comments regarding the rubric were varied, with some subjects indicating that they found it cumbersome and time-consuming and others noting it as a strength. table 1. frequency (percent) response on response items regarding camp and qe. question strongly agree agree disagree strongly disagree camp knowledge skills attitude for internship 3 (60%) 1 (20%) 1 (20%) 0 adequately samples competency areas 2 (40%) 2 (40%) 1 (20%) 0 documentation 3 (60%) 2 (40%) 0 0 developmental and graded in complexity 3 (60%) 2 (40%) 0 0 would adopt camp or similar 3 (60%) 0 1 (20%) 1 (20%) qe self-reflective 4 (80%) 1 (20%) 0 0 understands competence areas 2 (40%) 2 (40%) 0 1 (20%) would adopt qe or similar 1 (20%) 2 (40%) 1 (20%) 1 (20%) lewis, d., virden, t., hutchings, p. s., and bhargava, r. journal of the scholarship of teaching and learning, vol. 11, no. 3, august 2011. www.iupui.edu/~josotl 104 references bennett, d., pollock, p., & ryle, a. (2005). the states description process: the use of guided self reflection in the case formulation of patients with borderline personality disorder. clinical psychology and psychotherapy, 12, 50-57. doi:10.1002/cpp.416 borden, k.a., & mcilvried, e.j. (2010). applying the competency model to professional psychology education, training, and assessment: mission bay and beyond (43 – 53). in m.b. kenkel, & r.l. peterson (eds.), competency-based education for professional psychology, american psychological association, washington, dc. boud, d., keogh, r., & walker, d. (eds). (1985). reflection: turning experience into learning. london: kogan page. candy, p., harri-augstein, s., & thomas, l. (1985). reflection and the self-organized learner. in boud, d., keogh, r., & walker, d. (eds). (1985). reflection: turning experience into learning. london: kogan page. boud, d., keogh, r., & walker, d. (eds), reflection: turning experience into learning, 100-11. london: kogan page. cook-benjamin, l. (2003). portfolio assessment: benefits, issues of implementation, and reflections on its use. in banta, t. (ed). assessment update collections, 11-14. san francisco, ca: john wiley & sons. dewey, j. (1910). how we think. boston: d.c. heath & co. publishers. dimaggio, g., vanheule, s., lysaker, p., carcione, a., & nicolò, g. (2009). impaired selfreflection in psychiatric disorders among adults: a proposal for the existence of a network of semi-independent functions. consciousness and cognition: an international journal, 18, 653664. doi:10.1016/j.concog.2009.06.003 elbow, p. (2003). how portfolios show us problems with holistic scoring, but suggest an alternative. in banta, t. (ed). portfolio assessment: uses, cases, scoring and impact. 41-44. san francisco, ca: john wiley & sons. erlandson, p., & beach, d. (2008). the ambivalence of reflection – rereading schön. reflective practice, 9, 409-421. doi: 10.1080/14623940802475843 farnham-diggory, s. (1994). paradigms of knowledge and instruction. review of educational research, 64, 463-477. doi: 10.2307/1170679 fouad, n., grus, c., hatcher, r., kaslow, n., hutchings, p.s., madson, m., collins, f., & crossman, r. (2009). competency benchmarks: a model for understanding and measuring lewis, d., virden, t., hutchings, p. s., and bhargava, r. journal of the scholarship of teaching and learning, vol. 11, no. 3, august 2011. www.iupui.edu/~josotl 105 competence in professional psychology across training levels. training and education in professional psychology, 3, 550-553. doi: 10.1037/a0015832 hessler, a., & kuntz, s. (2003). student portfolios – effective academic advising tools. in banta (ed.) portfolio assessment: uses, cases, scoring and impact. 30-34. san francisco, ca: john wiley & sons. kaslow, n., grus, c., campbell, l., fouad, n., hatcher, r., & rodolfa, e. (2009). competency assessment toolkit for professional psychology. training and education in professional psychology, 3, s27-45. doi: 10.1037/a0015833 kenkel, m., & peterson, r. (2009). competency-based education for professional psychology. american psychological association: washington, dc. klenowski, v. (2002). developing portfolios for learning and assessment: processes and principles. london & new york: routledge/falmer. kösters, j., & ritzen, m. (2003). the amsterdam faculty of educations’ digital portfolio. in banta (ed.) portfolio assessment: uses, cases, scoring and impact. 30-34. san francisco, ca: john wiley & sons. leigh, i. w., smith, i, l., bebearu, m. j., lichtenberg, j. w., nelson, p. d., portnoy, s., rubin, n, j., & kaslow, n. j. (2007). competency assessment models. professional psychology: research and practice, 38, 463-473. doi: 10.1037/0735-7028.38.5.463 lombardi, j. (2008). to portfolio or not to portfolio: helpful or hyped? college teaching, 56, 710. doi: 10.3200/ctch.56.1.770 main, a. (1985). reflection and the development of learning skills. in boud, d., keogh, r. & walker, d. (eds.), reflection: turning experience into learning, 91-99. london: kogan page. marzano, r., & kendall, j. (2007). the new taxomony of educational objectives 2nd ed. thousand oaks, ca.: corwin press. mcguire, l., lay, k., & peters, j. (2009). pedagogy of reflective writing in professional education. journal of the scholarship of teaching and learning, 9, 93-107. retrieved from http://www.iupui.edu/~josotl pearson, m., & smith, d. (1985). debriefing in experience-based learning. in boud, d. keogh, r. & walker, d. (eds). reflection: turning experience into learning, 69-84. london: kogan page. roberts, m.c., borden, k.a., christiansen, m.s., & lopez, s. j. (2005). fostering a culture shift: assessment of competence in the education and careers of professional psychologists. professional psychology: research and practice, 36, 355-361. doi: 10.1037/0735-7028.36.4.355 lewis, d., virden, t., hutchings, p. s., and bhargava, r. journal of the scholarship of teaching and learning, vol. 11, no. 3, august 2011. www.iupui.edu/~josotl 106 saltiel, d. (2007). judgment, narrative and discourse: critiquing reflective practice. retrieved from http://www.leeds.ac.uk/medicine/meu/lifelong06/papers/p_davidsaltiel.pdf schafer, j., & giblin, m., (2008). doctoral comprehensive exams: standardization, customization, and everywhere in between. journal of criminal justice education, 19, 275-289. doi: 10.1080/10511250802137648 schön, d. (1987). educating the reflective practitioner: toward a new design for teaching and learning in professions. san francisco: jossey –bass. shelleyann, s., & dixon, k. (2009). partners in a learning organization: a student-focused model of professional development. educational forum, 73, 240-255. retrieved from http://www.tandf.co.uk/journals stoltenberg, c., mcneill, b., & delworth, u. (1998). idm supervision: an integrated developmental model for supervising counselors and therapists. san francisco: jossey-bass. tisani, n. (2008). challenges in producing a portfolio for assessment: in search of underpinning educational theories. teaching in higher education, 13, 549-557. doi: 10.1080/13562510802334830 zimmerman, b. j. (2004). sociocultural influences and student’s development of academic selfregulations: a social cognitive perspective. in mcierney, d.m. & van ellen, s. big theories revisited, 139-166. zimmerman, b. j. (2008) investigating self-regulation and motivation: historical background, methodological developments, and future prospects. american educational research journal, 45(1), 166-183. doi: 10.3102/0002831207312909 503 service temporarily unavailable 503 service temporarily unavailable nginx/1.14.1 3654 journal of the scholarship of teaching and learning, vol. 13, no. 5, december 2013, pp. 33 – 45. students’ perceptions of plagiarism reva fish1 and gerri hura2 abstract: while plagiarism by college students is a serious problem that must be addressed, students generally overestimate the frequency of plagiarism at their schools and blame students they do not know for the majority of incidents. this study looked at students’ estimations of the frequency of plagiarism at a large urban college and explored how that varied over the full range of types of plagiarism, from using another author’s ideas to submitting an entire document copied verbatim from another author’s work. analysis of student responses to survey items revealed they believe other students are far more likely than them to commit each type of plagiarism and they recognize that some types of plagiarism are more serious than others. the opportunity to reduce incidents of plagiarism by providing students with accurate information about plagiarism at their schools is discussed in the context of social norms theory. keywords: plagiarism, cheating, college, higher education, social norms theory i. introduction. while plagiarism is a widespread problem, college instructors tend to overestimate its frequency (hard, conway, & moran, 2006). students also believe plagiarism occurs more often than it does, to an even greater extent than faculty, and they generally attribute the high rate of incidents to strangers rather than people they know or themselves (engler, landau, & epstein, 2008). it is important to understand students’ beliefs about the frequency and nature of incidents of plagiarism at their schools. even though students expect faculty to impose consequences for academic misconduct (kuther, 2003; brown, 2012), they also look to other students’ behavior to determine how far they can push the boundaries of a professor’s course policies (feldman, 2001; mccabe, trevino, & butterfield, 2001; hard et al., 2006; rettinger & kramer, 2009). their opinion that some unidentified group of students at their school regularly submits work they did not do themselves can distort students’ understandings of acceptable strategies they should use to complete assignments. students who see some forms of plagiarism as less serious than others and who believe other students plagiarize frequently may become more likely to plagiarize themselves. this study looked at students’ estimations of the frequency of plagiarism at a large urban college and explored how that varied over the full range of types of plagiarism, from using another author’s ideas to submitting an entire document copied verbatim from another author’s work. it also looked at whether students believe some types of plagiarism are more serious than others. the consequences of students’ beliefs that plagiarism is a common practice and how institutions should address that are discussed.                                                                                                                           1 social and psychological foundations of education department, suny buffalo state, 1300 elmwood avenue, buffalo, ny 14222, fishrm@buffalostate.edu 2 adult education department, suny buffalo state fish, r., & hura, g. journal of the scholarship of teaching and learning, vol. 13, no. 5, december 2013. josotl.indiana.edu 34 a. research perspectives. plagiarism is a complex issue which has been studied using a variety of frameworks. some research has focused on student characteristics that predict a greater likelihood of committing plagiarism, including levels of moral reasoning and self-esteem as well as achievement and motivation orientations (angell, 2006; rettinger & kramer, 2009; williams, nathanson, & paulhus, 2010). this perspective attributes the decision to plagiarize to characteristics of the students, discounting outside factors that might contribute to the choice to plagiarize. other research has regarded incidents of plagiarism as being the result of teaching style (barnas, 2000) or classroom culture (brown, 2012; feldman, 2001) indicating the cause of plagiarism originates outside the student. from these perspectives, instructors are seen as contributing to students’ beliefs that they can submit another author’s work as their own by not providing an adequate level of rigor in their classrooms or by not checking student work for plagiarism. unintentional plagiarism has also been used as a framework for research (belter & du pre, 2009; blum, 2009; colnerud & rosander, 2009). this viewpoint often raises the question of whether students should be penalized when they are unaware they have plagiarized. while proof of intent to plagiarize is typically not believed to be necessary to support an accusation, whether students who are still learning to write academic papers should be expected to fully understand how to avoid plagiarism has been addressed in these studies. ethics, and in particular integrity, is another focus of the research on plagiarism (conway & groshek, 2009; feldman, 2001; kuther, 2003; mccabe et al., 2001; hart & morgan, 2010; hudd, apgar, bronson, & lee, 2009; kwong, ng, mark, & wong, 2010). that body of work examines plagiarism at the student, instructor, and institution levels, and emphasizes the need for institutions to convey the importance of honesty to students and for faculty to model ethical behavior for them. more recently the focus of plagiarism research has been on technology-facilitated electronic access to text as a primary cause of the increase in the number of incidents of plagiarism (jones, 2011; trushell, byrne, & simpson, 2012; wang, 2008). this method of plagiarism has become increasingly widespread through the effortless process of copying and pasting electronic text. some studies have found that students may believe information on the internet does not belong to a particular author and, therefore, can legitimately be used by them in course assignments. engler et al. (2008), hard et al. (2006), and the present study looked at plagiarism from the perspective of social/peer norms. according to social norms theory, individuals learn which behaviors are appropriate by observing the generally accepted behavior of others. for example, young adults have been found to overestimate the frequency of negative behaviors such as substance abuse by their peers, resulting in an inaccurate understanding of what is considered socially acceptable and an increase in those negative behaviors on their part (berkowitz, 2004; perkins, 2003; perkins & berkowitz, 1986). based on this theory, if students have the misperception that acts of plagiarism are common among their classmates, and that consequences, if any, are minor, they are more likely to commit plagiarism themselves. fish, r., & hura, g. journal of the scholarship of teaching and learning, vol. 13, no. 5, december 2013. josotl.indiana.edu 35 b. what is plagiarism? many studies of plagiarism do not provide an operational definition of it, seeming to assume there is a one common understanding that does not need explication. powers (2009) points out that this can affect research findings because students’ self-reports of plagiarism are affected by an individual understanding of the practices that could be considered plagiarism. further, faculty and students often disagree about exactly what constitutes plagiarism (kwong et al., 2010). definitions of plagiarism from several of the studies that provided one are listed in table 1. table 1. definitions of plagiarism. belter & dupre (2009): “one or more passages that was word-for-word the same as another source without appropriate citation and quotation marks.” p. 259 colnerud & rosander (2009): “using parts, or the whole, of a text written by another person without acknowledgement; submitting the same paper or parts of it, for credit in more than one course, falsification of information.” p. 506 hard, conway, & moran (2006): “presenting, as one's own, the ideas or words of another person or persons for academic evaluation without proper acknowledgement.” p. 1059 park (2003): “plagiarism involves literary theft, stealing (by copying) the words or ideas of someone else and passing them off as one’s own without crediting the source.” p. 472 wang (2008): “us[ing] somebody else’s work (words and thoughts) without attribution.” p. 743 williams, nathanson, & paulhus (2010): “any nonzero percentage detected by turn-it-in (after screening).” p. 294 a common element across definitions is that plagiarism is the act of using another author’s work without citation, thus portraying it as one’s own work. other common elements of definitions include descriptions of the length of the copied text, whether taking solely ideas from other authors is plagiarism, and the extent that the copied words were taken verbatim. for the present study a definition of plagiarism was developed that addressed these elements: plagiarism is representing another author’s ideas or words as your own in course documents or electronic postings. this would include submitting an entire document by another author as well as using a portion of text or ideas from another author’s work and not citing the source. this would include information obtained from the internet, from other students, and from published and unpublished documents. this definition was provided to the students on the survey they completed. c. plagiarism along a continuum.   incidents of plagiarism are viewed along a continuum, with some incidents regarded as more serious than others (blum, 2009; hudd et al., 2009; jones, 2011; kwong et al., 2010, salmons, 2007). studies of faculty and student understandings of plagiarism have found that faculty view most types of plagiarism as more serious than students view them (kwong et al., 2010). jones (2011) found that while all students recognized submitting an entire document written by another author as plagiarism, students saw copying a limited amount of text as less serious. seventy-five percent of students saw purchasing a paper online as plagiarism, 67% thought copying text fish, r., & hura, g. journal of the scholarship of teaching and learning, vol. 13, no. 5, december 2013. josotl.indiana.edu 36 verbatim without quotation marks was plagiarism, 50% saw paraphrasing text without citation as plagiarism, and 17% stated that students should not self-plagiarize by submitting the same work for assignments in different classes. d. student and faculty perceptions of plagiarism frequency. faculty and students tend to overestimate the frequency of student plagiarism (engler et al., 2008; hard et al., 2006; wang, 2008). students, in particular, see plagiarism as a common practice even though they report they have never plagiarized themselves (wang, 2008). students believe their friends are more likely to plagiarize than they are, but their friends are less likely to plagiarize than students they do not know (engler et al., 2008; kwong et al., 2010). it is important to consider student overestimates of plagiarism by others because students’ perceptions of peer behavior have a powerful effect on their own behavior (hard et al., 2006; mccabe et al., 2001; rettinger & kramer, 2009). both mccabe et al. (2001) and rettinger and kramer (2009) found that while there are a number of factors that predict cheating, knowing that other students have cheated has the greatest influence on a student’s decision to cheat. even faculty, whose role it is to discover and address incidents of plagiarism, overestimate its occurrence, although to a lesser degree than students (hard et al., 2006). an advantage to faculty overestimations of plagiarism is that it may make them more vigilant, benefitting students who do not plagiarize and who want it addressed (kuther, 2003). students generally appreciate instructors who can effectively monitor classroom learning and provide an appropriate level of rigor (barnas, 2000). they want faculty to show respect for all students’ efforts by not tolerating any form of cheating, including plagiarism – the most common form of cheating in higher education (trost, 2009). faculty can specifically mention in the course syllabus that submitting another author’s work will not be tolerated, and the consequences if this happens, so students do not mistakenly believe that cheating will be ignored (brown, 2012; feldman, 2001). when incidents of plagiarism are uncovered, if faculty discuss the circumstances with the class, without disclosing the name of the student who plagiarized, they can show their vigilance when reviewing assignments and prevent additional incidents of plagiarism by students who thought it would be ignored (feldman, 2001). the research reported here is a part of a larger study that explored the scope and nature of plagiarism by students at a large urban college in order to determine the current extent of plagiarism there and how past institutional efforts to curb plagiarism were faring. these included implementation of an academic misconduct policy and use of plagiarism detection software. the questions addressed in this report of the study are: 1. what is the frequency and nature of plagiarism admitted to by students? 2. what do students believe is the frequency and nature of plagiarism committed by other students? 3. do students view some types of plagiarism as more serious than others? 4. do students believe that the types of plagiarism they view as more serious are more likely to be committed by other students? fish, r., & hura, g. journal of the scholarship of teaching and learning, vol. 13, no. 5, december 2013. josotl.indiana.edu 37 ii. method. a. participants. a survey was conducted at a large urban public comprehensive college with over ten thousand students, undergraduate and graduate, enrolled each year. an email was sent to all students, inviting them to complete the anonymous electronic survey and providing them with an internet link to it. the number of emails sent varied by department, but all students received at least one email. information about the survey was also posted on the home page of the campus library website and on the webpage students use to access email, check grades, register for courses, and so forth. the data collection process was reviewed and approved by the college’s institutional review board. of the 626 students who responded to the survey, 334 students reported that they had been enrolled in classes which had assignments that could have been plagiarized and completed the survey items analyzed in the present study. assignments which could be plagiarized were described in the survey as writing assignments that included information that could have been obtained from another source and misrepresented as the student’s own work. the 334 students included 194 undergraduates and 131 graduate students. nine students did not report their student level. respondents ages ranged from 18 years to 62 years, and almost 52% of the students had a self-reported grade point average over 3.5, on a scale of 0.0 to 4.0. table 2 provides full demographic information about the sample. b. instrument. the student survey asked respondents about their views and experiences regarding plagiarism and was developed by reviewing published studies on plagiarism, examining efforts to address plagiarism at institutions across the country, and discussing current concerns with administrators and faculty at the institution where the study was conducted. this report of the research will focus on three questions from the survey. in the first of these questions, the students were asked to rate four types of plagiarism as not at all serious, somewhat serious, or very serious. the four types of plagiarism they rated were: • using ideas from another author’s work and not citing the source • using phrases from another author’s work and not citing the source • using sentences/paragraphs from another author’s work and not citing the source • submitting an entire document by another author as your own work in the second question, students were asked to indicate how often they thought students committed each of the four types of plagiarism in writing assignments. the response choices were never, once, rarely (few of them), occasionally (up to one-half of them), regularly (more than half of them), always (all of them). the third question was the same as the second question, but asked each student to indicate how often they had committed each of the four types of plagiarism. the response choices were the same as those in the second question. fish, r., & hura, g. journal of the scholarship of teaching and learning, vol. 13, no. 5, december 2013. josotl.indiana.edu 38 table 2. student demographics. student characteristic percent (n) gender male 32.6 (109) female 66.8 (223) gender not provided 0.6 (2) level freshman 6.6 (22) sophomore 5.1 (17) junior 16.5 (55) senior 29.9 (100) graduate 39.2 (131) level not provided 2.7 (9) grade point average < 2.00 0.9 (3) 2.00 to 2.50 3.9 (13) 2.51 to 3.00 9.3 (31) 3.01 to 3.50 20.0 (67) 3.51 to 4.00 51.5 (172) grade point average not provided 14.4 (48) age < 20 75. (25) 20 to 25 51.2 (171) 26 to 30 13.1 (44) > 30 18.9 (63) age not provided 9.3 (31) c. analysis. data analysis was carried out in two stages. in the first stage, descriptive statistics of the categorical and likert-type scale survey responses were used to answer the first two research questions. in the second stage, the third and fourth research questions were answered using inferential z tests to determine if there were statistically significant differences in proportions of the sample who selected survey item responses. in each analysis the requirement of at least five cases for each of the two responses compared, to approximate a normal distribution, was met. a type-1 error rate of α = .05 was used for all tests of significance. odds ratios (or) were used to determine the strength-of-effect for all significant results, with or 1.50, 3.00, and 5.00 used to indicate small, medium, and large effect sizes, respectively (chen, cohen, & chen, 2010). for ease of interpretation, all odds ratios were calculated so that a value greater than 1.00 would result (mchugh, 2009). fish, r., & hura, g. journal of the scholarship of teaching and learning, vol. 13, no. 5, december 2013. josotl.indiana.edu 39 iii. results. table 3 shows the results for question 1: what is the frequency and nature of plagiarism admitted to by students? a majority of the respondents said they had never used another author’s phrases (62.6%), sentences/paragraphs (82.3%), or entire piece of writing (96.4%). a majority of the students also reported that had either never or once used another author’s idea and portrayed it as their own work (40.7% and 10.5%, respectively). as evident from the values in the table, there was a systematic decline in the admissions of plagiarism as the amount of text that was copied and the rate of occurrence increased. none of the students reported always committing plagiarism of any type and few to none reported plagiarizing regularly (0.0% to 3.9%). table 3. survey question about how often respondent plagiarizes. question: how often have you done the following, without citing the source? n ev er pe rc en t( n) a o nc e pe rc en t( n) r ar el y pe rc en t( n) o cc as io na lly pe rc en t( n) r eg ul ar ly pe rc en t( n) a lw ay s pe rc en t( n) used another author’s ideas 40.7 (136) 10.5 (35) 32.6 (109) 12.3 (41) 3.9 (13) 0.0 (0) used another author’s phrases 62.6 (209) 10.8 (36) 18.6 (62) 6.6 (22) 1.5 (5) 0.0 (0) used another author’s sentences/paragraphs 82.3 (275) 6.6 (22) 7.2 (24) 3.0 (10) 0.9 (3) 0.0 (0) used entire document by another author 96.4 (322) 0.0 (0) 2.7 (9) 0.9 (3) 0.0 (0) 0.0 (0) a students significantly more likely to never commit each type of plagiarism than other students, p < .05. see table 4.   table 4 provides information about question 2: what do students believe is the frequency and nature of plagiarism committed by other students? the majority of respondents believe that other students are either occasionally or regularly plagiarizing ideas and phrases (39.8% and 40.1%, 48.5% and 22.8%, respectively), and that other students are rarely or occasionally plagiarizing sentences/paragraphs or entire documents by other authors (40.7% and 34.7%, 49.1% and 9.6%, respectively). a small proportion of respondents reported they believe other students always commit each of the four types of plagiarism (1.2% to 3.6%). overall, respondent’s reports of their beliefs about other students’ plagiarism do not show the systematic decline seen in the students’ reports of their own plagiarism as the amount of work that was copied and the frequency increased. in fact, almost half of the respondents (49.1%) reported that fish, r., & hura, g. journal of the scholarship of teaching and learning, vol. 13, no. 5, december 2013. josotl.indiana.edu 40 they believe other students have submitted an entire document by another author a few times (rarely). table 4. survey question about how often other students plagiarize. question: how often do you think most students do the following, without citing the source? n ev er pe rc en t( n) a o nc e pe rc en t( n) r ar el y pe rc en t( n) o cc as io na lly pe rc en t( n) r eg ul ar ly pe rc en t( n) a lw ay s pe rc en t( n) used another author’s ideas 1.5 (5) 1.5 (5) 13.5 (45) 39.3 (133) 40.1 (134) 3.6 (12) used another author’s phrases 2.7 (9) 2.4 (8) 22.2 (74) 48.5 (162) 22.8 (76) 1.5 (5) used another author’s sentences/paragraphs 5.7 (19) 7.8 (26) 40.7 (136) 34.7 (116) 9.9 (33) 1.2 (4) used entire document by another author 19.2 (64) 18.0 (60) 49.1 (164) 9.6 (32) 3.0 (10) 1.2 (4) a students significantly more likely to never commit each type of plagiarism than other students, p < .05. see table 3.   information in table 5 answers question 3: do students view some types of plagiarism as more serious than others? the pattern of responses shows that students found plagiarism more serious as the amount of material taken from another author increased. most of the students believed using another author’s ideas is at least somewhat serious. a majority of the students indicated that using another author’s phrases or sentences/paragraphs is very serious plagiarism (51.8%, 78.1%, respectively). almost all of the students reported that copying an entire document written by another author was very serious (96.4%). to answer the research question, the proportions of students who chose each response – not at all, somewhat, very – for plagiarizing another author’s ideas and for plagiarizing an entire document by another author were compared. there was a statistically significant difference at each of the three levels of seriousness. there was a significant difference between the proportion who indicated that using another author’s ideas was not at all serious and the proportion who indicated copying an entire document by another author was not at all serious, z = 7.736, p < .05, with a very large odds ratio of 16.51. the odds of a student believing that using another author’s idea is not at all serious is more than 16 times greater than the odds of a student believing that copying an entire document by another author is not at all serious. the proportion who responded that using another author’s idea was somewhat serious was significantly different from the proportion that indicated copying an entire document by another author was somewhat serious, z = 14.352, p < .05, with a very large odds ratio of 49.01. the odds of a student reporting that using another author’s idea is fish, r., & hura, g. journal of the scholarship of teaching and learning, vol. 13, no. 5, december 2013. josotl.indiana.edu 41 somewhat serious is about 49 times greater than a student reporting copying an entire document by another author is somewhat serious. finally, the difference in the proportion of students who said using another author’s idea is very serious was significantly different from the proportion of students who said using an entire document by another author is very serious, z = -18.069 p < .05, with a very large odds ratio of 66.52. the odds of a student believing that copying an entire document is very serious is over 66 times greater than the odds of a student believing that using another author’s idea is very serious. these findings indicate that students do believe that some types of plagiarism are more serious than others and that the level of seriousness of the plagiarism is based on the amount of information taken from another author. however, it should be noted the students reported they believed each type of plagiarism was serious in nature. table 5. survey question about seriousness of types of plagiarism. question: how serious an incident is each of the following? n ot a t a ll pe rc en t( n) so m ew ha t pe rc en t( n) v er y pe rc en t( n) using another author’s ideasa 20.1 (67) 51.2 (171) 28.7 (96) using another author’s phrases 4.5 (15) 43.7 (146) 51.8 (173) using another author’s sentences/paragraphs 2.1 (7) 19.8 (66) 78.1 (261) using entire document by another author 1.5 (5) 2.1 (7) 96.4 (322) ausing ideas significantly different from using entire document, at each level of seriousness, p < .05. question 4, whether students believe that the types of plagiarism they view as serious are more likely to be committed by other students than by them, was answered by comparing the proportion of students who indicated they had never committed each type of plagiarism and the proportion who indicated they believed other students had never committed that type of plagiarism. a statistically significant difference was found for each type of plagiarism, p < .05. students’ were more likely to report that they never plagiarized ideas, phrases, sentences/paragraphs, or an entire document than they were to report they believed that other students had never committed each of those four types of plagiarism, z = 12.421, 16.503, 19.953, 20.211, respectively. the odds ratios show that the difference in proportions increased as the amount of information plagiarized – the seriousness of the plagiarism – increased. the odds of a student reporting they would never plagiarize ideas, phrases, sentences/paragraphs, or an entire document compared to the odds they believed another student would never commit the same act were all very large – 45.20, 60.38, 77.27, and 113.20, respectively. fish, r., & hura, g. journal of the scholarship of teaching and learning, vol. 13, no. 5, december 2013. josotl.indiana.edu 42 iv. discussion. this study surveyed students at a large urban college to explore their beliefs about plagiarism. students were asked how often they commit plagiarism and how often then think other students commit plagiarism. they were also asked about how serious an incident they considered each of four types of plagiarism – using another author’s ideas, phrases, sentences/paragraphs, and submitting an entire document written by another author. most of the students in the sample reported never committing plagiarism of any type and there was a systematic decline in the admissions of plagiarism as the amount of text that was copied and the frequency of occurrence increased. the participants indicated they believe that some types of plagiarism are more serious than others, with taking larger sections of text from another author seen as the more serious incidents of plagiarism. still, even using another author’s ideas was believed to be at least somewhat serious by most students. the pattern of students being less likely to commit the types of plagiarism they saw as more serious was not evident in their reports of the plagiarism they believe was committed by other students. consistent with the findings from other research (engler et al., 2008), the participants reported that other students were markedly more likely than them to commit each type of plagiarism. in fact, a small proportion of participants reported they believe some students always commit each of the four types of plagiarism. the implications of these findings must be considered in relation to social norms theory (berkowitz, 2004; perkins, 2003; perkins & berkowitz, 1986) and the research by mccabe et al. (2001) and rettinger and kramer (2009) which found that when students believe others have cheated, they are more likely to choose to cheat. students who overestimate the frequency of plagiarism by classmates may view plagiarizing as a norm and choose to plagiarize to complete an assignment. in particular, they may be more likely to think it is okay to use another author’s ideas or text phrases – types of plagiarism, which they view as less serious. therefore, it is critical that students have accurate information about the frequency and types of plagiarism committed by students in their classes and at their school. institutions and course instructors must find ways to give students accurate information about the types of plagiarism that occur and number of incidents that are discovered. a campus-wide reporting system should be used to gather and distribute the information, and faculty can tell students the number and type of actual incidents they encounter semester to semester. supplied with this information, students will be less likely to overestimate the number of incidents of plagiarism at their school and may, therefore, be less likely to plagiarize themselves. informing students that instructors look for plagiarism in assignments and that there are consequences if it is discovered can help to deter students from plagiarizing and create an environment where it is clear that ethical behavior is valued. reducing plagiarism provides benefits beyond an ethical education environment; it can also improve student learning. when students do their own work, instead of copying it from another author, they learn research and writing skills, and they learn the topic content of the papers they write. the long-term benefits of an academic environment where integrity and learning are cultivated cannot be overestimated. one limitation of this study is the possibility that the participants who chose to respond to the survey may have been those who actually were less likely to plagiarize than their classmates and therefore their claim that others are more likely to plagiarize was accurate. more than half of the respondents had a grade point average over 3.5, and studies have found that students with lower grade point averages are more likely to plagiarize (belter & dupre, 2009; park, 2003). fish, r., & hura, g. journal of the scholarship of teaching and learning, vol. 13, no. 5, december 2013. josotl.indiana.edu 43 another limitation of this study is the reliance on student self-reports about their behavior. it is possible that some respondents were purposely untruthful or that they inaccurately reported they had not plagiarized. they may have viewed any plagiarism by them as justifiable behavior. survey responses may also have been inaccurate if students did not fully understand the meaning of the questions asked. it should be noted that the finding that students believe taking any amount of text is more serious than taking another author’s idea may be due to the order that the types of plagiarism were listed in the survey -with taking ideas listed first, followed by taking increasing amounts of text. while it is clear that an increased amount of copied text – from phrases/sentences to paragraphs to an entire document – indicates a more serious incident of plagiarism, it is not as clear whether copying another author’s ideas is less or more serious than copying text. course instructors, especially those who are published authors in their field, may believe that plagiarism of an original idea is more egregious. students, who typically have no experience in professional writing, may not understand the value of developing a unique idea in a field of study and, therefore, see taking another author’s idea as less important than copying text. even if instructors believe plagiarism of ideas is the most serious type of plagiarism by published scholars, they may believe the reverse when assessing student work – viewing copying of ideas as typical behavior in the process of developing writing skills. they may even believe that copying small amounts of text, such as phrases, are not serious incidents. while all incidents of plagiarism by students must be addressed, instructors should determine the appropriate consequences for each based on a student’s writing skills and knowledge at the time the assignment is completed, the instructor’s belief about whether the incident was intentional, and institution policies. in conclusion, while the findings from this study cannot be generalized because they are based on a relatively small number of students’ self-reports at one institution, the findings do provide information about a novel practice that may reduce the number of incidents of plagiarism. publicizing the nature and frequency of plagiarism on a campus is a relatively costfree and potentially effective way to not only reduce the time-consuming and emotionally difficult process of dealing with incidents of plagiarism, but to improve student learning as well. additional studies should be conducted at schools to explore their students’ understandings of plagiarism and to determine whether sharing accurate information about incidents of plagiarism will reduce the likelihood of incidents overall. acknowledgement this research was supported by a campus fellowship from the carnegie academy for the scholarship of teaching and learning. the authors appreciate the reviewers’ valuable suggestions for the improvement of this paper. references angell, l. r. (2006). the relationship of impulsiveness, personal efficacy, and academic motivation to college cheating. college student journal, 40(1), 118-131. barnas, m. (2000). “parenting” students: applying developmental psychology to the college classroom. teaching of psychology, 27, 276-277. fish, r., & hura, g. journal of the scholarship of teaching and learning, vol. 13, no. 5, december 2013. josotl.indiana.edu 44 belter, r. w., & dupre a. (2009). a strategy to reduce plagiarism in an undergraduate course. teaching of psychology, 36(4), 257-261 berkowitz, a. d. (2004). an overview of the social norms approach. in l. lederman, l. stewart, f. goodhart and l. laitman (eds). changing the culture of college drinking: a socially situated prevention campaign (187-208). new york, ny: hampton press. blum, s. d. (2009). my word! plagiarism and college culture. ithaca ny: cornell university press. brown, g. (2012). student disruption in a global college classroom: multicultural issues as predisposing factors. association of black nursing faculty journal, 23(3), 63-69. chen, h., cohen, p., & chen s. (2010). how big is a big odds ratio? interpreting the magnitudes of odds rations in epidemiological studies. communications in statistics – simulation and computation, 39(4), 860-864. colnerud, g., & rosander, m. (2009). academic dishonesty, ethical norms, and learning. assessment and evaluation in higher education, 34(5), 505-517. conway, m., & groshek, j. (2009). forgive me now, fire me later: mass communication students’ ethics gap concerning school and journalism. communication education, 58(4), 461482. engler, j. n., landau, j. d., & epstein, m. (2008). keeping up with the joneses: students’ perceptions of academically dishonest behavior. teaching of psychology, 35, 99-102. feldman, l. j. (2001). classroom civility is another of our instructor responsibilities. college teaching, 49, 137-140. hard, s. f., conway, j. m., & moran, a. c. (2006). faculty and colleges student beliefs about the frequency of student academic misconduct. the journal of postsecondary education, 77(6), 1058-1080. hart, l., & morgan, l. (2010). academic integrity in an online registered nurse to baccalaureate in nursing program. journal of continuing education in nursing, 41(11), 498-505. hudd, s. s., apgar, c., bronson, e. f., & lee, r. g. (2009). creating a campus culture of integrity: comparing the perspectives of fulland part-time faculty. the journal of higher education, 80(2), 146-177 jones, d. l. r. (2011). academic dishonesty: are more students cheating? business communication quarterly, 74(2), 141-150 kuther, t.l. (2003). a profile of the ethical professor: student views. college teaching, 51(4), 153-160. fish, r., & hura, g. journal of the scholarship of teaching and learning, vol. 13, no. 5, december 2013. josotl.indiana.edu 45 kwong, t., ng, h.m., mark, k.p., & wong, e. (2010). students' and faculty's perception of academic integrity in hong kong, campus-wide information systems, 27(5), 341-355. mccabe, d. l., trevino, l. k., & butterfield, k. d. (2001). cheating in academic institutions: a decade of research. ethics & behavior, 11(3), 219-233. mchugh, m. l. (2009). the odds ratio: calculation, usage, and interpretation. biochemia medica, 19(2), 120-126. park, c. (2003). in other (people’s) words: plagiarism by university students – literature and lessons. assessment & evaluation in higher education, 28(5), 471-488. perkins, h. w. (ed.). (2003). the social norms approach to preventing school and college age substance abuse: a handbook for educators, counselors, and clinicians. san francisco, ca: jossey-bass. perkins, h.w., & berkowitz, a.d. (1986). perceiving the community norms of alcohol use among students: some research implications for campus alcohol education programming. international journal of the addictions, 21(9/10), 961-976. powers. l. g. (2009). university students’ perceptions of plagiarism. the journal of higher education, 80(6), 643-662. rettinger, d. a., & kramer, y. (2009). situational and personal causes of student cheating. research in higher education, 50, 293–313. salmons, j. (2007). online social culture: does it foster original work or encourage plagiarism? retrieved march 14, 2009 from http://www.vision2lead.com/originality.pdf trost, k. (2009). psst, have you ever cheated? a study of academic dishonesty in sweden. assessment and evaluation in higher education, 34(4), 367-376 trushell j., byrne, k., & simpson, r. (2012). cheating behaviours, the internet and education undergraduate students. journal of computer assisted learning, 28(2), 136-145. wang, y. (2008). university student online plagiarism. international journal on e-learning, 7(4), 743-757. williams, k. m, nathanson, c., & paulhus, d. l. (2010). identifying and profiling scholastic cheaters: their personality, cognitive ability, and motivation. journal of experimental psychology: applied, 16, 293-307.   microsoft word v8n3dillon.doc journal of the scholarship of teaching and learning, vol. 8, no. 3, october 2008, pp. 36 – 49. how alumni narratives of intercultural competence can inform the scholarship of teaching and learning of intercultural communication randy k. dillon 1 abstract: through email correspondence and interviews with former students the author explores how the scholarship of teaching and learning can serve as a way to reflect upon philosophy and pedagogy in an intercultural communication course. key lessons of intercultural competence from former students and how these lessons have impacted their professional and or personal lives are shared. the essay includes an assessment of how issues of diversity and intercultural competence from former students can be implemented in the classroom. questions that draw from the scholarship of teaching and learning that challenge and sustain teaching and learning are posed. keywords: intercultural communication, intercultural competence, diversity, narrative, alumni i. introduction. all faculty, throughout their careers should themselves remain students. as scholars they must continue to learn and be seriously and continuously engaged in the expanding intellectual world.(boyer, 1990, p.36) after years of teaching a particular course it is not uncommon for an instructor to become complacent. other tasks and challenges scholarly, pedagogically, or administratively crowd out needed reflection on why the course has been a success or how it could be retooled. too often, it is easier to stick with the status quo and not make inquiries or adjustments until immediate attention and decisions are required. i teach an intercultural communication course, and have been doing so almost every semester since i arrived at my current institution 14 years ago. i enjoy teaching the course. students are attracted to the course, with what i believe, goes beyond availability or whether or not i teach it. with globalization and the increased need for more awareness of other cultures, the intercultural communication course is an easy “sell” as a requirement for many of our majors or as an elective for other students across the university. despite being well into my second decade of teaching intercultural communication i remain vigilant in not becoming complacent. i do not want to be accused by my students of referring to the same yellowed lecture notes. i welcome new and relevant content such as integrating current events representative of concepts and themes that need to get presented to students in an introductory undergraduate course. my students and i have together consulted internet news sources and in class debated the current cultural controversies. i have divided students into small groups where they consult on cultural 1 department of communication, missouri state university, 901 s. national avenue, springfield, mo 65897, randydillon@missouristate.edu dillon, r. journal of the scholarship of teaching and learning, vol. 8, no. 3, october 2008. 37 case studies. they then post their case study responses via blackboard online format where a blended classroom approach between traditional and online discussions can take place. one consistency that guides learning in my intercultural communication courses is the use of the pedagogical lens of intercultural competence. intercultural competence involves a combination of knowledge, skills, and attitude. knowledge is the cognitive ability to be both aware of one’s own communication as well as others’ communication, and learning the ways to build better intercultural relations. the skills component depends on how the individual can apply or carry out what he or she knows. attitude consists of one’s motivation to gain more knowledge about intercultural communication, and taking opportunities of applying skills one has learned (lustig & koester, 2005). a couple of years ago i began to take a more scholarly interest about teaching and learning in the intercultural communication course beyond adding new information or the “bells and whistles” of course delivery. i began by asking, what is a more systematic and scholarly way of assessing the course? in addition, how can i become more competent myself in order to make the course a satisfying learning experience? this paper focuses on my ongoing experiences with teaching the intercultural communication course and how the scholarship of teaching and learning has allowed me to reflect on my own teaching as well as challenge and sustain teaching and learning for the future. the scholarship of teaching and learning consists of a “systematic reflection on teaching and learning made public” (http://www.sotl.ilstu.edu). shulman (2004) writes, “scholars don’t engage in traditional research because they failed when doing it before; they do it because they have done it well and now want to learn more” (p. 21). researching teaching and learning within the intercultural communication course did not mean that i was doing things wrong. similarly, despite several years of successfully teaching the intercultural communication course, i had to rid myself of what rosenzweig (1999) notes is a tendency for experienced teachers to believe they know the answers because they have been doing it for so long. “in the scholarship of teaching, the teacher cares deeply about the discipline but, equally, about the learners and their connection to both the material, the discipline, and learning” (theall & centra, 2001, p. 42). as a way to learn more about teaching and learning within my own courses, i often turn to former students as collaborators. i do not expect these former students to have the answers. i believe that encouraging students to relate their journeys with intercultural competence can extend beyond the classroom, and can offer perspectives and potential for teaching and learning for future intercultural communication classes. likewise, darling (2003) states that the goal of the scholarship of teaching and learning “is not to abstract theoretic claims, but to offer contextualized accounts of our efforts to understand teaching better and to enhance student learning” (p. 48). honoring former students’ experiences assists in the process of understanding teaching and learning in the intercultural communication classroom. a. listening to student narratives. personal narratives consist of “stories lived and told” (clandinin & connelly, 2000, p. 9), and bring together one’s experience and identity (hantzis, 1995). sharing one’s own story is important for self understanding, and according to bateson (1997) “includes a dimension of justification, not only ‘what i did’ and ‘what happened’ but how and why i chose to understand it, and a readying of the self for the task that lies ahead” (p. viii). goffman (1981) refers to narrative as “strips of personal experience” from the past that is “replayed” (p. 174). narrative dillon, r. journal of the scholarship of teaching and learning, vol. 8, no. 3, october 2008. 38 functions not only provide a report of an experience, but also provides something to reexperience (goffman, 1975, p. 506). “[this] reexperiencing is not only for the teller but also for the audience so that they can empathetically insert themselves into replaying, vicariously experiencing what took place” (cortazzi, 1993, p. 39). elbaz (1990) writes that “story is the very stuff of teaching, the landscape which we live as teachers and researchers” (p. 32). the study of teachers narratives—teachers’ stories of their own experiences—is increasingly being seen as central to the study of teachers’ thinking, culture and behaviour” (cortazzi, 1993, p. 5). likewise, narratives from students about their experiences of learning are “an important and powerful dimension in our pedagogical thinking” (knowles & holt-reynolds, 1991, p. 187). my journey into listening to student narratives began when a former student named mindy emailed me reporting that she was in an intensive spanish language program in guanajuato, mexico. mindy added that the issues talked about in the intercultural communication course that she had taken with me in her senior year at college had helped inspire her to live and study in another country. i suspected that hearing narratives like mindy’s would inform future teaching and learning in my intercultural communication courses. more specifically, i sought to hear from former students about their experiences with or understanding of diversity prior to taking my class; perhaps what sparked their interest in the course initially? i wanted to know from students their recollections of intercultural competence, specifically points of knowledge, skills, dilemmas or challenges as well as their attitudes about and motivations to learn about intercultural communication that each had experienced while taking my class. moreover, i was especially interested in if and how former students encountered diversity issues after taking the course, and whether issues covered in the classroom became realities in their personal and/or professional lives. with such questions in mind, i set out to design my project. ii. method. a. preliminary e-mail contact. the criteria i used for contacting former students as participants in this project included that each must have graduated, and that each had been a student in an undergraduate intercultural communication course that i had taught during the time period of fall 1994 and summer 2004, which constituted my first ten years of teaching the course. i wanted former students who had had a variety of personal and professional experiences with intercultural communication and diversity. personal correspondence with several of my former students was also considered as a way to contact participants. i also used a snowball sampling technique where students and other colleagues recommended participants for the project. in order to obtain a participant’s correct contact information i utilized my department’s alumni database. i first contacted participants via email or phone and explained the project. i requested that if interested to send me back a “letter” e-mail their preliminary comments. first i asked each former student to provide a background on oneself that explains one’s experiences with or understanding of diversity prior to taking my class in intercultural communication. with this request i was specifically interested in the individual student’s story concerning first experiences with cultural difference. i then asked that each former student share points of knowledge, skills, dilemmas or challenges as well as any motivations to learn more about intercultural communication experienced within the intercultural communication course. finally, i wanted dillon, r. journal of the scholarship of teaching and learning, vol. 8, no. 3, october 2008. 39 each participant to comment on how he or she encountered diversity issues after taking the course, and whether any issues covered in the intercultural communication classroom became realities in one’s personal and/or professional lives. with a small but significant research grant from my university, my goal was to carry out correspondence with eight former students followed up by a personal interview. b. personal interview. once i had received e-mail responses i arranged for a follow up interview with each former student participant to further explore the role intercultural competence plays in their experiences with diversity and intercultural communication. although i did not use the term “intercultural competence” in the interviews, i specifically looked for elements of intercultural competence –knowledge, skills, and motivation – as a means to access what they related to me in their responses. the interviews were semi-structured in that some questions were asked of all former student interviewees with freedom allowed for interview participants to talk about other important issues. extensive notes were taken by the researcher during the interviews and checked for accuracy by repeating back to the interviewee at the end of the interview questioning. data from the interviews were analyzed using glaser and strauss’s (1967) constant comparative method for themes and categories. themes for the interviews were compared against themes acquired from the emails written by the students in order to insure qualitative research that is rigorous (taylor & trujillo, 2001). iii. results. interviews with eight former students of my intercultural communication course were obtained. these eight students -five females and three males -were in my intercultural classes at different times over a ten year period (fall 1994 semester through the summer 2004 term). no two students shared the intercultural communication class together. i selected these particular former students, not only for reasons of accessibility, but also for the variety of life experiences they represented since graduation. at the time of interviewing five of the participants resided in the midwest united states and interviews with these individuals were conducted in person. the remaining three participants lived on the east, west, and gulf coasts of the united states and were interviewed over the phone. participants were first asked to recall experiences growing up and their understanding of diversity prior to taking the intercultural communication course. all eight participants recalled vivid early experiences with intercultural communication and diversity or the lack of it. most experiences occurred through school friendships. mary recounted a “comical” early memory of one her elementary school friends: i remember being at my black girl friend’s house and seeing her mom putting oil in her hair, and i thought what was that all about? my mom doesn’t put oil in her hair. i don’t put oil in my hair. we’re always trying to get the oil out of our hair and here they are putting on oil. for hans, as a u.s. american growing up in germany during his first seven years his earliest memories were about being the “outsider.” “i was the foreigner and the one who was speaking different.” in another example, hans recalled sharing the church building with time dillon, r. journal of the scholarship of teaching and learning, vol. 8, no. 3, october 2008. 40 designations for services. “at 9am the italians met, and at 10 the germans had the building for their services, and then at 11am we americans had the building for our church services.” stacy recounted, “there was a black guy we all knew that i went to school with and was my friend. but joey was often personalized. he was separate from being black. he was just joey.” for nora her earliest memory of interacting with someone from a different race was one where others immediately pointed out differences, but due to her youth and innocence she did not comprehend. i lived in flint, michigan during my kindergarten year and my best friend was black. my aunts would says things such as, “do you notice anything different between your friend and you?” they had to prompt me to acknowledge the skin color difference, but i was only five at the time. mindy noted that her first experience with someone from a different race was when she was best friends with an african american girl from the third through sixth grade. however, the friendship ended when each went to separate schools. meeting and interacting with people from different cultures occurred throughout childhood and her teen years. “i was always intrigued with people who were different from me.” in fact, mindy embraced being around others who speak different languages and dialects and are from other cultures. for example, shortly after my interview with mindy she took a job with a different company and has transferred to australia. kurtis was unique among the eight participants because of his previous military experience in japan. although kurtis said that language proved a barrier to interacting much with the japanese he was able to travel the country and recounted experiences with communicating with the locals. one time in particular, he and a friend were hiking near mt. fuji and were invited by a japanese couple to a karaoke bar. “we got up there and sang songs from top gun 2 [film soundtrack]. the place went crazy. we had a really good time.” for the remaining student participants sustainable diversity interactions did not occur until each had entered college. christopher said that he traveled extensively with his family around the u.s. and canada and even went on a class tour of europe while in community college. “i knew on an intellectual level that there were people in the world much different than me. even that knowledge, my daily contact was with people, who for the most part, had the same background and beliefs that i did.” for marlo, similar to what she suspects was the case for many of her other contemporaries at her university, her first memorable accounts with diversity occurred after she had left the place she was raised. she shared that “my hometown [in northwest arkansas] was too small and rural for there to be a lot of diversity. it wasn’t until i went to college that i made my first black friend. not to mention, my first indian friend, and asian friend.” another area of discussion asked the participants to share important points of knowledge, skills, dilemmas or challenges as well as any motivations to learn more about diversity and communication experienced within the intercultural communication course itself. specific knowledge and skills were recalled from the course such as how language and culture are intertwined, different ways that nonverbal communication can be interpreted, cultural identity issues, identifying culture shock, masculine and feminine communication styles, and how cultures differ on a variety of distinctions including whether the needs of individual or the needs of the group are emphasized. 2 student reference top gun motion picture soundtrack; top gun was directed by tony scott and released 16 may 1986 (usa). dillon, r. journal of the scholarship of teaching and learning, vol. 8, no. 3, october 2008. 41 all eight students reported arriving to class with certain, but not similar expectations. they realize that culture is complex. for nora, one of the main things she recalled from the course was how her views about diversity in the u.s. changed. i do believe that the course was life-changing for me. especially it made me question my assumptions about how/what i knew about the u.s. i had superficial ideas about intercultural communication before taking the course. i didn’t have much understanding of u. s. american culture. it was too pervasive and i was a part of it, but the course challenged my notions of u.s. americans. i hadn’t comprehended the differences of cultures in the u.s. until i took the course. participants commented that the course served as a mirror reflecting back on one’s own culture. for kurtis, it required him to take a closer look at his own identity. “the military, i later learned in class, has a culture of its own and that i was a part of that culture.” for hans the course help prompt him to question his choices about not only his professional goals, but also the decisions he was about to make his personal life and relationships. studying intercultural communication during her last year in college further confirmed for mindy her goals of registering for a language immersion program and living in a foreign country. the participants remembered specific lessons and points from the course. marlo recounted a lesson from one of the first days in class for a lecture on the different metaphors for diversity. the image that stuck with me all these years was that of a tapestry. each person was a thread in the fabric, interconnected and creating the bigger picture, while still being affected complimented, overshadowed, or interweaving with threads of different color, texture, and feel. i remember this discussion vividly because the imagery was so easy to picture in my head. the world was a big tapestry of people, all individuals, but all connected and associated by the other people who surround us. i knew i would like the class after that first day. other participants also talked about what they specifically learned in the intercultural communication course, and how this continues to play a role in their current workplaces and professional lives. mary acknowledged: one thing that i recently recalled from intercultural class is that we are citizens of the united states; u.s. americans and i thought about this the last couple of days working with the group from canada (barrage) who came through st. joseph on tour and their performance last night. i just was more aware on how i interacted with them. it made me ponder, and i knew that they were happy to be back from their performance in hong kong, they were not quite home. the u.s. is not canada. it just made me aware of differences and the way we interpret the world. christopher confirmed the importance of intercultural communication and the competency that goes with being a better communicator. during that semester and long after the class was over, i slowly began to recognize people around me who were not only raised in dillon, r. journal of the scholarship of teaching and learning, vol. 8, no. 3, october 2008. 42 a different environment, but also recognized that they held viewpoints that were quite separate from mine. they held onto their beliefs—sometimes quite passionately—and yet i could still find common ground with them. if nothing else, the class began opening my eyes to the similarities we all share as humans. in the final part of the interviews participants were asked to reflect on how their experiences and how their thoughts and actions specifically when it comes to intercultural communication competence played a role in their lives since graduation. as mentioned previously for mindy the course helped contribute to her making up her mind to enroll in a spanish language immersion program in mexico. she also has had to monitor and adjust her communication style due to her work at an export company where a majority of her colleagues are from the netherlands. instead of communicating in an informal way, typical of many u.s. americans, mindy reported, “i had to learn that formality is important.” it was not just knowing this, but also being able to carry this knowledge out through a skill such as how to communicate effectively and appropriately through online and email interchanges. she has also had to effectively work through the stereotypes that europeans have of u.s. americans, especially females. nora remarked that through her position as the executive director for the county arm of a national political party that motivation to learn about the changing racial and social demographics in her community has been made significant by a growing hispanic bloc of voters. this development has consequences not only for her own knowledge and skills of working effectively with this new voter demographic, it also has direct consequences in turning out the vote for nora’s political party. nora added, “i would like to think that the class and what i have learned thus far also has a positive impact on the way i treat anyone who comes from social categories different from my experience.” after graduation, kurtis was accepted to officer training school eventually making second lieutenant in the united states marine corps. the position requires not only deployment to other parts on the world, but more frequently communicating with fellow officers, superiors, and recruits from different parts of the country. for kurtis meanings can dangerously be misconstrued, and the way one talks with people from other cultures as well as those individuals from one’s own culture is a persistent lesson of intercultural competence. in communicating you must exercise patience and understanding. they might misinterpret what we say. you have to keep this in mind. english is a hard language to understand, there are so many different meanings. it is important that we take the time to clarify ourselves about what was said and how we can make ourselves clearer. while living in los angeles lessons of intercultural competence acquired by hans consisted of learning about what he identified as “subcultures within subcultures.” when pressed with what he meant by this statement hans explained, “well in the midwest, if they are puerto ricans, they are puerto ricans, but in la there were groups within groups. these people talk with these people. and there were distinctions within racial, ethnic groups. it may depend on where you live—neighborhood, skin color.” hans emphasized that categories cannot be assumed and that if he wanted to be successful as a salesperson he had to work through not only cultural affiliations, but also group membership, and a person’s own individual needs. dillon, r. journal of the scholarship of teaching and learning, vol. 8, no. 3, october 2008. 43 the knowledge, skills, and motivation required in communication and intercultural competence plays a role locally as well. christopher remained in the area and is a recruiter for a massage training school. multiculturalism is a part of everyone’s life and learning to deal with diversity issues is a lesson of intercultural competence that one must face whether they are in another country, large u.s. american city, or a small midwestern city that on the surface may not seem to have a lot of diversity, but does in its own way. christopher commented: the class did not equip me with the ability to deal with every type of human being on the planet. rather, it was an introduction to the idea that people come from all different types of backgrounds and belief systems...when someone comes into my office and shares his or her life experience with me, those experiences may be drastically different than my own. i may or may not agree with that person or be able to understand where she is coming from. but regardless of where that person has been or want they believe, i can listen and learn something from him. one interesting result reported by the interview participants was how social economic class became more of an issue for them especially in their lives for more than any of the former students had expected. for example, stacy told how she was shocked when in her student teaching post in a small rural community how teachers were “idolized” because “the teachers in town were the only ones who could pay their bills.” stacy continued: for a long time i always thought diversity was just a black/white issue, but it is more. one of the eye openers in my teaching experience is the culture of being poor.” students did not have a stable home life. they would go from home to home. maybe mom would have a boyfriend and they would stay at his house for a while. at our school 80 percent of kids were either on free or reduced lunch. again, teachers were idolized. the children were so needy. they would attach to you, perhaps because i seemed stable. i had great parents there, but the kids needed more. finally, regarding overall lessons concerning communication and the knowledge, skills, and motivation that contribute to intercultural competence, hans added that in any culture it is probably important to “talk less than 50 percent of the time and just mainly listen.” people will tell you things. you just have to learn to shut up. even though we are so uncomfortable with silence in this culture, you have to allow them to talk and you listen. also, realize that how different they may be, every other person has stories just like you. try to learn about them, to understand where they are coming from. people have different experiences and stories, even if their lives seem to parallel yours. when it comes to learning about communication and intercultural competence hans could not have said it better. iv. discussion. the results found in this research project confirm what kohls (2001) maintains that one of the building blocks of diversity is to understand one’s self and culture better. participants were dillon, r. journal of the scholarship of teaching and learning, vol. 8, no. 3, october 2008. 44 quick to tell their personal stories about early experiences and understanding of diversity. because it is essential for students to realize that they are already part of a multicultural world, there is a need to personalize one’s participation in this multicultural world. therefore, incorporating early in the intercultural communication course an assignment for students to safely talk about their early experiences and realizations of culture, communication, and difference is an effective tool for students to personalize issues. it is also a way for all to find common ground in that everyone has a culture and everyone probably has a story to tell about initial experiences with cultural difference. likewise, hearing stories from others, especially those from different cultures and backgrounds, is an effective way to learn about diversity and intercultural communication. sharing these stories are not only fun (childhood encounters often do result in amusing accounts), but also can be educational in discerning what lessons of intercultural competence (knowledge, skills, and motivations) can be learned from these stories. participants conveyed how the course provided them with a new perspective regarding intercultural communication, not only in relation to diversity found around the world, but also within the united states. what was intriguing was how each participant transformed what was talked about in the course and applied it to their own intercultural competence. paul ricoeur (1981) asserts, “our deeds escape us and have effects which we did not intend” (p. 207). not only do we interpret what we say and do, but others incur their own interpretations, and arrive at their own learning. marlo said she was significantly impacted early in the course with class discussions on how diversity could be visualized through different metaphors. marlo especially liked the diversity metaphor of a tapestry where “each person was a thread in the fabric, interconnected and creating the bigger picture, while still being affected complimented, overshadowed, or interweaving with threads of different color, texture, and feel.” this tapestry metaphor, according to marlo, continues to provide a basis for her learning about diversity. what i had thought of was a minor activity in the intercultural communication course was for marlo the trigger for understanding what constitutes cultural diversity. in assessing my current teaching of intercultural communication, i realized i had dropped such discussions about metaphors of diversity, substituting other points and issues. listening to marlo has prompted me to reevaluate my teaching. perhaps i should add back in these metaphors of diversity? indeed, what may be trivial to the instructor may in turn serve as a powerful learning point for the student. in the intercultural courses i have taught since undergoing this project i have required students to become more participatory and have encouraged them to complete in class projects that call upon each to share a story. throughout the course, as well as on the final exam, i ask students to reflect upon the stories that were shared in class. often students report that this is where learning about cultural identity and diversity is at its most vibrant. i have not done a thorough assessment yet to determine any particular learning outcomes, other than what is self reported by students. however, such activities do have potential for further questions and study regarding the scholarship of teaching and learning in the intercultural communication course. pedagogy on diversity and intercultural competence continues to evolve and these interviews have also led me to reconsider what new areas i should concentrate on more in my intercultural communication course. one of these areas is the topic of social class. dealing with the issues surrounding social class became important for almost all of the participants whether it was in their personal or professional lives. in some cases it became a surprising issue of diversity dillon, r. journal of the scholarship of teaching and learning, vol. 8, no. 3, october 2008. 45 that they had never directly encountered before or even had anticipated. when asked what issue of diversity will need to be focused on in the future and likewise should be emphasized in intercultural communication courses, six of the eight participants put social class issues at the top. keshishian (2005) notes that “with respect to intercultural relations, it is entirely plausible that the homeless person in the u.s. will have more in common with another homeless person in guatemala, say, than with the ceo of general motors” (p. 215). this statement captures the sentiment, that even though two people may be from the same national culture, with the same skin color, who in other ways would be assumed to be similar, may in fact be worlds apart due to different social and economic class backgrounds. social class in my past intercultural communication courses has not received the same attentiveness as does other forms of diversity (e.g. race, ethnicity, national origin, sex, sexual orientation, age, disability). likewise, current textbooks in intercultural communication often do no adequately address social class issues. (again, see keshishian (2005) for an expansive critique on this topic). this new awareness regarding social class issues has made me “retool” my coverage and assignments in my intercultural communication courses, and i have added a section in my graduate level communication and diversity in the workplace course that focuses on social class issues. assigned readings include brenda allen’s (2004) book difference matters: communicating social identity, alan johnson’s privilege, power, and difference (2006), as well as hidden dimensions of class in the workplace published in 2002 by authors ruby payne and don krabill. doubts about diversity and interacting with others from different cultures would not be part of one’s personal and professional life after graduation were quickly put to rest by all eight former students. each was enthusiastic to share one’s personal story. each had powerful lessons pertaining to intercultural competence. however, i felt guilty that i was the only one hearing them. to help with this i have shared these eight alumni experiences with diversity and intercultural competence by incorporating them into course lectures and discussions. it is clear that these former students in the spirit of the scholarship of teaching and learning had become my teachers and this education should be shared with current students in my intercultural communication courses. for example, here is a story from mary about her current struggles with age diversity in her workplace that current students will likely soon identify with: age has become a real diversity issue in my job here. everybody i work with is at least 30 years older than me. for example, later today i’m leaving for a conference in indianapolis with two males who are 70 plus and i’m 22! i just can’t make age an open issue, but it is there. i was never treated like a kid growing up, never baby talked to, and i think this kind of helps me look at the world as an adult. the conference in indy—it’s my first one-is where we choose the shows for the next season, meet with other presenters and artists, work out scheduling of tour dates, there are featured artists performing. it goes on for 3 days. now back to the age issue – there is the issue of deciding what to wear for this conference. i mean i’m 22 and i don’t want to pack things that make me look like a teeny bopper, but then i don’t want to look too old, when i’m really not. i mean i’m not 40 and i can’t go out and get and wear a coordinating suit/pants outfit. dillon, r. journal of the scholarship of teaching and learning, vol. 8, no. 3, october 2008. 46 mary and her fellow alums from the “real world” of intercultural competence have been invited back into my classroom to share their knowledge, their skills, and their motivations about intercultural communication competence with my current and future students. some alumni have been able to make it back personally to talk with students during a departmentally organized “communication week with alumni.” often these interactions produce additional ideas for collaboration, internships, and instruction in courses such as the intercultural communication course. it is my hope that these interactions can become more frequent. realistically, alumni lead busy lives and cannot personally be in the classroom on a regular basis. however, seizing opportunities such as interviewing and recording alumni on topic areas, to be later integrated in a class, is something that i anticipate doing. once such a project is realized, further study could be conducted on how these interviews affect students’ learning outcomes. one of this study’s strengths is that it allowed for personal in depth perspectives from former students. one on one interviews provided opportunities to probe for additional observations from the interviewee. in addition the interviews provided a format for reestablishing connections between the teacher and former students. unlike what is traditionally requested of alumni--that is to provide financial support for their alma mater--this project asked students to give back to their alma mater through the sharing of their personal stories that can inform teaching and learning. as diamond and kashyap (1997) observed in their study of reasons why alumni make contributions, attachment to one’s university goes beyond supplying and meeting financial needs. alumni want an attachment to their alma mater that also includes one’s time and expertise. current students also benefit from learning from those who had sat at the desks and in the classes before them. limitations are a factor in every study, and this one is no exception. all of the interviewees were white. it would be expected that a minority students would have their own unique perspective about growing up, attending college where they are in the minority, and their experiences with intercultural competence after graduation. another limitation is that the questions were formulated by the researcher/instructor with little consultation from former or current students. soliciting questions from former students could provide a richer base for information on diversity and culture and how it is influencing one’s personal and professional life. a. implications for the future. hutchings and shulman (1999) propose, “a scholarship of teaching is not synonymous with excellent teaching. it requires a kind of ‘going meta,’ in which faculty frame and systematically investigate questions related to student learning” (p. 13). questions for further study should be guided by a systematic inquiry or focus into student learning called for in the scholarship of teaching and learning. these questions are: what is happening in the classroom? what would the student learning look like? what sort of evidence can i gather to document that student learning is occurring and providing proof it took place? additional questions to consider include: is remembering course content a measurement of learning. if students do not remember content does it mean learning did not occur? do interviews and personal narratives provide enough evidence students have learned or do i have to see the evidence myself? how can we use these results of learning beyond the classroom? bass (1999) “the movement for a scholarship of teaching seeks first and foremost to legitimate a new set of questions as intellectual problems.” the current study has led the author dillon, r. journal of the scholarship of teaching and learning, vol. 8, no. 3, october 2008. 47 to consider what could be done in the future regarding alumni interviews, intercultural competence and the scholarship of teaching and learning that informs teaching and learning beyond the parameters of the current study. some of these ideas have already been shared in this paper. one other suggestion is to broaden the scope of the questions asked of participants. this can be done by asking current students what they would like to know from former students who have been out of college, two, five, ten, and 15 years plus. what lessons did they learn? what advice would they give to a current student? this advice could be on issues of culture and diversity, but questions could also be about knowledge, skills, and other forms of competence important to being successful after graduation. current students and alumni could engage in discussions about how it is like out in the “real world”. both identify with one another due to similar interests, experiences in a major academic program, and careers. discussion could be expanded to include how alumni juggle work schedules with pursuing further education, balancing job and family, and managing student loans and debt and how one’s personal and professional choices affects one’s daily life. as has been noted each participant in this study was enthusiastic and willing to provide his or own story. the methodological form of narrative provides a way to interact with former students and capture their perspectives on intercultural competence. additional interviews with other former students are needed. one suggestion for achieving this task on a broader level is to teach a methods course in narratives where current students can engage in interviews with former students who are now alumni of the university. in addition to being a valuable resource for learning more about the intercultural issues presented in this study talking with former students serves as a communication bond between alumni and the university that should be further tapped. references allen, b. j. (2004). difference matters: communicating social identity. long grove, il: waveland press. bass, r. (1999). the scholarship of teaching: what’s the problem? inventio 1(1). bateson, m. (1997). “foreword.” in learning from our lives: women, research, and autobiography in education. edited by a. neumann and p. peterson, pp. vii-viii. new york: teachers college press. boyer, e. (1990). scholarship revisited. washington, dc: carnegie foundation. clandinin, d., connelly, f. (2000). narrative inquiry: experience and story in qualitative research. san francisco, ca: jossey-bass. cortazzi, m. (1993). narrative analysis. london: the falmer press. darling, a. (2003). scholarship of teaching and learning in communication: new connections, new directions, new possibilities. communication education, 52(1), 47-49. dillon, r. journal of the scholarship of teaching and learning, vol. 8, no. 3, october 2008. 48 diamond, w. d. & kashyap, r. k. (1997). extending models of prosocial behavior to explain university alumni contributions. journal of applied social psychology, 27(10), 915-928. elbaz, g. (1990). knowledge and discourse: the evolution of research on teacher thinking. in c. day, m. pope, & p. denicolo (eds.), insights into teacher thinking and practice. london: falmer press. glaser, b. g., & strauss, a. s. (1967). the discovery of grounded theory: strategies for qualitative research. new york: aldine de gruyter. goffman, e. (1975). frame analysis: an essay on the organization of experience. harmondsworth: penguin. goffman, e. (1981). forms of talk. oxford: blackwell. hantzis, d. m. (1995). “performing experience(s)/shifting to self.” paper presented at the speech communication association convention, san antonio, tx. hutchings, p. & shulman, l. (1999). the scholarship of teaching: new elaborations, new developments. change 31(5), 11-16. illinois state university (2007). scholarship of teaching and learning. retrieved august 30, 2007 from http://www.sotl.ilstu.edu. johnson, a. j. (2006). privilege, power, and difference. (2 nd ed.). boston, ma: mcgraw-hill. keshishian, f. (2005). a historical-materialist critique of intercultural communication instruction. communication education, 54(3). 205-222. knowles, j.g. & holt-reynolds, d. (1991). shaping pedagogies through personal histories in pre-service teacher education. teachers’ college record, 93(1), 87-113. kohls, l. r. (2001). survival kit for overseas living (4 th ed.). yarmouth, me: intercultural press, inc. lustig, m. w. & koester, j. (2005). intercultural competence: interpersonal communication across cultures (5 th ed.). boston, ma: allyn and bacon. payne, r. k. & krabill, d. l. (2002). hidden rules of class at work. highlands, tx: aha process, inc. ricoeur, p. (1981). hermeneutics and the human sciences. cambridge, uk: cambridge university press. rosenzweig, r. (1999). the scholarship of teaching: two suggestions and one caution. inventio 1(1). dillon, r. journal of the scholarship of teaching and learning, vol. 8, no. 3, october 2008. 49 shulman, l. (2004). visions of the possible: models for campus-support for the scholarship of teaching and learning. (pp. 9-23). in becker, w. e. & andrews, m. l. (eds.) the scholarship of teaching and learning in higher education: contributions of research universities. bloomington and indianapolis, in: university of indiana press taylor, b. c., & trujillo, n. (2001). qualitative research in organizational communication. in f. jablin & l. putnam (eds.). the new handbook of organizational communication, (pp. 161-194). newbury park, nj: sage. theall, m. & centra, j. a. (2001). assessing the scholarship of teaching: valid decisions from valid evidence. (pp. 31-43.) in caroline kreber (ed). scholarship revisited: perspectives on the scholarship of teaching (new directions for teaching and learning –series. san francisco, ca: jossey-bass. v10n2kaufka journal of the scholarship of teaching and learning, vol. 10, no. 2, june 2010, pp. 25 33. beyond the classroom: a case study of first-year student perceptions of required student-faculty conferences beth kaufka1 abstract: instructors of first-year students must understand that many of the frustrations they encounter with this demographic are a normal part of a transition process. not only are first-year students asked to adjust to new academic demands, they also must learn how to live independently in a new environment. an effective way to facilitate academic development is to require conferences that: 1) address the student as a person within the first-year context, and 2) focus on course content and academic development. the former is initially prioritized over the latter to meet the specific needs of first-year students. in the fall semester of 2009, the author conducted a case study with three sections of critical writing, an interdisciplinary general education critical thinking and writing course. students met with the instructor throughout the semester, and at the end of the course, they completed a 25-question in-class anonymous, multiplechoice survey to gauge their perceptions of the impact of required conferences on their academic experiences. the data show that students very clearly perceived the conferences as helpful for their learning and overall experience of the course. keywords: freshmen, first-year students, required conferences; studentfaculty interaction, retention, transition, best practices, general education, office hours, engagement. i. introduction. often, when i tell new academic colleagues what i do and what i love – “i teach first-year general education” – i am met with groaning, exasperation, and frustration. they cringe, shrink back into themselves and commend me for having the patience to deal with students at the introductory level. then, typically, they launch into dissertations about how unprepared first-year students are, how freshmen don’t or can’t read, how they demonstrate all the signs of academic apathy: tardiness, sleeping in class, cell phones, incomplete homework, poor grades. they stare in disbelief when i say that i truly love teaching freshmen, that i find it a highly rewarding experience filled with opportunities, and that really, when it comes down to it, i find it an honor and privilege. first-year students are unique in that, although they have expectations about what their college experiences will be like – a blend of mythologies from the media, their friends, and their own desires – many of them have little to no experience with college life and are flabbergasted by the reality of their first year. as instructors of first-year students, we are able to help usher these students into our institutions, these foreign lands where many of them are on their own for the first time. we are able to witness their transformation from high-school teenagers to college students exploring their new lives. at the same time, we have to be careful – much of the 1 division of university studies, north carolina agricultural and technical state university, 1601 e. market st, greensboro, nc 27411, beth.kaufka@gmail.com. kaufja, b. journal of the scholarship of teaching and learning, vol. 10, no. 2, june 2010. 26 www.iupui.edu/~josotl discourse around first-year students is filled with all too romantic notions of transformation. we also need to be realistic about the unique challenges these students present. students new to a university environment have a diversity of academic and non-academic obstacles to overcome, particularly in their first semester. i have been teaching first-year writing in various capacities for the past nine years. to help address the specific needs of first-year students, i require two conferences during the semester and structure them into my syllabus and grading scale. my experience with writing center pedagogy has informed my desire to work one-on-one with students, but more anecdotally, in the years i have required conferences, i have found that, if i make a personal connection with students early on in the semester, they are more likely to return to office hours when they need help with course content. my two conferences are semi-structured. the first acts as an icebreaker, the second as personalized academic support. in the fall semester of 2009, i conducted a case study with three sections of critical writing, an interdisciplinary general education critical thinking and writing course. students met with me throughout the semester, and at the end of the course, they completed a 25-question inclass anonymous, multiple-choice survey to gauge their perceptions of the impact of required conferences on their experiences. the data show that my students very clearly perceived the conferences as helpful for their learning and overall experience of the course. required conferences can be effective in any discipline and, although this paper specifically discusses first-year students, may be implemented at any level. ii. what we know. over and over the research shows that students find it meaningful to build relationships with faculty. richard light (2001), in making the most out of college, reports that students “point out repeatedly that getting constructive, somewhat personalized advice might be the single most underestimated feature of a great college experience” and how “certain faculty members make a special difference in their lives” (p.5). in one of his studies, light also found that four-fifths of students reported that the specific, critical moment that profoundly changed them happened outside the classroom (p.8). similarly, the first principle in chickering and gamson’s (1987) highly referenced “seven principles for good practice in undergraduate education” is contact between students and faculty. as these authors argue, “frequent student-faculty contact in and out of class is an important factor in student motivation and involvement. knowing a few faculty members well enhances students' intellectual commitment and encourages them to think about their own values and plans” (chickering and gamson, 1987). student-faculty interaction is most often manifest through academically common practices: collaboration on research projects, independent studies/special projects, departmental or university committees, student organizations, advisement, and office hours. for example, the national survey of student engagement (nsse) findings support high levels of student-faculty interaction. “working with a professor on a research project or serving with faculty members on a college committee or community organization lets students see first-hand how experts identify and solve practical problems. through such interactions teachers become role models, mentors, and guides for continuous, life-long learning” (nsse, 2008). although such avenues are well traveled and can positively impact students, these interactions typically engage a certain type of student, one who is already prepared and committed to high academic engagement. traditional practices of student-faculty interaction are kaufja, b. journal of the scholarship of teaching and learning, vol. 10, no. 2, june 2010. 27 www.iupui.edu/~josotl often seen as rewards and are actively sought after by high-achieving, engaged students. they are largely voluntary. requiring conferences for all students is an attempt to provide the opportunity for increased student-faculty interaction to all enrolled in the course, to mandate engagement for everyone. in the conferences, the students who sit in the back and avoid eye contact are required to chat about the class, their experiences, and what they need to improve the conditions for their learning. in an interview, john n. gardner (2003), highlights mandatory interventions. he says: “i think we’ve gotten too focused on treating them as independent adults and letting them sink or swim on their own…but the older i got, the more i realized that my students…were more likely to be successful if i made them do certain things. once the students did them, they found out that they were helpful, and then they pursued them voluntarily. one of the things we’ve learned from the instrument your first college year…is that students recognize they need help and they report all kinds of stress, but do they go get the help? no, they don’t. a best practice would be to mandate student participation in more of these interventions” (p. 12). light (2001) reports that students who seem to grow most significantly in the higher education experience include interacting with faculty as one contributing factor; however, they also report that this interaction is difficult because it doesn’t always come naturally (p. 10). i’d bet most faculty hold office hours, but i’d also bet that overall, the number of students who attend are low – or at least until it’s nearly too late and the student’s learning (and grade) is in jeopardy. seeking out faculty for help can be especially challenging for first-year students who might expect faculty to be unavailable, who might be intimidated by their professors, who might have poor time management skills and therefore do not make the effort of attending office hours, who might be overwhelmed with their transition into college life. tim clydesdale (2007), author of the first year out: understanding american teens after high school, found that "most of the mainstream american teens [he] spoke with neither liberated themselves intellectually nor broadened themselves socially during their first year out." rather, he discovered that what teens really focus on during their first year is basic daily life survival on their own, i.e. how to negotiate new friends, money, time management, dorm life. in other words, rather than experiencing the mind-blowing, life-changing class that reorients the student’s whole world view to an informed, academic one – the kind of experience we see in movies like wonder boys – students are occupied with worries about finishing their homework and getting to class on time when they need to replace the pre-paid cafeteria id card they lost, not wanting to call home to tell their parents they need help, and all of this in the midst of a new group of friends, mere strangers. this reality for first-year students is reflected in upcraft and gardner’s (1989) definition of freshman success. they “believe freshman succeed when then they make progress toward fulfilling their educational and personal goals: 1) developing academic and intellectual competence; 2) establishing and maintaining interpersonal relationships, 3) developing an identity, 4) deciding on a career and life-style; 5) maintaining personal health and wellness; and 6) developing an integrated philosophy of life” (p.2). telling is the fact that all but the first of upcraft and gardner’s goals focus on psycho-social personal growth of first-year students. it is also worthy to tangentially note that student perceptions of such interactions can increase retention (o’gara et al., 2009). students who have that extra support feel more connected to the university and are motivated to do what it takes to remain in the institution. kaufja, b. journal of the scholarship of teaching and learning, vol. 10, no. 2, june 2010. 28 www.iupui.edu/~josotl jaswal et al. (2008) note that “retention research shows that the earlier a student is connected to the social and academic systems of the college the greater their academic achievement and thus their commitment to graduating” (p. 55). retention of freshmen not only benefits the students themselves, but also, from an administrative standpoint, benefits the entire institution. iii. case study. as noted above, in the fall semester of 2009, i conducted a small case study with my three sections of critical writing, a critical thinking and writing course in the university studies interdisciplinary general education core curriculum at north carolina agricultural and technical state university (a&t). north carolina agricultural & technical state university is a historically black university in greensboro, nc known for our civil rights legacy, particularly for our a&t four – the four freshmen who refused to get up from the woolworth’s whites-only lunch counter. currently, our first-year students are 91% african american, 52% female, and largely from north carolina at 78%. the average age of our first-year students is 20-years-old (wabash, 2009). although a&t first-year students fit the standard prototype for “freshmen,” (18-years-old, just out of high school, full-time students who reside on campus, attending a four year university), it should be noted that nationwide, “the student landscape is dotted with almost as many profiles as there are students, and our thinking about first year students should reflect their diversity…the average age is slightly over 25 (ishler, 2005, p.17). overall, first year students are diverse in age, class, race, ethnicity, sexual orientation, preparedness, etc. our course objectives include critical writing and reading skills, effective communication, research, use of technology, assessment, and collaboration. as a required part of the curriculum, students meet with me twice during the semester for one-on-one meetings, the first at the very beginning of the course, and the second anytime thereafter. to gauge student perceptions of the impact of these meetings on their learning, i gave students an in-class anonymous multiple-choice survey, consisting of 25-questions. a. conference one. the first required conference is a simple, friendly get-to-know-you meeting (unless the student wants to work on course content), and happens within the first two weeks of the semester. i bring a sign-up sheet to class and pass it around so students can sign up for a fifteen-minute conference. in this conference, i do not prioritize academic issues because i really just want to get to know my students. my objectives, made explicit, for the first conference are: • to build trust with students by getting to know them on a personal level and allowing them to get to know me too (not only as faculty, but also as a “real” person); • to be sure my students know where my office is and to tell them on a one-on-one level that i can and should be used as a resource; • to be sure students understand the objectives of the course and to answer any questions they might have early on. although we as instructors, of course, want our students to develop in academic course content and process, it is critical that we recognize the fact that academics might be the last thing on their minds and that this is a normal part of transforming into college students. as professionals, we can help alleviate some of the burden of their transition and ultimately facilitate academic success. kaufja, b. journal of the scholarship of teaching and learning, vol. 10, no. 2, june 2010. 29 www.iupui.edu/~josotl a study done by palmer et al. (2009) shows that many first-year students feel as if they don’t belong at the university because they reside in a liminal “betwixt space” (between their home-space and the university-space) and that a variety of complex social-interpersonal interactions influence how well they transition more fully into the university space. one thing we can do as instructors is demystify the institution by giving the institution a human face, a kind of mentor or in-between figure to help them bridge the gap. shanley and johnston (2008) state that for first-semester college students, success is found not only in their organizational abilities and time management skills, but also in knowing their professors [my emphasis]. from my conversations during these meetings, i have discovered that first-year students typically expect two types of college professors: the hard-edged, unforgiving, stern professor, an immovable stone wall of knowledge and high expectations or the sloppy liberal-minded professor who doesn’t take attendance, is slack with assignments, has fun and engaging lectures, and leaves class satisfied at her/his own brilliance. both of these kinds of stereotypical professors do not demonstrate the care that freshmen need and ultimately desire. both prioritize easy teaching and their own careers, and both unfortunately fail to demonstrate real concern for student learning and tend to represent a cold institution that sees students as numbers. so, during the initial required conference, i offer them a different kind of professor. i start with personal, but not prying, questions about their college experiences so far. i ask questions like: how are you? how’s the transition to college? from where did you move? what do your friends and family think about you being in college? what is your roommate like? what is your dorm like? what is the hardest thing you’re dealing with now, and how can i help? do you have a major? what classes are you really excited about? i believe it is fair that while i ask students to disclose some personal information about themselves that i also share personal information about myself throughout the conference, although i am very intentional about not making the conference about me, my lessons, and my advice. this first conference is devoted to making sure my students feel that they have a voice, and that it will be heard. to close, i wrap it up by reiterating the objectives of the conference and ask if they have any questions about the course. and finally, i casually sum up our conversation to let my students know i have been listening and to help me solidify some of the individual details in my own mind. b. conference two. students schedule the second conference themselves at any time during the rest of the semester. i have found that after my first conference, it is fairly easy to get most students to return. in fact, many of them come regularly both to work on assignments and to just say hi when they’re in the building. because the students themselves schedule these conferences, they tend to be spread out through the semester, so i do not have to set aside such a large amount of time at once. and, to help alleviate some of the time demand, i encourage students to simply come to my office hours, when i have time set aside for them anyway. the second conference tends to be around 30minutes, like a typical writing center session. for the second conference, my objectives are different. rather than a personal conference just to get to know students, i focus on academic skills and strongly encourage them to bring in a course project to work on. (but, i do give them credit anyway if they just want to talk about something else.). in these conferences, i can explain things we discussed in class in a more tailored way to suit individual students. i can “personalize instruction” by having a more kaufja, b. journal of the scholarship of teaching and learning, vol. 10, no. 2, june 2010. 30 www.iupui.edu/~josotl interactive dialogue with them. i can ask more direct questions, allow more time for them to process, and focus our discussion more directly to their learning style. for example, if i know someone is a visual learner, we can sketch out an image, a diagram, or a graph together to elucidate concepts or make connections. or, i can draw upon their specific interests, interests i began to learn about in the first conference, and use examples/analogies that make more sense to them as individuals. most students find one-on-one help extremely enlightening; i can’t describe how fun it is to witness students’ light bulbs turn on, and how excited they become when they understand something in a new way. c. findings. i have required conferences for years, assuming they made a difference, hoping the time i invested was actually worth it; however, until this past fall, i did not collect data. although my case study is small, with only 52 participants, the findings are encouraging. survey questions 1-3 provided logistical information: 78.8% met with me two or more times, with 34.5% meeting more than the requirement. during the conferences, students reported that they mostly worked on writing (88.5%), but also ranking high were their personal difficulties and/or student life (75%) and overall academic performance (73%). 48% reported discussing academic motivation, while 32% said they worked on general study skills. 78.9% of the conferences lasted between 10-30 minutes. the rest of the survey data is included in the following table, but it is interesting to highlight some important content of the students’ perception of the conferences. 92% of students believed the conferences helped them feel i cared about them as individuals. 71% believed the conferences helped them with reading and writing skills, so much so that 82% believed the course should continue to require conferences for this course. 88.4% agreed that, generally, first year courses should require at least one conference. these numbers are obviously high, which highlights the fact that over half agreed that they might not have met with me had the conferences not been mandatory. students very clearly perceived the helpfulness of the conferences for their learning and overall experience of the course. table 1. survey results for questions 4-25 survey question strongly agree agree neutral disagree strongly disagree i feel that the required conferences were worth my time. 73.1% 21.2% 5.8% 0% 0% in general, it is important to me that i know my instructors care about me as an individual. 78.8% 15.4% 5.8% 0% 0% the conferences helped me feel my instructor cares about me as an individual. 67.3% 25% 4, 7.7% 0% 0% the conferences helped me develop my writing skills. 25% 46.2% 19.2% 9.6% 0% the conferences helped me develop my reading skills. 25% 46.2% 19.2% 9.6% 0% the conferences helped me develop my study skills. 13.5% 26.9% 44.2% 15.4% 0% the conferences helped me feel more comfortable participating in class than i otherwise would’ve felt. 15.4% 32.7% 40.4% 7.7% 3.8% the conferences motivated me to stay on task and do my coursework. 26.9% 53.8% 13.5% 5.8% 0% the conferences helped me understand course content i did not understand in class. 30.8% 34.6% 23.1% 9.6% 1.9% the conferences helped me learn about other resources. 26.9% 36.5% 26.9% 9.6% 0% the conferences helped me feel as though i am part of the a&t community. 11.5% 25% 46.2% 13.5% 1.9% the amount of time i spent on homework for this course increased because of the conferences. 11.5% 30.8% 34.6% 19.2% 3.8% kaufja, b. journal of the scholarship of teaching and learning, vol. 10, no. 2, june 2010. 31 www.iupui.edu/~josotl during my conferences, i was able to ask questions that i would not have been able to ask during class time. 46.2% 38.5% 11.5% 3.8% 0% the level of my engagement in the course materials increased because of my conferences. 17.3% 46.2% 28.8% 7.7% 0% the conferences made me more comfortable approaching my other instructors in other courses. 21.2% 38.5% 26.9% 13.5% 0% the conferences made me feel a sense of belonging at a&t. 11.5% 26.9% 51.9% 5.8% 3.8% student conferences in this course should continue to be required. 55.8% 26.9% 13.5% 1.9% 1.9% my conferences made it more likely that i stay at a&t for another year. 23.1% 25% 23.1% 25% 3.8% if conferences were not a requirement of the course, i might not have met with my instructor. 21.2% 32.7% 19.2% 19.2% 7.7% in general, first-year courses should require at least one conference. 59.6% 28.8% 4, 7.7% 1.9% 0% i feel as though the conferences were an important part of my learning in this course. 28.8% 44.2% 15.4% 9.6% 0% overall, my conferences improved my experience at a&t. 19.2% 32.7% 36.5% 5.8% 3.8% *data was collected on 12/1/2009 from beth kaufka’s fall 2009 sections of unst 110 critical writing 014, 034, 036 from 52 respondents. the degree to which my case study is generalizable is certainly questionable, largely due to my small sample size and my particular personality. furthermore, this study only assesses students’ perceptions of learning; the next step, then, is to investigate the correlation between student perceptions and actual student learning, in addition to other student-related issues such as engagement and retention. note, however, that although this study itself may not be generalizable, the survey findings are consistent with the literature on engagement and retention, as cited in the literature discussed in this article. either way, faculty should feel optimistic about this method of interacting with students, in addition to the more traditional avenues such as collaboration on research projects and serving on committees – and faculty should be encouraged to design more thorough studies of the impact of required student-faculty conferences on learning. iv. implementation trouble spots. i’ll be the first to admit required conferences are necessarily time-consuming, particularly the first round. think about it. let’s say you have 25 students in a class, and you teach three sections of the course for a total of 75 students. let’s say you spend fifteen minutes with each student, this comes to nearly 19 hours without factoring in any breaks, any meetings that run over fifteen minutes, students forgetting to come, getting lost on their way to your office, or standing you up altogether. depending on what you want your schedule to look like during the scheduled conferences, this can take an entire week or even two. you might factor in some kind of consequence for students who miss their first conference. i let students reschedule, but for only partial credit. i recommend you run through familiar lesson plans and schedule light grading during your week of scheduled conferences, in addition to scheduling around committee work; i’ve never had to cancel a class for conferences. another significant challenge with time comes from the fact that, after their first conference, after you demonstrate your capacity for support, many students will use your office hours. (some of us tend to relish our un-used office hours as a time to get work done. there are certainly times that i do!). and outside of office hours, you’ll find that your students pop in “just to say hi.” you can also expect to write more letters of recommendation for internships, jobs, scholarships, and study abroad programs. this can be at once both an honor and a burden. (i have been a reference for nine students already this semester, and we are just half way through.) kaufja, b. journal of the scholarship of teaching and learning, vol. 10, no. 2, june 2010. 32 www.iupui.edu/~josotl one way to help manage time is to give your conferences some shape by defining your goals. you will probably have larger, overarching goals like helping students develop into selfregulated, self-motivated learners. but it is helpful to have specific conference goals, and to state those goals up front with students so they understand the intentions of the meeting. do you want to focus only on course goals, for instance: clarifying the very first major assignment and helping students get started? or do you want to provide motivation and inspiration? encouragement? consider, also, structure and amount of preparation. do you want students to come with a completed assignment to discuss? will you have a time limit, and how will you stick to it? do you want to have a kind of script to ensure all students get the same kind of information and attention from you? or, do you want to just roll with it and see where the conversations lead you? whatever way, it is helpful to both you and your students to be clear about your objectives. it is also critical that, despite how much you care about your students, you maintain high academic standards and hold them accountable to course policies. when students make bad choices, they must suffer through the consequences. if you are doing these conferences in an effective way, they actually help students understand that you will hold them accountable for their actions, and they will not resent you for doing so. in fact, effective conferences can mediate the burden because students should know your expectations of them. one of my students wrote in her end-of-semester reflection essay: “the meetings gave me a better understanding of your expectations [my emphasis] and my mistakes. this only allowed us to build a connection with you as our teacher and in my opinion it showed me that you were a concerned teacher who was willing to listen to our problems, assist us, motivate us, and point us in the right direction. i guess this is why i looked forward to coming to class, because you were the first teacher to show support.” and, these conferences help mitigate the accusation that “you just don’t like me” when there is a consequence for a poor academic choice. in fact, i have never had a student lodge that complaint with me. v. conclusion. as discussed above, the research clearly shows that students value personal relationships with faculty, that good teaching and high quality interactions with faculty promote growth, and the more contact students have with their teachers, the better. in general, good undergraduate education teaching practice encourages high levels of student-faculty interaction. required oneon-one student-faculty conferences can be an effective way to positively affect student learning. required conferences ultimately become very memorable moments for students. for me, they are critically important for setting the tone for the class and helping students feel cared about in a way that enhances their learning. they often tell me about this directly, and many of them write about it in their end of the semester reflections. they write in surprise that i actually care about them. i am always both happy and saddened that this conference means so much to them – happy that i have helped them in their overall experience of their first year, and saddened that they don’t feel cared about (or respected) in many of their other courses. the data here suggest that required conferences help students get off to a good start, help facilitate in them an awareness of what kind of students they are, their learning styles, the particular challenges they have overcome or are still dealing with. i can help them understand themselves by listening to them and telling them what i think they are communicating: it sounds kaufja, b. journal of the scholarship of teaching and learning, vol. 10, no. 2, june 2010. 33 www.iupui.edu/~josotl like you are an auditory learner; it sounds like you’re saying that you miss home; it sounds like you’re saying you would be willing to get some tutoring. it is also nice that students can learn about you, too, which helps facilitate a relationship of mutual respect. i tell them about my children, about the kind of student i was/am, about my learning style and the subject areas and skills that i find difficult, how i overcome my own obstacles with writer’s block, procrastination, time management, and stress. part of our job as first-year instructors is to put a real human face on our institutions. we have the responsibility of demystifying the university while simultaneously creating a feeling of respect for higher education and creating a realistic version of what it takes to be successful. this is all nice and good, but really, when i am honest with myself, my motivation is a selfish one. the impact on student learning is secondary to the kind of joy i experience getting to know a new group of unique, interesting, and fun students each and every semester. it helps take the work out of my work. references chickering, a. w., and gamson, z. e. (1987). seven principles for good practice in undergraduate education. aahe bulletin, 39 (7), 3–7. ishler, jennifer l. crissman. (2005). today’s first year students. eds. upcraft, m. lee, john n. gardner, and betsy barefoot. challenging and supporting the first-year student. ca: john wiley & sons, inc., 15-26. jaswal, faisal, and teresa mcclane jaswal. (2008). tiered mentoring to leverage student body expertise. new directions for community colleges, 2008 (144), 55-61. o'gara, lauren, melinda mecher karp, and katherine hughes. (2009). an exploratory study of student perspectives. community college review, 36 (3), 195-218. palmer, mark, paula o’kane and martin owens. (2009). betwixt spaces: student accounts of turning point experiences in the first-year transition. studies in higher education, 34 (1), 37-54. schroeder, charles. (2003). the first year and beyond. about campus, 8 (4), 9-16. shanley, mary kay and julia johnston. (2008). 8 things first-year students fear about college. journal of college admission, 3-7. upcraft, m. lee, john n. gardner and associates. (1989). the freshman year experience. ca: jossey-bass. wabash national study of liberal arts education. (2008). overview of findings from the first year. retrieved december 22, 2009, from http://www.liberalarts.wabash.edu/study-research/. 503 service temporarily unavailable 503 service temporarily unavailable nginx/1.14.1 503 service temporarily unavailable 503 service temporarily unavailable nginx/1.14.1 503 service temporarily unavailable 503 service temporarily unavailable nginx/1.14.1 v11n1ginsberg journal of the scholarship of teaching and learning, vol. 11, no. 1, january 2011, pp. 1 – 12. growing the scholarship of teaching and learning through institutional culture change sarah m. ginsberg1 and jeffrey l. bernstein2 abstract: the scholarship of teaching and learning represents an important movement within higher education. through this work, the profession of teaching is able to build upon itself through sustained inquiry and an evidence-based culture. however, for the scholarship of teaching and learning to take hold on a campus, a culture shift often needs to occur, during which time actions by campus leaders, change agents and facilitators lay the groundwork for, and effect, institutional change. this paper uses an organizational theory approach to sketch out a model by which this culture change can occur. it then uses our experiences at a regional comprehensive university in the midwestern united states to elaborate on culture change models. our experiences teach valuable lessons about how the scholarship of teaching and learning can become an important element within a campus culture. keywords: institutional culture, organizational change, institutional change. i. introduction. on may 17, 2009 at eastern michigan university (emu), representatives from 11 universities came together as members of the sotl collaborative to discuss how to support the scholarship of teaching and learning (sotl) on each campus. the meeting took place the day before the first sotl academy conference at emu, which would attract 150 people. as those present at the first meeting of the collaborative began discussing common goals for increasing engagement in sotl on campuses, many began to make comments about the need to change the “culture” of their institutions in order to gain support for, and acceptance of, work in the scholarship of teaching and learning. what we believe that they meant was that in order for sotl to gain a foothold within an institution, the core technology3, or mission, of many institutions might need to be expanded from either teaching or “pure” research (what boyer, 1990 called the “scholarship of discovery”), to include scholarly investigation of teaching and learning, perhaps culminating in publications based on these investigations. those in the room believed that teaching needs to be valued, recognized, and rewarded more, in order for this to occur. changing the dominant core technology of a long-standing, large scale institution is not an easy task; universities typically have a history of being known for a specific type of work and are also large bureaucracies not prone to rapid change (scott, 1998). in this context of culture shifts, we then began to consider how our efforts have contributed to changing the culture of our own institution to promote sotl, to consider what else we need to do, and to reflect on the applicability of the lessons we have learned to other institutions. 1 department of special education, eastern michigan university, 115 porter building, ypsilanti, mi 48197 2 department of political science, eastern michigan university, 601 pray-harrold, ypsilanti, mi 48197 3 in the organizational theory literature, the word ‘technology’ is used to refer to the work that an organization performs, transforming “inputs into outputs” such as transforming students into professionals (scott, 1998, p. 21; see also thompson, 1967). ginsberg, s. m. and bernstein, j. l. journal of the scholarship of teaching and learning, vol. 11, no. 1, january 2011. www.iupui.edu/~josotl 2 the idea of culture, which accepts a key set of ideas and perhaps rejects another, conflicting set of ideas, is not something that can be imposed, but must develop over time and with the support of continued social interactions (morgan, 1997). at many colleges and universities, minimal focus is placed on the competing technology of sotl. this is largely because the primary mission of the institution focuses either on teaching or on research, rarely both, and seldom focuses on research about teaching and learning. the barriers to accepting and valuing this work are quite different in teaching-focused institutions (which may need to be convinced that giving faculty course releases and financial support for this research can be a good investment) and at research universities (which may need to be convinced that this work represents legitimate scholarship that ought to count, along with disciplinary research, in tenure and promotion decisions). these foci of teaching and disciplinary research may be quite important to colleges and universities, whose identities and place within the marketplace of higher education arise through their identity either as a teaching or research institution. however, if institutions that emphasize teaching or discipline-based research have much to gain from involvement with the scholarship of teaching and learning movement, then a culture shift becomes vital. the technology of sotl must gain support within the organization and among its membership in order to bring about a change in the culture of the institution. at both teaching and research institutions, changing the underlying institutional culture is important for legitimizing this work. “legitimacy refers to the degree of cultural support for an organization” (scott, 2001, p. 59); there can be no movement toward sotl without increasing its legitimacy and value on campus. ii. why the scholarship of teaching and learning? before we can reasonably advocate for change in institutional culture to promote the scholarship of teaching and learning, we should address the value of this work in general. the scholarship of teaching and learning starts from the idea that teaching is serious, scholarly work, rather than work that academics do separate from their scholarship (boyer, 1990; glassick huber and maeroff, 1997).4 shulman (1993, 1998) suggests that academics do not talk enough about teaching, or build upon our knowledge about good teaching to construct a scholarship around it. according to shulman (1998, p. 6): a scholarship of teaching will entail a public account of some or all of the full act of teaching – vision, design, enactment, outcomes and analysis – in a manner susceptible to critical review by the teacher’s professional peers and amenable to productive employment in future work by members of that same community. in this vision, scholars of teaching and learning can build on our knowledge of good practice, in an evidence-based manner, and generate a body of scholarly work (through the peer review process) similar to what we do in our disciplinary research. clearly, not all faculty must engage in the scholarship of teaching and learning. faculty do not have the time to pursue all areas of scholarly work – some might choose (quite reasonably) to focus on work in the scholarships of discovery, integration and application 4 as one telling example, our university’s tenure and promotion process requires candidates to put forward evidence of their work in service, teaching, and scholarship. teaching, as conceived here (and at many other universities), is placed in a completely different category than scholarship; this division of an academic’s portfolio is certainly the norm in higher education. ginsberg, s. m. and bernstein, j. l. journal of the scholarship of teaching and learning, vol. 11, no. 1, january 2011. www.iupui.edu/~josotl 3 (boyer, 1990).5 however, at the institutional level, and for the academy in general, a flourishing scholarship of teaching and learning becomes critical. as hutchings and shulman (1998) argue: [t]he scholarship of teaching is a condition – as yet a mostly absent condition – for excellent teaching. it is the mechanism through which the profession of teaching itself advances, through which teaching can be something other than a seat-of-the-pants operation, with each of us out there making it up as we go. as such, the scholarship of teaching has the potential to serve all teachers – and students. (italics in original) to this, we would add that the scholarship of teaching and learning has a significant home in the disciplines – as huber and morreale (2002) remind us, while good teaching has elements in common across disparate fields, teaching and learning is also significantly different across the disciplines (see also gurung, chick and haynie, 2008). thus, sotl work has the potential not only to advance teaching and learning generally, but also to focus these advancements within one’s disciplinary home; engaging in sotl does not mean turning one’s back on disciplinary work. ideally, work in sotl enhances a scholar’s contribution to his or her discipline, albeit in a non-traditional fashion. iii. research-based principles of change: a theoretical framework. assuming the reader sees value in sotl, the next step becomes making our institutions more hospitable homes for this work, both to support those engaging in this work and to enable the work to enhance the core mission of the institution. kezar (2001) notes that organizational culture is one of the major perspectives or theories that can be used to examine change in institutions. organizational culture is “what a group learns over a period of time as that group solves its problems of survival in an external environment and its problems of internal integration” (schein, 1990). shared perceptions, thoughts, and beliefs must first shift for individuals members in order to facilitate a transformation of the larger culture to incorporate new perspectives. people may be reticent to accept new views as an organizational culture experiences stability and its members experience reduced anxiety as patterns of responding and thinking become more automatic (schein, 1990). this equates to a fear of change, in which individuals and institutions prefer to maintain consistency rather than attempt to deal with the uncertain effects of new ideas. change is slow. this is a result of the long standing histories in institutions of higher education and of the sense of comfort that is often associated with the accepted culture of the organization (kezar, 2001). one of the key mechanisms of change in the organizational culture model is the leader (morgan, 1997; schein, 1990). the leader within an organization has the opportunity to model a set of beliefs and behaviors that the group members identify with and want to emulate. leaders can encourage cultural change when they demonstrate that a new belief is valued within the organization by members of the institution, such as by dedicating resources 5 we would, however, argue that all faculty members have a professional obligation to practice good teaching (defined by mckinney (2004) as “promot[ing] student learning and other desired student outcomes” (p. 8) and also to engage in scholarly teaching, in which they “reflect on their teaching, use classroom assessment techniques, engage in systematic course design, update their courses, discuss teaching issues with colleagues, try new teaching techniques, and read and apply the literature on teaching and learning in their discipline.” (p. 8) whether they take the next step toward the scholarship of learning is, and ought to be, their own decision about how to structure their time and professional goals. ginsberg, s. m. and bernstein, j. l. journal of the scholarship of teaching and learning, vol. 11, no. 1, january 2011. www.iupui.edu/~josotl 4 to sotl. they can also encourage cultural change by making official statements of endorsement to the organization, such as by including sotl in a strategic plan. the challenge for using leadership to successfully change the culture of a university is that if the members do not find the leader credible or close enough to their own thoughts, they may turn away from him or her as a leader, or separate into a subculture (schein, 1990). in addition to the leader, we would suggest two other categories of actors play a critical role. the first is the change agent, the individual (or individuals) on the ground who most strongly advocate for change. the change agent often has the strongest desire to see this change occur, as well as perhaps the most to gain from seeing it happen (and the most at risk in case this change does not happen, or in case it happens with deleterious consequences). the change agent may have the specific substantive expertise that the leader does not possess, but may lack the institutional clout or role to effect this change. thus, in addition to the change agent, the process of bringing about institutional change may also require a facilitator, someone in the position to be the bridge between the change agents (with the passion and on-the-ground expertise) and the leader (with the institutional clout to make things happen). ideally, the facilitator possesses enough of each to enable culture change to occur. as noted by scott (2001), roles played by different actors within the context of the organization can be critical to bringing about action, or cultural change. the roles, and the positions of the individuals filling them, are shown below in table 1 for easy reference; the actions of specific individuals will be discussed later in the paper. table 1. roles involved in bringing about culture change. role importance person playing role leader possesses institutional power and influence to help change institutional culture associate provost (avp) change agent possesses passion and on-the-ground substantive knowledge to help make change occur faculty members (authors of this article) facilitator possesses combination of institutional clout and on-the-ground knowledge to help smooth the process of change director of faculty development center cultural change is also slow because culture “perpetuates and reproduces itself” through socialization within the organization (schein, 1990, p. 115). as new members join an organization, they seldom have a full understanding of the cultural assumptions that are held by long-standing members of the culture. therefore, as we accept new members into our organizations, such as hiring new faculty, it is up to each of the senior members to indoctrinate the new person about what the members of the culture value. this can be done by the change agents, facilitators, and the leaders, each acting in their appropriate institutional roles. this process takes time. as new beliefs are incorporated into the culture, it may take a number of years before enough new members are brought in and identify with the newer aspects of the culture, such as the value of sotl. in addition, longer-standing members of the community may continue to challenge changes to the accepted cultural norms. while there will always be subcultures (schein, 1990), the numbers of those willing to adopt the new ideas into their culture will likely need to outweigh the numbers who are unwilling to accommodate the new ideas ginsberg, s. m. and bernstein, j. l. journal of the scholarship of teaching and learning, vol. 11, no. 1, january 2011. www.iupui.edu/~josotl 5 before true cultural shifts can take place. challenges and conflicts over beliefs are common in periods of cultural change (scott, 2001). given the size and history of most post-secondary institutions, gaining cultural support for a new and competing technology such as sotl represents a significant challenge. to bring about change, understandings of a goal or a technology must be altered at all levels, from broad groups (such as administrators) down to the individual faculty members (kezar, 2001). kezar outlined a “complex set of research-based principles” (p. 5) that is the basis of change in higher education. using a number of these principles, we will examine, as a case study, our efforts to begin changing the culture of eastern michigan university such that sotl may become an accepted technology within the institution. the following description of the steps we took to move our institution’s culture toward one of accepting sotl as a competing technology are matched with a number of the principles of change (kezar, 2001). our reflections on our actions, and the resulting impact that each had on emu’s culture, often address more than one of the principles of change at each individual step. iv. a case study of organizational change: sotl at emu. having sketched out a vision for how institutional change can occur, our next task is to use this model to shed light upon our case. below, we report on the process by which the scholarship of teaching and learning has become a more significant part of the institutional culture at eastern michigan university. our discussion of the case study is largely chronological, and makes explicit reference to the elements of change identified by kezar and other scholars of institutional change; the key elements of the case study, and the links they make to the theoretical model, are detailed below in table 2, along with a timeline of when these steps first occurred.6 we note in the final section of this case study that change has not occurred to the full extent to which we would hope, but that our progress to date has revealed more than modicum of success, and given us much to build on. a. promote organizational self-discovery. the formal work of the scholarship of teaching and learning at eastern michigan university began with bernstein’s selection as a carnegie scholar in the carnegie academy for the scholarship of teaching and learning (castl) program for the 2005-06 academic year.7 as a condition of his being accepted for the program, eastern michigan university needed to agree to provide institutional support for sotl. as part of this support, the director of emu’s faculty development center and agreed that bernstein would lead a faculty development seminar on the scholarship of teaching and learning during the 2006-07 academic year (as of this writing, the seminar continued for four years with four classes of fellows under his direction; ginsberg is 6 in the timeline, we concentrate on when these actions were initiated, in connection with the first meeting of the sotl collaborative and the first iteration of the sotl academy conference. the activities we describe are, of course, ongoing activities. in describing the process by which we brought about this culture shift, however, we focus most on the early stages of these activities, as they were institutionalized. 7 this is not to suggest that there was no work being done in this area before that point. some faculty (like ginsberg) had been doing and publishing work in the scholarship of teaching and learning before that – ginsberg had been a regular participant in sotl conferences by that point. and others (including bernstein) had been doing work that fits within the framework of the scholarship of teaching and learning even if he had not been specifically framing his work around the sotl movement. ginsberg, s. m. and bernstein, j. l. journal of the scholarship of teaching and learning, vol. 11, no. 1, january 2011. www.iupui.edu/~josotl 6 currently running a modified version of the seminar). ten faculty members joined the seminar the first year, representing all five of the university’s different colleges. table 2. principles of change applied to growth of sotl at eastern michigan university. principles of change (drawn from kezar, 2001) example of actions taken timeline promote organizational self-discovery -emu agrees to provide institutional support for sotl per carnegie’s castl program 2005-06 – bernstein is carnegie scholar 2006-07 – first sotl seminar held (ongoing) articulate core characteristics lay groundwork for change -sotl symposium establishes the breadth and depth of sotl for wider campus audience winter 2008 – sotl symposium begins (ongoing) april 2006 – randy bass lecture december 2008 – dan bernstein lecture realize that change in higher education is often political -obtain support and endorsement of avp of research winter 2008 – gain support from avp (leader) connect the change process to individual and institutional identity -garner support from campus facilitators/boundary spanners (college deans and fdc director) winter 2008 – gain support from faculty development director (facilitator) fall 2008 – gain support from college deans balancing external forces and the internal environment -formation of sotl collaborative fall 2008 – contact institutions to offer membership in collaborative may 2009 – first collaborative meeting as part of the seminar, faculty members spent the fall term designing an inquiry into teaching and learning in one of their winter term classes. they spend the winter term implementing the project, collecting data on student learning, and analyzing the data they had collected. throughout the year, the group came together as a cohesive unit, relishing their time together to talk about issues of student learning that were often marginalized in other segments of the university. these faculty members (and the ones who participated in the seminar in subsequent years) were an interdisciplinary mix of individuals who together became strong supporters of the work, and advocates for it within their departments and colleges. each year of the seminar, participants wrote chapters in a book published by the faculty development center. while the chapters were occasionally uneven and the book lacked the cachet that would be associated with publication by an external press, the essays were generally of high quality. the book itself became an artifact, an object people could hold in their hands and point to as representative of what could be done in the scholarship of teaching and learning at emu. with the publication of each volume, this work was made visible to others (in part through a well-attended book launch in which the editor and authors each speak briefly about their chapters). as participants in the seminar came up for tenure and promotion (close to half were untenured when they participated in the seminar), their presentation of these chapters as evidence of scholarly work began to push the envelope and slowly establish the case that scholarly ginsberg, s. m. and bernstein, j. l. journal of the scholarship of teaching and learning, vol. 11, no. 1, january 2011. www.iupui.edu/~josotl 7 inquiries into student learning represents true scholarship that should be counted as such by the university. an additional advantage of working with many untenured faculty is that once they became tenured, they would likely remain at the university for a long time and continue to be strong supporters of the scholarship of teaching and learning at emu during their careers at the school. the university initially agreed to support bernstein in his efforts as a carnegie scholar by providing institutional support for sotl upon completion of his time as a carnegie scholar. however, it is unlikely that they had any idea as to what shape that support would take, nor how it would impact the campus. the sotl seminar, as implemented by bernstein and supported by the faculty development center’s director, began the process of the organization self-discovery (kezar, 2001). it was not necessarily their intention to do so, but it was an eventual outcome. b. articulate core characteristics. as the seminar was starting to take on a life of its own, we began to explore ways to enhance the visibility of this work (and to provide more opportunities for discussion of teaching and learning). one vehicle we have used for this has been the sotl symposium, a regular seminar series in the scholarship of teaching and learning. this began in the winter 2008 semester, in the middle of the second year of the faculty development seminar. we have held approximately four talks a semester, with half coming from emu faculty (usually alumni of the faculty development seminar) and the remainder coming from faculty at nearby campuses. these talks showcase some of the best work coming out of our campus, and also provide an opportunity for emu faculty to learn from experts in this work outside our campus. we generally average around a dozen people per talk, with participants representing a diverse range of faculty, lecturers, and often students. a second component of our articulating the core characteristics of sotl has been the outside speakers we have brought to campus (apart from the sotl symposium). in the last four years, we have hosted randy bass to provide a workshop on the scholarship of teaching and learning (as well as a keynote address at a teaching and learning fair) in april 2006 and dan bernstein to do a workshop on course portfolios (as well as a sotl symposium talk on the peer review of teaching project) in december 2008. both workshops drew large and appreciative audiences. the process of expanding sotl from the initial small group of faculty who participated in the sotl seminar began establishing for a wider campus audience just what the characteristics of sotl included. these symposium presentations, which were open to the university community, allowed participants to see variety in the work of sotl scholars and begin identifying key features that articulate the core characteristics of sotl, such as the emphasis on student learning and value being placed on reflection in the teaching process (kezar, 2001). bringing in acknowledged leaders in the field provided opportunities for the campus community to learn about some impressive work and to expand their understanding of the shared, core characteristics associated with sotl. inclusion of these speakers in campus activities not only clarified the characteristics of sotl, but they allowed people to see the possibilities for which we could reach here on campus. the exposure to a variety of sotl work from those within the campus and across the country allowed participants to increase their understanding of sotl characteristics and provided germination of the sotl culture in new members. ginsberg, s. m. and bernstein, j. l. journal of the scholarship of teaching and learning, vol. 11, no. 1, january 2011. www.iupui.edu/~josotl 8 c. realize that change in higher education is often political. in late 2007, the two of us began to discuss the idea hosting a regional sotl conference on our campus. one of our primary goals was to increase the visibility and recognition of sotl work happening on our campus. in addition to the work that we were doing, we knew there were many colleagues who were making strong contributions to the sotl literature as well. a conference was an opportunity to highlight this work, possibly bringing notoriety to our campus, as well as bringing together colleagues from nearby campuses. through a series of networks that each of us had in place at neighboring institutions, we were aware of what we believed to be a fair interest in sotl in the region. we looked around at schools playing a leadership role in the scholarship of teaching and learning movement and saw no reason we could not do the same. the next step, as we saw it, was to determine if the higher levels of the university would endorse our efforts to form and host this conference. we immediately identified one of the university’s two associate provosts as the likely leader for our efforts. bernstein had been involved in a small reading group on sotl with this individual, when he had been the administrator on a team of four faculty and one administrator that composed and submitted emu’s application to be part of the carnegie foundation’s campus leadership program in the scholarship of teaching and learning. bernstein had been pleased to learn that the associate provost had previously done some research on student learning in his classes and that he continued to have an interest in this kind of work. as associate provost, he controlled some resources that could aid a conference, and also had institutional clout that would be essential in pushing this work along. his support for the conference would be necessary for us to move forward. the associate provost was immediately supportive of the idea, albeit with an administrator’s eye for the big picture of the institutional mission and the institutions’ place among the greater higher education community. we had presented him with plan for generating interest in the conference, along with the best evidence we could muster that we could do so. we also shared with him a budget showing various scenarios for how the conference could break even, based upon likely attendance, fixed and discretionary expenses (we aimed to be frugal, but also not to make the conference appear cheap), and registration fees. based on our preparation and previous reputation for getting things done, the associate provost offered the go-ahead to the conference. we began publicizing it and immediately ramped up our planning. the support he offered at the beginning of the process continued throughout, up to and including the days of the conference (when he participated in a panel discussion at the conference and took time out of his busy schedule to attend many sessions and both keynote addresses). we believe we helped maintain this support through periodic updates as the conference planning was moving along. as he saw what we were doing, and how careful and conservative we were being in the planning, he began to commit more resources in support of our efforts. as noted below, he also began to use the power of his office more and more to increase our reach. by providing support from a leadership role, this administrator modeled his belief that sotl would be of value to the institution and that support of it could be valuable to the technology of “education first,” the university’s marketing tag line. additionally, he demonstrated an awareness of the image of the institution within the greater community, another principle of change (kezar, 2001, p. 6). as leaders can effectively do, he provided the movement ginsberg, s. m. and bernstein, j. l. journal of the scholarship of teaching and learning, vol. 11, no. 1, january 2011. www.iupui.edu/~josotl 9 with credibility by making an official endorsement of it and by providing us with key financial resources in the form of seed money (morgan, 1997; schein, 1990). the political nature of the leadership support was invaluable to the forward momentum of sotl on campus (kezar, 2001). a leader’s support of this nature signaled to other university leaders above and below him that sotl was of value to the institution. it also signaled to the faculty community that sotl would be accepted, to some degree, in our scholarly agendas. d. connect the change process to individual and institutional identity. in an attempt to both build attendance at the conference, and to build campus-wide support for the scholarship of teaching and learning, we next began to broaden our conversations on campus in support of the conference. a first conversation was with the director of our institution’s faculty development center. long supportive of this work, the director was eager to help support the conference. this involved much expert guidance, many intangible contributions and small kindnesses, and a generous contribution of funds toward the registration fee for every emu presenter at the conference. besides making it easier for emu people to attend the conference, her support was a powerful signal that the institution was supportive of the conference. a second strategy that we pursued for reaching out was to approach the deans of emu’s different colleges. we discussed with them possible panels that they might like to see at the conference (a few of which ended up appearing on the program). we also solicited their support for the conference through encouraging their faculty to attend and signaling to them that their college dean considered this to be valuable work. one conversation, with the dean of the college of technology, was particularly valuable as we learned of his long-standing commitment to the work of the faculty development center, that we were previously unaware of. he offered to pay the remainder of the registration fee for any faculty member from his college who attended, and agreed to be part of a panel on sotl and academic careers at the conference by reaching out to individuals who were able to facilitate change, such as the director of the faculty development center and the college deans, we extended the individual connections we were making beyond faculty and one administrative leader. . the financial support from a dean was indicative to members of the culture under his leadership that the institution was embracing this technology as part of its identity. the director of the faculty development center’s commitment of funds also signaled to a broader section of the university community, above and below her, that this work was worthy of integration into our institutional identity. these key individuals also had the capability of helping us connect the change to the institutional identity (kezar, 2001) as their public support of the conference signaled to the faculty that the institution valued sotl work. e. balancing external forces and the internal environment. as we were working toward planning a conference, we began to consider a variety of ways to involve other institutions in the conference. one motivation for doing this was partially the desire to increase attendance at the conference, and hence increase the revenue we brought in. we were both familiar with conference models that offered group discounts to schools that brought a certain number of people to the conference. we liked this idea, but wanted to do more ginsberg, s. m. and bernstein, j. l. journal of the scholarship of teaching and learning, vol. 11, no. 1, january 2011. www.iupui.edu/~josotl 10 with it. we aimed to build something larger that might contribute to building capacity for sotl within the surrounding community. the idea we settled on was to build, from the grassroots, a community of schools that would come together to support the scholarship of teaching and learning. we called this group the sotl collaborative. functionally, we asked each participating school to pledge $500 in support of the collaborative. in exchange, anybody from their institution attending the conference would receive a $25 discount on the conference registration. we also planned a meeting of each school’s collaborative representative for the night before the conference, in order to talk about these ideas some more. the collaborative attracted thirteen schools, from a wide range of institution types (ranging from research-intensive schools to regional comprehensives to liberal arts colleges). in forming the collaborative, we sent a signal within our own institution that the work we were doing in advancing sotl had receptive audiences outside our school. we also positioned eastern michigan university as a leader in this kind of work in our region; this leadership role was attractive to the administration of our school. bringing together all of these institutions provided useful ideas for advancing this work; it also enabled us to form valuable partners that we continue to call on. as we proceed, we have discovered that finding a place for the sotl collaborative outside its role as a vehicle for supporting the conference presents a challenge. after a few attempts, we are currently working with member schools to create a workshop for people new to sotl that will be conducted before the 2011 conference, and can be transported to member schools at a reduced rate. we are also exploring other ways to pool resources and link the schools together. the collaborative remains an excited, albeit uncertain, work in progress universities, like other large institutions, do not exist in a vacuum, but are situated in a context of peer institutions, community constituents, and internal members (morgan, 1997). understanding of the intersection between the demands of the environment, including the balance between those that are internal and those that are external is a key component to bringing about change. the external environment can energize the organization through its provision of resources, including funds and intangible support, such as encouragement and shows of support. by enticing those constituents in the external community to invest in our process, we brought about a balance between the leaders’ desire to promote our institution’s contribution to sotl and our collaborating institutions desire for their own success. the greater value and investment placed on our efforts by colleagues from the external environment, the more likely our own leaders were to see that the culture shift move the image of our institution forward as a leader in sotl (morgan, 1997; kezar, 2001). v. conclusion: looking forward. at this time, we cannot declare that the culture of our institution has completely changed to the point that sotl is embraced by all members of the institution. however, progress is definitely visible in a number of key areas. the leadership of the university continues to support the sotl academy conference, now in its third year. the active support has moved further up amongst the leaders as the provost has not only joined us to welcome conference attendees, he has suggested to other leaders on campus that information regarding sotl is valuable to student academic success. as a result, our opportunities to share information about sotl and our efforts on campus reach new audiences and legitimacy increases (scott, 2001). recently the provost ginsberg, s. m. and bernstein, j. l. journal of the scholarship of teaching and learning, vol. 11, no. 1, january 2011. www.iupui.edu/~josotl 11 requested that we make a presentation about the sotl work on campus in an academic student success summit. this request signals to us that the higher levels of administration are recognizing the value of sotl within the culture of our institution. yet, our challenges also continue. as change agents, we now are confronting the fact that both our leader (the associate provost) and facilitator (faculty development director) have left the university in the past year. once new people are inserted into those positions, we will need to build support among these people as we seek to continue the process of cultivating institutional change. as noted earlier, new members joining an organization may bring their own set of values and it will be our responsibility to share our vision of the role of sotl within the culture of the institution in order to enable ongoing support from the individuals in these roles. we continue to reflect on our process and the actions that we can take to further support sotl on our campus and within our community. in an effort to continue leveraging the external forces to help change the culture (kezar, 2001), we have begun to incorporate participation from key members of the sotl collaborative in growing both the conference and the collaborative itself. in this way, our internal efforts are reinforced by those outside of our environment, and are at the same time strengthened by their participation and support. at the time of this writing, the results remain to be seen, however, we are optimistic that the increased functions of external constituents will serve to support the acceptance of sotl on our campus and on theirs. the process of changing the culture of a large institution, such as a long-standing university, is a slow one (scott, 1998). for those would be agents of change, the process can be frustrating and seemingly without success, particularly in the early stages. however, kezar’s (2001, p 5-6) research-based principles of change suggest that there are indeed mechanisms that can be successful in bringing about the type of change that is required in order to move a teaching or research oriented institution forward to embracing a new and possibly competing core technology such as sotl. while our work here does not represent a fully-fleshed image of a model of change from a cultural theory perspective (kezar, 2001; morgan, 1997; scott, 1998) it does hold the potential to enlighten and encourage others who would like to see their higher education institutions begin to embrace sotl. for those who are, as we were, lamenting the need for the need for cultural shift within their institution, our case study connected can be a model of just one possibility. it is important to keep in mind that no one action is going to have a far-reaching effect. rather it is the sum of parts, implemented over a long period of time that will hopefully add up to equal more than just the sum of individual steps taken. the opportunity to reflect back on the steps taken and their resulting movement forward in changing a culture is a valuable one. references baker, p.j. (1986). the helter-skelter relationship between teaching and research: a cluster of problems and small wins. teaching sociology, 14(1). 50-66. boyer, e.l. (1990). scholarship reconsidered: priorities of the professoriate. princeton: carnegie foundation for the advancement of teaching. glassick, c.e., huber, m.t. & maeroff, g.i. (1997). scholarship assessed: evaluation of the professoriate. san francisco: carnegie foundation for the advancement of teaching. ginsberg, s. m. and bernstein, j. l. journal of the scholarship of teaching and learning, vol. 11, no. 1, january 2011. www.iupui.edu/~josotl 12 gurung, r.a.r., chick, n.l. & haynie, a. (2008). exploring signature pedagogies: approaches to teaching disciplinary habits of mind. sterling, va: stylus. huber, m.t. and morreale, s. (2002). disciplinary styles in the scholarship of teaching and learning: exploring common ground. washington, dc: american association for higher education and the carnegie foundation for the advancement of teaching. hutchings, pat & shulman, l.s. (1999). “the scholarship of teaching: new elaborations, new developments.” change 31 (5), p. 10-15. kezar, a. (2001). understanding and facilitating change in higher education in the 21rst century. eric digest. washington dc: george washington university. (eric document reproduction service no. ed457763). morgan, g. (1997). images of organization. thousand oaks, ca: sage publications. schein, e. h. (1990). organizational culture. american psychologist, 45(2), 109-119. scott, w. r. (2001). institutions and organizations (2ed). thousand oaks, ca: sage publications. scott, w. r. (1998). organizations: rational, natural, and open systems. upper saddle river, nj: prentice-hall. shulman, l. s. (1993). teaching as community property: putting an end to pedagogical solitude. change, 6, 6-7. shulman, l. s. (1998). introduction. in pat hutchings (ed.) the course portfolio: how faculty can examine their teaching to advance practice and improve student learning (pp. 5-12). sterling, va: stylus. thompson, j. d. (1967). organizations in action. new york: mcgraw hill. 3447 journal of the scholarship of teaching and learning, vol. 13, no. 5, december 2013, pp. 90 – 105. the impact of an interdisciplinary learning community course on pseudoscientific reasoning in first-year science students timothy m. franz1 and kris h. green2 abstract: this case study examined the development and evaluation of an interdisciplinary first-year learning community designed to stimulate scientific reasoning and critical thinking. designed to serve the needs of scholarship students majoring in mathematics and natural sciences, the six-credit learning community course was writing-intensive and emphasized general scientific reasoning and critical thinking skills. success of the course was measured using a pre-test/post-test design that assessed students’ paranormal beliefs. outcomes of the study indicated students’ paranormal beliefs were significantly lower at the end of the semester than at the beginning, which was used as a surrogate measure of scientific reasoning that was directly relevant to course content. supplementary analyses demonstrated that their (a) paranormal beliefs were significantly lower than other students and (b) students self-identified the importance of the scientific reasoning skills learned in the course without being prompted on their teachercourse evaluations. the results of this study can inform the design of interdisciplinary, scientific reasoning courses. keywords: pseudoscientific thinking, critical thinking, learning community, scientific reasoning, first-year students i. introduction & background. many college professors attempt to promote scientific reasoning and critical thinking within their courses. according to shermer (2002, p. 18) scientific reasoning is: a set of methods designed to describe and interpret observed or inferred phenomena, past or present, and aimed at building a testable body of knowledge open to rejection or confirmation. further, halpern (1997, p.4; see also halpern, 1998), defines critical thinking as: thinking that is purposeful, reasoned, and goal directed – the kind of thinking involved in solving problems, formulating inferences, calculating likelihoods, and making decisions when the thinker is using skills that are thoughtful and effective for the particular context and type of thinking task. clearly, scientific reasoning and critical thinking skills are an essential foundation of skepticism and evidence-based reasoning that are the foundation for science. thus, a goal of most first-semester introductory science classes is to acculturate students in reasoning based on evidenced obtained through the scientific method (druger, 2002). while this has always been a component of science education, more emphasis is being placed on understanding the nature of science and scientific reasoning of late (national research council, 1998). even though many of these early scientific-reasoning skills may be generalized beyond a specific scientific discipline, 1 department of psychology, st. john fisher college, 3690 east ave., rochester, ny 14618, tfranz@sjfc.edu. 2 mathematics department, st. john fisher college, 3690 east ave., rochester, ny 14618, kgreen@sjfc.edu. franz, t. and green, k. journal of the scholarship of teaching and learning, vol. 13, no. 5, december 2013. josotl.indiana.edu 91 one cannot separate scientific content from scientific reasoning and critical thinking (see for example, nelson, 1999) although critical thinking can be developed supplementally in such a way that it strongly supports deeper learning of scientific content (adey & shayer, 1993). other studies have examined the impact of learning communities on reasoning and critical thinking (e.g., browne & minnick, 2005) in general, focusing on learning communities that pair an introductory science course with a course from another discipline. the present study focuses on whether a specially designed learning community for science majors can influence students’ pseudoscientific thinking and thus improve their skeptical, scientific, and critical reasoning. the science scholars learning community (lc) was a two-course, team-taught program designed for a specific group of scholarship recipients at st. john fisher college. the lc program at fisher was structurally similar to those at other school institutions; first-year students were housed in the residence halls near their lc peers so that the students developed a close-knit cohort that offered academic and social support during the adjustment to college. according to inkleas, soldner, longerbeam, and leonard (2008, p. 502), this lc fell into the category of “small, limited resourced, primarily residential life emphasis” programs. the students in the science scholars learning community came to the college with a sound basic high school understanding of science and were specifically selected for the scholarship program based on excellent high school performance in science and mathematics. the course cluster was designed around comparing pseudoscientific thinking and scientific reasoning and emphasized methods of improving scientific reasoning and critical thinking by learning, discussing, and using scientific methods. thus, our primary research questions for this study were 1. in what ways can this lc impact students’ pseudoscientific thinking? 2. how does the pseudoscientific thinking of the students in the learning community compare with the general student population? a secondary research questions, assuming a positive answer to the first questions was: 3. do improvements in pseudoscientific thinking translate into improvements in critical thinking and scientific reasoning? a. brief background about learning communities. the learning communities national resource center (n.d.) offers several models of learning communities, but in general, learning communities are a collection of courses in which a cohort of students participates. these courses can be either loosely connected or tightly linked with integrated content (smith, macgregor, matthews, & gabelnick, 2004). in some cases, the same group of students is enrolled in several courses, such as calculus and introductory physics, and a seminar, which helps students make connections between the linked courses. a coordinated studies model, on the other hand, places the cohort in a team-taught block that covers the material of several traditional courses with integration among all topics. these learning communities, which are planned collaboratively among the faculty members involved combine content knowledge with skills practice (smith et al., 2004). according to smith and colleagues, learning communities are one of the solutions to recent calls for educational reform, because students are actively engaged and reflective. the reflection in action (e.g., schön, 1987) helps students to build metacognitive structures necessary for the critical and creative thinking required about academic content. because of this, learning communities have been used to improve retention, attendance, and social behaviors for first-year franz, t. and green, k. journal of the scholarship of teaching and learning, vol. 13, no. 5, december 2013. josotl.indiana.edu 92 students (inkelas, daver, vogt, & leonard, 2007). in fact, they have also been used specifically in science and engineering programs to help students make the transition to college (smith, et al., 2004) and improve general critical thinking skills (quitadamo, brahler, & crouch, 2009). a similar example of a science-based lc involving the coordinated studies model is provided by morgan, carter, lemons, grumbling, and saboski (1995). in fisher’s cohort-based cluster learning community model, courses in the lc are linked by a common theme, and the students are housed together to provide a community of support outside of the classroom. such living-learning communities tend to promote peer-support networks and enhance campus involvement (dabney, green, & topalli, 2006) at fisher, each lc fits into one of two formats: (a) a standard first-year composition course paired with one content-area, general education course or (b) a cluster of two content courses taught in a writing-intensive format. the science scholars lc is an example of the second model. in either model, all courses in a cluster revolve around a common theme, such as the vietnam war, the nature of self, the economics of sports, or, as in this case, the nature of scientific thinking. lcs have been a part of the fisher first-year experience for over a decade. similar to the findings at other schools, the lcs have resulted in higher first-year retention rates (e.g., dabney et al., 2006; dodge & kendall, 2004), crossdisciplinary communication and collaboration among the faculty, and tighter-knit student cohorts. one example of a science-based lc involving the coordinated studies model is provided by morgan and colleagues (1995). the course of study involves a year-long experience combining introductory courses in biology with a literature course and an environmental science course in addition to a one-credit seminar. their findings showed that students in the lc improved significantly on intellectual development, suggesting that the lc had improved their reasoning and critical thinking skills (see also work by browne & minnick, 2005) and students’ ability to apply these to decision-making and value judgments. b. critical thinking, scientific reasoning, and pseudoscientific beliefs. halpern’s (1997) definition of critical thinking, cited above, is a broad attempt to capture the multitude of thinking skills that might be involved in “critical thinking” by generalizing the concept as dealing with “thinking that is purposeful, reasoned, and goal directed.” it seems, though, this view of critical thinking as a generic skill that can be acquired independent of specific content may be flawed. recent studies have found that many academics describe critical thinking differently (moore, 2013) using such a variety of descriptors that one is tempted to believe that there is no single elephant being described by all of the blind academics. however, deeper analysis of the concept (davies, 2013) yields a commonality to all of these definitions that is instantiated differently in different disciplines. thus, there is a common core of learning to think in a way that prioritizes logic and evidence over instinct, and this core can be approached and learned in a way that allows it to easily transfer across disciplines. at the same time, each discipline has its own criteria for what constitute logic and evidence. in the sciences, critical thinking is instantiated by the term “scientific reasoning” which combines aspects of analysis with specific skills related to experimental design (national research council’s committee on undergraduate science education, 1999). generically then, critical thinking includes the judgment and a skeptical stance toward evidence that is presented. this allows one to then test to what degree a given assertion is supported by the evidence and reasoning presented, and suggests a tradeoff between learning to franz, t. and green, k. journal of the scholarship of teaching and learning, vol. 13, no. 5, december 2013. josotl.indiana.edu 93 think critically and accepting paranormal beliefs. this tradeoff is partially supported by the literature. for example, past research has examined the relationship between critical thinking and paranormal beliefs, showing in general (cf., roe, 1999) that critical thinking is negatively related to paranormal beliefs (aarnio & lindeman, 2005; cheung, rudowicz, kwan, & yue, 2002; messer & griggs, 1989) or reported past paranormal experiences (royalty, 1995). further, past research has demonstrated a negative correlation between paranormal beliefs and reasoning ability (hergovich & arendasy, 2005) and that paranormal beliefs are unrelated to age (aarnio & lindeman, 2005). other studies, however, have found that higher levels of education do not necessarily translate into lower paranormal beliefs (farha & steward, 2006). this same study also examined paranormal beliefs across the disciplines, finding that students in the sciences fell somewhere in the middle in terms of paranormal beliefs (social sciences had the highest percentage of believers) and that were virtually no differences attributed to gender. thus it seems that even further training within a scientific discipline, which one expects to focus on scientific reasoning, is not sufficient to eliminate a student’s beliefs in paranormal (or more broadly, pseudoscientific) phenomena, suggesting that individuals possess the capacity to engage in different modes of thinking selectively even though the selection may not be made a conscious level. this also suggests that without explicitly applying the generic skill of critical thinking to the specific contents of pseudoscience and the paranormal, students may develop strong scientific reasoning and critical thinking skills while still holding uncritical beliefs in phenomena such as spontaneous human combustion and fortune telling. c. college and student profiles. st. john fisher is a private liberal arts college in the catholic tradition located between the city of rochester, new york, and the eastern suburbs. admission is competitive, and the majority of students are drawn from a 100-mile radius of the campus, resulting in a student population mostly from upstate new york. many students come from the outlying rural school districts, and many are drawn to fisher because of the catholic heritage of the school. most of our science scholar learning community students fall close to national and state averages in academic achievement in high school (e.g., mean national high school gpa of 88% versus 94% for the science scholars). at the time of this study, students in the top two tiers of scholarships with an interest in biology, chemistry, computer science, mathematics or physics are also offered the opportunity to apply to the science scholars program. most students have completed several advanced placement courses in mathematics and science or have participated and received credit for other college-credit courses before high school graduation. almost all of the science scholars have completed four years of mathematics and science in high school and are academically motivated students with a strong interest in the sciences. biology and chemistry are the most common major choices followed by mathematics. the remaining minority of students are equally split between computer science and physics. d. course background and design. the science scholars learning community was a writing-intensive, two course (6-semester hour) experience. the explicit connection between writing and science helps improve students’ scientific writing (kokkala & gessell, 2002) and provides them with valuable practice applying the tools of the course independently. rather than give two separate grades for the two courses in franz, t. and green, k. journal of the scholarship of teaching and learning, vol. 13, no. 5, december 2013. josotl.indiana.edu 94 which the students were officially registered, students received the same grade for both courses, and both instructors participated in reading all major papers. in addition, a substantial component of the course grade was based on in-class and small writing assignments, as well as a group research project that culminated in a large team-run debate. unlike many of the lcs at fisher, the science scholar learning community was not composed of existing first-year courses like introductory psychology or chemistry 101. thus, there were no departmental expectations to provide the foundation for future courses in a discipline. instead, two new courses were created to allow the science scholar learning community to explore the nature of science independent of a particular discipline. one course was listed in the interdisciplinary studies program while the other was listed under the mathematics, science, and technology integration program. while neither course counted for credit in any major, both courses provided students with credit towards meeting the college core requirements. together, these courses explored the nature of science by comparing scientific fields and scientific thinking with pseudoscience and science fiction topics. one of the faculty members in this lc normally taught in the department of psychology, and the other in the mathematical and computing sciences department. thus, by design, students were exposed to different perspectives on scientific and critical thinking. the faculty members met regularly to discuss the course and the students and to ensure that each class period connected to the previous and subsequent periods and maintained the structure of the lc. this also meant that each of the freshman-level writings completed during the semester was directly tied to all lc content rather than being unrelated at times. for example, when students were introduced to the idea of critically analyzing evidence, we also discussed how to summarize sources and properly cite them, with the student practice being tied to their particular paper. during the semester, both instructors worked on the writing topics with the students, as appropriate to the flow of the lc and student needs. both instructors used writing assignments to help students practice the skills and thinking that were the goals of the lc. the course had two primary texts. the first provided an outline of skepticism, scientific reasoning, and critical thinking, in contexts ranging from alien abduction scenarios and witch hunts to psychic powers and holocaust deniers. shermer’s (2002) why people believe weird things provided an outline of skepticism, scientific reasoning, and critical thinking in a wide range of contexts. the second text, taking sides: clashing views on controversial topics in science and technology (easton, 2002, 2004), was a collection of pro and con essays on a variety of science and technology issues. these two books were supplemented with newspaper articles, magazine articles, internet sources, videos, and other media as appropriate. throughout the experience, students were expected to locate and evaluate additional resources, using the information literacy skills emphasized in the lc program at fisher. in addition, we used two separate writing references (aaron, 2003; lester & lester, 2003). e. course structure and implementation. the students in the science scholar learning community met for approximately three hours on tuesday and three hours on thursday afternoons. each class consisted of a variety of pedagogical modalities, including short lectures, class discussions, student presentations and debates, in-class assignments, and/or watching films or film clips that supported course content. during the term, students explored and wrote four papers, each dealing with different scientific topics. we began the course each semester by using a demonstration of psychic powers and tied franz, t. and green, k. journal of the scholarship of teaching and learning, vol. 13, no. 5, december 2013. josotl.indiana.edu 95 this demonstration to a unit of introductory scientific reasoning. the other instructor served as a confidant in these tricks, encouraging the students to take on evidence presented and think critically about it, and having them list various things they could change about the demonstration. the goal was for the students to think critically while the “psychic” tried to adapt as needed to overcome their suggestions and tests. in each year of the lc, the students eventually found a controlled change in the demonstration which negated the abilities for the instructors to successfully complete the trick. all of this served to accomplish one main learning goal: it established the course content as learning how to think about the world critically. students often reported that this one activity, on the first day of the semester, made a considerable difference for them in the ability to begin to think critically about information. throughout the course, the instructors provided experiences to support the development of critical thinking in a structure similar to that in cognitive acceleration (adey & shayer, 1993) through stages of concrete preparation, cognitive conflict, metacognition and bridging. concrete preparation was provided by connections to real situations and experiences, such as the introductory psychic demonstration. cognitive conflict was generated through readings, discussions and various media presentations exposing students to multiple components and perspectives on the issues. the writing component of the lc provided opportunities for metacognition to be manifested, and parallels among the various situations and the common ways in which thinking goes wrong served as a bridge throughout the course. the course structure used active learning principles to help to improve learning (yoder & hochevar, 2005). for example, students were frequently engaged in small group discussions and short presentations to the class during class time. almost every class period involved some form of informal or formal writing, usually a short paragraph or so related to the current discussion, either to prime the students before the discussion or to summarize their ideas after. some of these focused specifically on improving their writing skills, such as revising a particular paragraph or sentence of their work, or re-writing a paragraph using a different voice. ii. method. clearly, the class was designed to stimulate critical thinking and scientific reasoning. during two of the four years that we co-taught the course, we assessed whether students’ level of reasoning improved. to do so, we conducted a pre-test/post-test nonexperimental design using the paranormal beliefs scale (tobacyk & milford, 1983) to assess their level of reasoning about paranormal phenomena. we also compared students’ scores on the paranormal beliefs scale to a) a control group of means from students in the original tobacyk and milford (1983) study, and b) a control group of students at the college. finally, we conducted a qualitative analysis to examine students’ unprompted reports of learning about critical thinking and scientific reasoning through the comments on our teacher-course evaluations. a. design and participants. our design met the criteria of a quasi-experiment (see cook & campbell, 1979). the key participant group included the students from two cohorts of the lc. to learn about contentspecific paranormal and pseudoscientific critical thinking, we collected information from the first-year science scholars students in the class during two different years. in the first year of the study (2003, the second year we taught the class), we collected data from 22 of 23 enrolled franz, t. and green, k. journal of the scholarship of teaching and learning, vol. 13, no. 5, december 2013. josotl.indiana.edu 96 students in the class (n = 14 women, n = 9 men), for a 96% response rate. in the second year (2004, the third year we taught the class), we collected data from all 21 enrolled students (n = 14 women, n = 7 men), for a 100% response rate. the main phase of the study utilized a pre-test/post-test design. specifically, responses to the paranormal beliefs scale (pbs, tobacyk & milford, 1983) at the beginning of the semester were compared to responses to the pbs at the end of the semester. data from the key participants were also compared to several other data sets: (a) normative data from tobacyk and milford and (b) posttest data from faculty and other students from a second phase of the study (spring 2007). b. measures. the key surrogate measure of reasoning for the study was the 25-item paranormal beliefs scale created by tobacyk and milford (1983). we used this scale because our curriculum directly compared pseudoscientific claims (e.g., psychic powers and alchemy) to what is known in their respective scientific fields (e.g., psychology and chemistry). the paranormal beliefs scale provides one complete measure of paranormal beliefs that is based on an average of the items. it also includes seven subscales, including traditional religious beliefs (e.g., the soul continues to exist though the body may die), psi (e.g., mind reading is not possible), witchcraft (e.g., black magic really exists), superstition (e.g., black cats can bring bad luck), spiritualism (e.g., reincarnation does occur), extraordinary life forms (e.g., the loch ness monster of scotland exists), and precognition (e.g., astrology is a way to predict the future). most measures had sufficient inter-item reliability; those that did not may be due to the small sample sizes used to calculate the α coefficients (see table 1). table 1. cronbach’s α coefficients for paranormal beliefs scale and each subscale as a function of sample and pre-test/post-test administration. sample: science scholar students 2006/07 pre-test α post-test α students (n = 43) (n = 43) (n = 228) entire paranormal beliefs scale (25 items) .85 .89 .92 traditional religious beliefs scale .90 .88 .85 psi subscale (4 items) .81 .70 .84 witchcraft subscale (4 items) .69 .80 .84 superstition subscale (3 items) .85 .88 .81 spiritualism subscale (4 items) .78 .80 .80 extraordinary life forms subscale (3 items) .88 .95 .90 precognition subscale (3 items) .71 .54 .75 in all of the groups, we had acceptably low measures of skewness and kurtosis (all < 1, well under the generally accepted minimum of 2), and the variances in different groups all had acceptably low fmax test results (all < 2, well under the generally accepted cutoff of 3). further, though the students in this design are not fully independent because they were in the same class, it is common for researchers to assume independence in evaluations of classroom behavior because the students are working independently. thus, the study appropriately met the requirements of normality, heterogeneity of variance, and independence necessary to conduct ttests and anovas. franz, t. and green, k. journal of the scholarship of teaching and learning, vol. 13, no. 5, december 2013. josotl.indiana.edu 97 c. procedure. the paranormal beliefs scale was administered at the beginning of the first class meeting, using a paper-and-pencil questionnaire. the same measure was also administered on the last day of the semester (14 weeks later). to guarantee anonymity, students selected a pseudonym that they used on both the pretest and posttest so that their responses could be matched. for comparison purposes, we also collected information from other sjfc students. unlike the previous study, these data were collected using an online survey. all undergraduate students (n = 2,703) were sent a link to the online survey and brief explanation of the study via email the week following finals but prior to commencement. they were given 10 days to complete the online survey. a total of 326 participants responded by at least starting the survey. participants in these comparison samples first saw a page with consent information, and were required to consent prior to participating. four did not consent, and their data were removed. respondents were asked to indicate their status (i.e., undergraduate or graduate students). those who declined to indicate status (n = 19) or indicated they were graduate students (n = 25) were immediately routed to the debriefing by the system. to exclude students who were in the classroom sample, respondents were also asked (a) whether they were science scholars and (b) what year they began at st. john fisher college. this resulted in the elimination of 12 more responses. finally, 38 participants did not fully complete the paranormal beliefs scale, for a total usable sample size of 228 participants. all phases of this study were reviewed and approved by the st. john fisher college institutional review board. each participant in our classes consented to allowing us to use his or her data in the study (those in the online survey who did not were immediately routed to the debriefing). all data were collected anonymously, and participants were debriefed either face-toface or with a written online paragraph, depending on the type of administration. d. analysis plan. the main analyses for the students in our classes utilized paired-samples t-tests to compare their pretest scores to those from their posttest. we also conducted one-sample t-tests comparing the scores from the students in our classes to the averages provided by tobacyk and milford (1983). finally, we compared posttest scores of the students in our classes to other students’ scores using a one-way anova. all analyses were conducted using the entire scale average as well as with all seven subscales. iii. results. a. analyses of research questions. the first research question asked whether students in our class had lower paranormal beliefs at the end when compared to their scores at the beginning. we tested this question using pairedsamples t-tests for the entire paranormal beliefs scale and all of the subscales. all but one posttest mean was statistically lower than the pre-test mean, and of those that were significant, all were in the medium to large range (table 2). the only subscale that did not decrease significantly was the belief in extraordinary life forms. thus, students had lower paranormal beliefs at the end of the course than they did at the beginning. franz, t. and green, k. journal of the scholarship of teaching and learning, vol. 13, no. 5, december 2013. josotl.indiana.edu 98 table 2. means and standard deviations for paranormal beliefs scale and each subscale from students in our class as a function of pretest-posttest administration (n = 43). pretest m posttest m t-test p cohen’s d entire paranormal beliefs scale (25 items) 2.71 2.34 7.10 <.001 .74 (.45) (.54) traditional religious beliefs scale 3.92 3.73 3.73 <.001 .20 (.88) (1.04) psi subscale (4 items) 2.48 1.94 5.14 .001 .77 (.73) (.67) witchcraft subscale (4 items) 2.41 1.91 5.56 <.001 .67 (.74) (.75) superstition subscale (3 items) 2.00 1.75 3.04 .004 .27 (.97) (.91) spiritualism subscale (4 items) 2.80 2.35 6.22 <.001 .59 (.73) (.79) extraordinary life forms subscale (3 items) 1.88 1.85 .44 .66 .03 (.88) (.97) precognition subscale (3 items) 3.22 2.68 4.69 <.001 .69 (.77) (.80) our second research question examined whether students in our class had lower paranormal beliefs at the end of the semester than the students in the original tobacyk and milford (1983) study. we tested this question using independent-sample t-tests comparing the pre-test and post-test means and standard deviations to the means and standard deviations from tobacyk and milford. as table 3 shows, all of the post-test means were significantly lower than the means from tobacyk and milford. interestingly, many of the pre-test means were also significantly lower. furthermore, as can be seen in the table, the nonsignificant finding from research question 1, which compared pre-test to post-test scores on the extraordinary life forms subscale, may be due to the fact that the science scholar students were significantly lower in pre-test means than tobacyk and milford’s. further, other research has reported anomalies in this subscale (aarino & lindeman, 2005). another aspect of the second research question examined whether students in our lc had lower paranormal beliefs at the end of their semester than the typical student at st. john fisher college. we tested this by comparing post-test scores using independent samples t-tests. as depicted in table 4, the results of these analyses demonstrated that the students in our learning community scored significantly lower on the entire paranormal beliefs scale than the typical st. john fisher college student. additionally, the students in our learning community scored lower on each of the paranormal beliefs subscales than the typical st. john fisher college student, although only four of seven of these comparisons were statistically significant. b. supplementary qualitative analysis on teacher-course evaluation comments. to answer the third research question, a supplementary qualitative analysis using the teachercourse evaluations across the two years of the study was conducted. the teacher-course evaluations ask for quantitative assessment and optional written feedback about areas such as course goals and objectives, aspects of the course students found beneficial, overall impressions, franz, t. and green, k. journal of the scholarship of teaching and learning, vol. 13, no. 5, december 2013. josotl.indiana.edu 99 and additional comments. without being asked or prompted, 14 students wrote that the course improved critical thinking, eleven used the term skeptic or skepticism, eleven discussed how course content improved their thinking and questioning, eight discussed how it improved their ability to evaluate and/or analyze, and one specifically discussed how it improved scientific reasoning. for example, one student wrote that the course “emphasized the critical thinking portion.” another wrote that the content “expanded our learning and made us skeptical.” another comment stated that “many discussions helped me to think scientifically and made me much more articulate.” finally, one student summed up the course experience by saying, “i learned to think more critically.” the unsolicited comments provide further evidence of meeting the goal of improving scientific reasoning and evidence-based critical thinking. table 3. means and standard deviations for paranormal beliefs scale and each subscale from students in our class (n = 43) compared to means from tobacyk and milford (1983) as a function of pretest-posttest administration. tobacyk & milford pre-test post-test m (sd) m (sd) cohen’s d m (sd) cohen’s d entire paranormal beliefs scale (25 items) 3.07 2.71* .55 2.34* 1.47 (.48) (.45) (.54) traditional religious beliefs scale 4.24 3.92* .36 3.73* .52 (.90) (.88) (1.04) psi subscale (4 items) 3.19 2.48* .90 1.94* 1.65 (.84) (.73) (.67) witchcraft subscale (4 items) 2.77 2.41* .45 1.91* 1.07 (.85) (.74) (.75) superstition subscale (3 items) 2.08 2.00 .09 1.75* .38 (.82) (.97) (.91) spiritualism subscale (4 items) 2.64 2.80 .21 2.35* .37 (.79) (.73) (.79) extraordinary life forms subscale (3 items) 2.82 1.88* 1.10 1.85* 1.07 (.83) (.88) (.97) precognition subscale (3 items) 3.52 3.22* .37 2.68* 1.02 (.84) (.77) (.80) * means differ from tobacyk & milford (1983) at p < .05. these self-reported changes were also seen in the ways the students presented and used evidence in their writing and in classroom activities, such as the team debate. for example, during we often witnessed the students challenging each other’s evidence and claims during the debate. one side would make an assertion, and students on the other side would immediately begin digging on the internet to locate the information and explore other evidence related to it. this led to much deeper debates and discussions than one might expect if students came prepared only to work with previously prepared notes, as it allowed for spontaneous exchanges and an analysis of new information as it was presented. it should be noted that students were directed that they be “engaged” in the debate even when they were not speaking, but that the instructors did not specifically require students to conduct on-the-spot searches to challenge the opposing side; the students carried out these activities on their own, powerfully demonstrating franz, t. and green, k. journal of the scholarship of teaching and learning, vol. 13, no. 5, december 2013. josotl.indiana.edu 100 some of the ways that they had internalized the concept of critical thinking. such displays occurred in all four years of the lc. table 4. means and standard deviations for post-test paranormal beliefs scale and each subscale from students in our class (n = 43) compared to means from the faculty (n = 86) and other students (n = 228). post-test other sjfc students m (sd) m (sd) cohen’s d entire paranormal beliefs scale (25 items) 2.34 2.74* .69 (.54) (.62) traditional religious beliefs scale 3.73 3.84 .11 (1.04) (.91) psi subscale (4 items) 1.94 2.62* .87 (.67) (.88) witchcraft subscale (4 items) 1.91 2.38* .58 (.75) (.87) superstition subscale (3 items) 1.75 1.94 .22 (.91) (.83) spiritualism subscale (4 items) 2.35 2.77* .50 (.79) (.88) extraordinary life forms subscale (3 items) 1.85 2.05 .20 (.97) (.95) precognition subscale (3 items) 2.68 3.32* .74 (.80) (.92) * means differ from the posttest mean at p < .05. iv. discussion and implications for practice. our evaluation demonstrated that, at least as measured using the paranormal beliefs scale, an interdisciplinary lc that emphasizes the scientific method through the use of writing and other active-learning techniques can decrease students’ pseudoscientific thinking. specifically, at the end of the semester students in the lc scored lower on the paranormal beliefs scale than (a) they did in the beginning of the semester, (b) than other students at the college, and (c) than mean scores (used as norms) provided in the original research by tobacyk and milford (1983). this decrease was not the primary goal for the lc; indeed, it was largely a product of students’ applying the main topics and tools of the course to the content (pseudoscience) we chose as a vehicle for exploring critical thinking. pseudoscience was selected as the content primarily for two reasons. first, we expected it to be engaging, allowing students to consider a variety of situations and ideas in different contexts that can be entertaining. second, the lc was designed for students from biology, chemistry, computer science, mathematics, and physics. pseudoscience allowed the students to bring many of these disciplines into the discussion, providing an interdisciplinary approach that we value. our lc examined scientific reasoning and critical thinking through the use of formal writing, informal writing, debates, group projects, in-class group exercises, and problems that compared pseudoscientific concepts to scientific counterparts. through this comparison, students were compelled to analyze and evaluate claims using a generalized scientific method. franz, t. and green, k. journal of the scholarship of teaching and learning, vol. 13, no. 5, december 2013. josotl.indiana.edu 101 specifically, our course was structured around shermer’s (2002) classification of where thinking goes wrong, including problems in scientific thinking, problems in pseudoscientific thinking, logical problems in thinking, and psychological problems in thinking. thus, the lc seems to have served the first-year science scholar students well in meeting their needs to understand scientific thinking and improve their writing without being indoctrinated into a specific discipline. further, the class allowed students to extend critical thinking and scientific reasoning concepts beyond what they typically experience in the classroom and/or a laboratory course. the students in the lc were drawn from several different majors in the sciences and mathematics. thus, while the ideas of scientific thinking were also being taught in other courses, the only common experience to all students was this lc. even when critical thinking processes are made explicit, they tend to be discipline-focused so that biology courses develop thinking like a biologist while physics courses teach how to think like a physicist. discipline-focused reasoning can limit both the tools one uses in reasoning as well as the domain to which the reasoning is applied. the lc described here, as well as the scale used, lies far from any of these particular science disciplines. thus, one can reasonably conclude that the writing intensive, interdisciplinary experience of the lc was one of the tools that furthered their ability to analyze such claims and consider evidence. other colleges and universities could easily modify this lc and apply it to creating an lc that serves their students because the techniques and evaluation described here are quite portable. the active learning techniques can be modified and used by instructors at most institutions regardless of academic emphasis or size. the writing projects and classroom debates could easily be incorporated into other courses, and the taking sides text is available in many areas (such as climate change) that would provide a starting point for such activities. finally, by applying critical thinking skills and scientific reasoning to pseudoscientific topics, students were highly engaged, a necessary first step to learning. v. limitations and future research. as with any classroom-based research, this study has limitations. for example, our students were a select group of high performers, making it quite easy for us, as novice writing instructors, to work with them on improving their writing. second, we only evaluated two of the four years when we taught the course. however, we modified and improved it considerably after the first year so the evaluation would be of a considerably different course. further, we kept the course structure mostly constant among the final three of our four years, so an evaluation during the fourth year would likely yield similar results to that of years two and three. finally, we used the paranormal beliefs scale (tobacyk & milford, 1983) as a surrogate measure of scientific reasoning. given the focus of our course on contrasting pseudoscientific thinking with scientific reasoning, this measure is likely reasonable at tapping some of the thinking around pseudoscience. the paranormal beliefs scale by itself may be only a surrogate measure of scientific reasoning and critical thinking, but the results from this study are supported by other, anecdotal data from the course. one such experience relates to the way a group of the students managed to turn our initial psychic demonstration against us. during a later class activity exploring the statistics of esp-type card guessing, several students in the class achieved a perfect 10/10 ratio guessing the cards with four different card viewers. random guessing would only explain this event as a once-in-the-history of the universe likelihood. after a discussion of this, the students franz, t. and green, k. journal of the scholarship of teaching and learning, vol. 13, no. 5, december 2013. josotl.indiana.edu 102 admitted to having created a method of “tells” so that a confederate would easily know what kind of card was being viewed simply from the way the viewer held the card. such actions clearly demonstrate students moving toward deeper critical thinking in the sense of bloom’s taxonomy (anderson & krathwohl, 2001), moving well beyond the lower levels of reasoning and far into the application and creation levels of reasoning. with respect to future research, critical thinking can be evaluated either by using the literature to identify or develop a content-specific critical thinking scale (renaud & murray, 2008) or instead by using a general critical thinking scale (e.g., cheung, rudowicz, kwan, & yue, 2002). specific critical thinking scales, such as those to better understand scientific reasoning about biology (mcmurray, beisenherz, & thompson, 1991), critical thinking about diversity (pascarella, palmer, moye, & pierson, 2001), or for evaluating paranormal beliefs (tobacyk & milford, 1983) can measure content-specific critical thinking. on the other hand, any instructor who modifies our course content could also consider general methods of evaluating critical thinking, including the widely used watson-glaser critical thinking appraisal short form (watson & glaser, 2008), which has at least some use in assessing general critical thinking (e.g., loo & thorpe, 1999). another future question that remains is whether a course such as this would work beyond our self-selected scholarship students or with non-science students. as reported, we had a set of scholarship students who were, on average, highly motivated, more prepared for college than the typical fisher student, and focused on learning science in their first year. programs like this tend to have a critical thinking emphasis (inkleas & weisman, 2003). regardless of our sample, the active methods used in the course should help to motivate many students (yoder & hochevar, 2005). further, our interdisciplinary focus improved general critical thinking and scientific reasoning within and beyond the classroom and could apply to students who are not science majors. thus, it is likely that this will work beyond the sample, and future courses should test and evaluate its reliability as an instructional approach. learning communities are used, in part, to improve retention rates and student satisfaction. while these are admirable goals, lcs can also be used to improve general scientific reasoning and critical thinking (browne & minnick, 2005) and also, as this research demonstrates, can improve science-specific reasoning. acknowledgments we thank our students for participating in this evaluation as well as steve brauer and theresa nicolay for reading and commenting on earlier drafts of the manuscript. references aarnio, k., & lindeman, m. (2005). paranormal beliefs, education, and thinking styles. personality and individual differences, 39, 1227-1236. aaron, j. e. (2003). the little, brown essential handbook for writers. new york: addisonwesley educational publishers. adey, p., & shayer, m. (1993). an exploration of long-term far-transfer effects following an extended intervention program in the high school science curriculum. cognition and instruction, 11(1), 1-29. franz, t. and green, k. journal of the scholarship of teaching and learning, vol. 13, no. 5, december 2013. josotl.indiana.edu 103 anderson, l., & krathwohl, d. (eds) (2001) a taxonomy for learning, teaching and assessing: a revision of bloom’s taxonomy of educational objectives. new york: longman. browne, m. n., & minnick, k. j. (2005). the unnecessary tension between learning communities and intellectual growth. college student journal, 39, 775-783. cheung, c., rudowicz, e., kwan, a. s. f., & yue, x. d. (2002). assessing university students' general and specific critical thinking. college student journal, 36, 504-525. committee on undergraduate science education, national research council. (1999). transforming undergraduate education in science, mathematics, engineering, and technology. washington dc: national academy press. cook, t. d., & campbell, d. t. (1979). quasi –experimentation: design & analysis issues for field settings. boston: houghton-mifflin company. dabney, d., green, l., & topalli, v. (2006) freshman learning communities in criminology and criminal justice: an effective tool for enhancing student recruitment and learning outcomes. journal of criminal justice education, 17(1), 44-68. davies, m. (2013). critical thinking and the disciplines reconsidered. higher education research & development, 32(4), 529-544. dodge, l., & kendall, m. (2004). learning communities. college teaching, 52(4), 150-5. druger, m. (2002). more than science. journal of college science teaching, 31(5), 340 – 1. easton, t. a. (2002). taking sides: clashing and controversial views in science, technology, and society, 5th ed. gilford, ct.: dushkin/mcgraw-hill. easton, t. a. (2004). taking sides: clashing and controversial views in science, technology, and society, 6th ed. gilford, ct.: dushkin/mcgraw-hill. farha, b., & steward, g. (2006). paranormal beliefs: an analysis of college students. the skeptical inquirer, 30(1), 37-40. halpern, d.f. (1997). critical thinking across the curriculum: a brief edition of thought and knowledge. mahwah, nj: lawrence erlbaum associates. halpern, d.f., (1998) teaching critical thinking for transfer across domains: dispositions, skills, structure training, and metacognitive monitoring. american psychologist, 53, 449-455. hergovich, a., & arendasy, m. (2005). critical thinking ability and belief in the paranormal. personality and individual differences, 38, 1805-1812. franz, t. and green, k. journal of the scholarship of teaching and learning, vol. 13, no. 5, december 2013. josotl.indiana.edu 104 kokkala, i., & gessell, d. a. (2002). writing science effectively: placing biology and english students in an author-editor relationship. journal of college science teaching, 32, 252-57. inkelas, k.k., & weisman, j.l. (2003). different by design: an examination of student outcomes among participants in three types of living-learning programs. journal of college student development, 44, 335-368. inkelas, k.k., soldner, m., longerbeam, s.d., & leonard, j.b. (2008). differences in student outcomes by types of living-learning programs: the development of an empirical typology. research in higher education, 49, 495-512. inkelas, k.k., daver, z.e., vogt, k.e., & leonard, j.b. (2007). living-learning programs and first-generation college students’ academic and social transition to college. research in higher education, 48, 403-434. learning communities national resource center. (n.d.). washington center for improving the quality of undergraduate education at evergreen state college retrieved on january 30, 2013 from http://www3.evergreen.edu/wacenter/presources.asp?pid=73 lester, j. d., & lester, j. d., jr. (2003). principles of writing research papers. new york: pearson education. loo, r., & thorpe, k. (1999). a psychometric investigation of scores on the watson-glaser critical thinking appraisal new form s. educational and psychological measurement, 59, 9951003. mcmurray, m. a., beisenherz, p., & thompson, b. (1991). reliability and concurrent validity of a measure of critical thinking skills in biology. journal of research in science teaching, 28, 183-191. messer, w.s., & griggs, r.a. (1989). student belief and involvement in the paranormal and performance in introductory psychology. teaching of psychology, 16, 187-191. moore, t. (2013). critical thinking: seven definitions in search of a concept. studies in higher education, 38(4), 506-522. morgan, p., carter, j., lemons, j., grumbling, o., & saboski, e. (1995). preliminary assessment of a science learning community for first-year students. journal of college science teaching, 25(2), 102-109. national research council. (1998). national science education standards, fifth printing. washington, d.c.: national academy press. franz, t. and green, k. journal of the scholarship of teaching and learning, vol. 13, no. 5, december 2013. josotl.indiana.edu 105 nelson, c. e. (1999). on the persistence of unicorns: the trade-off between content and critical thinking revisited. in r. aminzade and b. pescosolido (eds.), the social worlds of higher education: a handbook for teaching in a new century (168-184). thousand oaks, ca: pine forge press. pascarella, e. t., palmer, b., moye, m., & pierson, c. t. (2001). do diversity experiences influence the development of critical thinking? journal of college student development, 42, 257-271. quitadamo, i, brahler, c.j., & crouch, g.j. (2009). peer led team learning: a prospective method for increasing critical thinking in undergraduate science courses. science educator, 18, 29-39. renaud, r. d., & murray, h.g. (2008). a comparison of a subject-specific and a general measure of critical thinking. thinking skills and creativity, 3(2), 85-93. roe, c.a., (1999). critical thinking and belief in the paranormal: a re-evaluation. british journal of psychology, 90, 85-98. royalty, j., (1995). the generalizability of critical thinking: paranormal beliefs versus statistical reasoning. the journal of genetic psychology, 156, 477-488. schön, d. (1987). educating the reflective practitioner. san francisco: jossey-bass. shermer, m. (2002). why people believe weird things: pseudo-science, superstition, and bogus notions of our time. new york, ny: henry holt and company, llc. smith, b.l., macgregor, j., matthews, r.s., & gabelnick, f. (2004). learning communities: reforming undergraduate education. san francisco john wiley & sons, inc. tobacyk, j., & milford, g. (1983). belief in paranormal phenomena: assessment instrument development and implications for personality functioning. journal of personality and social psychology, 44, 1029-1037. watson, g.b., & glaser, e. m. (2008). watson-glaser critical thinking appraisal short form manual. san antonio, tx: pearson education. yoder, j. d., & hochevar, c.m. (2005). encouraging active learning can improve students’ performance on examinations. teaching of psychology, 32, 91-95. microsoft word v9n2sullenberger-8019.doc journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009, pp. 1 9. enhancing the success of sotl research: a case study using modified problem-based learning in social work education sabrina williamson 1 and valerie chang 2 abstract: this article describes a study which utilizes modified problem-based learning (mpbl) as a teaching method in undergraduate social work practice classes. the authors report both qualitative and quantitative findings of the research. additionally, the authors reflect on the use of the mpbl method and on the lessons learned throughout this research on the scholarship of teaching and learning. keywords: problem-based learning, transfer of learning, social work practice i. introduction. a primary objective of postsecondary education is to prepare students for their professional careers. as such, when students declare a major and begin to pursue studies in a particular area, they begin a trajectory of movement from thinking like a student to thinking like a professional in their chosen discipline. in any given field, thinking like a professional involves discerning meaning from information that is presented to them, organizing knowledge around major principles, responding to changes in context, and accessing and retrieving knowledge smoothly (thompson, licklider, and jungst, 2003). in schools of social work, as with other professional schools, faculty are responsible for developing and delivering curriculum that aids in the student’s transition from novice to professional (koerin, harrigan, and reeves, 1990). in social work educators must identify and teach the step-by-step thinking process used by professional social workers and also be sure that students know how to appropriately apply knowledge and use professional skills as they work with clients. however, despite best efforts students often experience disconnects in their learning when they move from the classroom to professional practice (lager and robbins, 2004). classroom knowledge often is not transferred to the real world experience. consequently, students struggle in their internships as they either don’t use what they learned in class or apply very little of what they learned. in this article we will describe a study in which modified problem-based learning (mpbl) was used in two undergraduate social work practice courses to teach students the process of thinking like professional social workers. we will also discuss the “lessons learned” by the authors during the research process. during phases of data collection and analysis, the authors learned invaluable lessons about teaching and learning and about research in the field of scholarship of teaching and learning. we hope our reflections on this project will help other faculty as they develop their courses and consider engaging in sotl projects. the authors wish to thank indiana university’s mack center for inquiry on teaching and learning for their 1 school of social work, indiana university bloomington, 1127 atwater, bloomington, in 47401, sabawill@indiana.edu 2 school of social work, indiana university purdue university indianapolis, 902 west new york street, indianapolis, in, vchang@iupui.edu williamson, s. and chang, v. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 2 support in this project, both in funding the authors as faculty fellows and in providing guidance and direction throughout the study. ii. literature review. problem-based learning is a learner-centered approach in which learners “conduct research, integrate theory and practice and apply knowledge and skills to develop a viable solution to a defined problem” (savery, 2006, p.12). in social work, the “problem” that students receive is a client scenario that is representative of a real-life situation that an individual, family, group or community might face. students receive parts of the case a section at a time, replicating the process of working with a case in the field. additionally, pbl is pedagogy based on the premise that acquiring process skills is as important as assimilating content (margetson, 1991.) using problem-based learning, students are presented with scenarios faced by social workers in the field and are challenged to find information, apply previously acquired knowledge and work with their colleagues (other students) in planning for assessment and developing interventions. as students work on the problem-based learning cases, faculty can help them learn the process of “thinking like a professional” (middendorf and pace, 2005). pbl is similar to case-based learning with the primary difference between the two being the manner in which information relevant to the case or problem is presented to the students. typically, instructors utilizing case-based learning techniques present an entire case study to student groups rather than in the discrete parts utilized in problem-based learning. course instructors help students analyze the problem, consider solutions, and determine the actions that a professional would take in the situation (herreid, 1994). in addition to its use in social work education, problem-based learning has also been utilized extensively in other professional schools, including business (c.f. saatci, 2008), medicine (c.f. spencer, 2003), and education (c.f. edwards and hammer, 2005). in social work, lam (2004) reports on a bsw program that utilizes pbl in four courses throughout the curriculum. this study measured students’ change in competence, values, and clinical skills before and after pbl instruction and compared these scores with pre-test and post-test scores of students in prior years who had completed the traditional (non-pbl) curriculum. lam found that students who received pbl instruction were more efficient in searching out information and in taking responsibility for their learning. other program faculty, however, rated the performance of pbl students as weaker than non-pbl students in written examinations concerning theories and practice skills. similar to lam’s finding, chan and ng (2004) reported that graduates from pbl training module were found to be “more self directed and creative than their predecessors” (p.316). altshuler and bosch (2003) used pbl in school social work policy and school social work practice classes to prepare students for work in various school settings. at the end of the pbl course work, they sought student feedback concerning perception of content and skills learned. overall, the students thought that the pbl approach was conducive to learning. gelman and mirabito (2005) discuss using pbl techniques with msw students specifically to teach crisis intervention. while not an empirical paper, the authors present the vignettes used to teach students and provide teaching points about how instructors can utilize this method in the classroom. while other authors have written about modifying problem-based learning techniques (c.f. goodnough, 2005 and baldwin, bankston, anderson, echtenkamp, haak, smith and williamson, s. and chang, v. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 3 iatridis, 2002), there does not seem to be uniformity in what modifications are made. a method for using modified problem-based learning (mpbl) in social work education was developed by chang and is described in chang, scott, decker (2009). like pbl, students using mpbl receive a section of a case at a time. this replicates what happens in most disciplines where the student has basic introductory information initially and gradually learns more. also like pbl students work in groups in order to share perceptions and knowledge. unlike pbl the students lead their own groups. with traditional pbl trained teaching assistants work with each student group. the teaching assistants are trained to ask the kind of questions that lead students to explore more deeply and to think like professionals. since we do not have the resources for trained teaching assistants, we modified pbl first by teaching students to be leaders in their groups and second by following each section of a case with the kind of questions that professionals use to guide their thinking. the instructors provided consultation and guidance to the student groups as they used their life experience, knowledge from this course and previous courses, and new material found by doing research to fill information gaps to discuss and answer the case questions. iii. description of study. this study was conducted on two campuses of a midwestern school of social work with undergraduate students. one campus is in a large urban area where students are predominantly commuters. the other campus is in a traditional college town setting where the majority of students live on or adjacent to campus. mpbl was simultaneously used in two undergraduate practice classes, one on each of the campuses. each course has the same course objectives and uses the same textbooks. students take this course in the fall semester of their junior year as their first introduction to practice skills prior to any practice experiences. the research team included the two social work faculty members who taught the practice course at the two research sites, a postdoctoral fellow who has taught social work courses, and a graduate student/research assistant. all of the data was collected by the faculty members. the entire team participated in the analysis of the data. human subjects approval was obtained from the university institutional review board prior to the start of data collection. a. instructional methods. in each course, instructors spent the first four to five weeks of the semester reviewing and introducing theoretical foundations of social work practice. after completing this section of the course, the instructors introduced the concept of modified problem-based learning. students were told that they would work in a small group. each group would have a case that they would work with for the remainder of the fall semester. instructors of the classes indicated that they would serve as consultants to the groups, but that the students were responsible for responding to the questions about the case. in the first week of working on the cases, each student group was given the first section of a case and questions related to that section. the fourteen questions on part one of the case were designed to help students prepare to work with a client by identifying facts and knowledge needed and how they planned to work with and collaborate with their clients. questions related to the first section of the case included: williamson, s. and chang, v. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 4 what are the key facts in the case? what additional information will you need prior to meeting this client? what concerns do you have about working with this client? what are your preliminary impressions related to the case? (chang, scott, decker, 2009, p. 79) following each additional section of the case, students received additional questions to guide their thinking about the case. questions asked about such things as changes in initial impressions, additional information needed, problems and goals identified, treatment plan developed, level of motivation, appropriate action, and eventually ending the work with the client. student groups were given in class time to discuss their case, to share their perceptions and information gained. each group received a different case and turned in a written report answering each question. after they turned in their work related to part one, they received part two of the same case. again, each group was expected to collaborate and to turn in a written document that addressed the questions related to the second part of the case. each group received written feedback from the instructor concerning their responses to the questions. in one class toward the end of the semester, students were given time to share information about their cases and their responses to the questions about the cases. each group engaged in a dialogue with the instructor and other students about their thinking related to their case. in the other class students reported on and discussed their case on the day they turned in each case report. in this class there were several discussions about the cases during the semester. the mpbl work was done in conjunction with more traditional course work of lecture, discussion, and role-plays. parts one and two of the cases were distributed in the fall semester in the first practice class. students received the remaining five parts of their respective cases in their next practice course. b. sample. the sample consists of thirty-five students. this represents 90% of the combined population of the two practice classes. ninety-seven percent of the sample was female, 68% were traditional college aged (20 to 24) and 37% had worked in social service type jobs during the previous year. in terms of ethnicity, the sample consisted of 83% who identified as caucasian, 9% who identified as hispanic, 6% who identified as african-american, and 3% who identified as other. c. data collection and analysis. during the first practice course, each student in the research project completed individual pretests and posttests related to a case that had not been assigned to any of the groups. in the pretest, students answered questions related to parts one and two of the case. this pretest was conducted at the end of the first course module; i.e. after theoretical foundations of practice had been reviewed but before practice skills were introduced and before students began working in their mpbl groups. the post-test was identical to the pre-test and was administered at the end of the semester after students had been working in modified problem-based groups and after they had been exposed to new information and skills in the course. pre-test and post-test data were analyzed both quantitatively and qualitatively. quantitatively, a rubric was developed and used to score each student’s answers on the pre-test and posttest. the rubric, which consisted of answers to the case questions that are consistent with best practices in the social work profession, enabled researchers to evaluate the information williamson, s. and chang, v. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 5 on each test. using spss, t tests were then run to determine if there were significant differences between each student’s pre-test and posttest. additionally, each student’s answers to the pre and post-test were entered into a database and then analyzed in atlas-ti. content analysis (stemler, 2001) was used to determine qualitative differences in the students’ answers that might indicate improvement in the path of “thinking like a professional”. iv. findings. quantitatively, the findings were non-significant. of course with such a small sample it would be very difficult to have significant findings. qualitatively, there is indication that many students showed improvement from pre-test to post-test. student comments on the experience of using modified problem-based learning illustrate many of these areas of improvement. students appreciated the fact that the clients presented in the cases were real, not “made up”, and that the problems faced by the social worker were ones similar to what they would face in their future practice. students’ reports included comments such as: “it was good to apply what we were learning to real people,” “it allowed me to get a real picture of social work,” “it was the next best thing to working in the field.” students also benefited from working with their peers. they said, “it was useful to learn to collaborate in a group and research relevant material” and “being able to discuss situations with other people was great and it helped me prepare to meet with others to professionally and respectfully discuss issues.” students also recognized that the use of cases assisted them in thinking critically about client situations. student comments to support this theme included, “this process taught me to look at a problem from all angles”, “using the cases helped me to think of possible ethical dilemmas and value conflicts” and “this taught me the importance of researching more than the obvious information.” finally, students perceived that the use of cases helped them be more effective and confident in their field placements. they said that “using cases helped me to have the opportunity to experience case management before entering my practicum” and “it helped me become more efficient in my field placement” and “using cases helped me to realize the magnitude of my work, that everything isn’t easy and that there will always be bumps in the road”. in summary, doing mpbl enhanced students’ confidence in their readiness to work with clients, increased their ability to think like professionals, to use professional vocabulary, to understand the need for assessment throughout the life of the case, and to realize the on-going need for more knowledge related to client issues, background and culture, and necessary about resources. while this is gratifying to see, without a control classroom the question remains as to whether these students showed this type of improvement because of mpbl or if they would have improved in similar ways in a traditional practice course. v. lessons learned about teaching and learning. reflecting on this project, we learned ways to improve our teaching. first, it is important to do a pre-test at the beginning of the semester to assess students’ knowledge related to all of course objectives as well as a pre-test related to a case. the pre-test used in this project did not cover all the course objectives. a better pre-test would invite students to evaluate their knowledge and skills related to all the course objectives. if on the pre-test a student identifies that they have knowledge related to a course objective, a follow-up question should ask them to identify how the knowledge was acquired, e. g., by reading, in another course, or on the job. reviewing the williamson, s. and chang, v. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 6 pre-test allows the instructor to tailor the course to the needs of the students. after identifying the strengths that the students bring into the course, the instructor can develop ways to build on these strengths. for example, in the pre-test a number of students identified the importance of building a relationship and establishing rapport in the pre-test. if the instructor had reviewed the pre-test, s/he would have noticed this strength and been able to enhance and build on it. in the post-test some of these students focused on exploring what was wrong with the client and didn’t mention building a relationship with the client. the pre-test also gives information about the gaps in the students’ knowledge. knowing the students’ areas of weakness from the beginning of the course allows instructors to tailor their teaching and assignments to help students’ master key course objectives. using mpbl, instructors can design questions that invite students to explore, discuss and learn information and skills important to their professional growth. another lesson is the importance of allowing time for instructor follow through. if the students had been given the post-test a few weeks before the end of the semester, rather than on the last day of class, instructors could have reviewed the post-test answers and helped the students correct any misunderstandings. for example, some students wrote responses that focused on an aspect of the case that was accurate but less important than other aspects of the case. after realizing this problem, the instructor could have helped the students think through the case again and learn to identify the most important aspects of the case. since applying concepts to social work practice requires higher level learning, students need additional opportunities to practice. therefore, follow through should continue in subsequent practice classes as well as in the field practicum discussion seminar. to enhance student learning, the field practicum seminar should emphasize the concepts learned in the previous course and require students to use the same mpbl case questions as they work with actual clients. the field seminar assignments should require students to demonstrate using the same thinking skills developed by using mbpl in the previous practice classes. a third important lesson is the importance of establishing measureable, achievable course objectives. for this course the objectives were established by a committee that was focusing on accreditation standards rather than what was possible. as educators interested in scholarly teaching we need to continue to take a stand on the establishment of measureable, achievable objectives; otherwise instructors often try to put more content into each class than students can retain and appropriately use in practice. in this course the pressure to cover course objectives that include too much led the instructors to use lecture, role-plays, and mpbl. the instructors were caught between their belief in the value of active collaborative learning and their requirement to cover a great deal of content. this leads to another important lesson. adding a new approach or method of teaching can be positive, but in the case of mpbl it should have been the central focus and main teaching and learning method. adding new approaches takes time. to enhance the value of using mpbl, more class time needs to be allowed for the student groups to discuss, process, work on the cases. each student should have written out all the case answers before the group discussion of the case answers. other course assignments should be structured to emphasize or use the work with the cases. each student group should have regular opportunities to discuss their thinking about each case with the whole class. we know that classroom assessment of learning is very important (angelo and cross, 1993). mpbl is an excellent way to assess learning. since during the semester, the student groups were able to appropriately answer the case questions, the instructors believed the students williamson, s. and chang, v. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 7 had learned to appropriately apply content to cases. however, at the end of the semester, some individual students were not able to replicate best answers to the same questions related to a different case. the students reported that working with the cases helped them feel more confident as they approached working with clients in field placement. if the instructors had spent more time on mpbl, all the students might have not only felt more confident but individually been able to demonstrate greater ability to critically think through case material. the lesson for us is that multiple ways of assessing learning should be used. specifically we should have had individual students answer the case questions. the instructor who could identify thinking problems should evaluate these individual answers. the group could work with all the individual answers as well as do additional research to develop more sophisticated, professional responses to each question. vi. lessons learned about sotl research. besides lessons about teaching, we also learned some important lessons about research on teaching and learning. before starting this research we studied the use of case-based and pbl with graduate students and were excited about the value of using mpbl with undergraduate students. the best course to introduce mbpl was a practice course that is offered only in the fall semester. unfortunately, by the time we had developed our ideas the fall semester was starting. not wanting to wait until the next fall, we jumped into the project too quickly. both authors were teaching this practice course. each course had the same objectives and same textbooks and we thought our plans for using mbpl were the same. looking back we identified that our methods of using mbpl were similar but had several differences. one author spent more class time on mpbl than did the other author. one author had student groups report and discuss their cases more frequently during the semester. in retrospect we are aware that we should have taken the time to write a manual with step-by-step directions for using mbpl. having a written manual would also allow the study to be replicated. also, we could have had one class use mpbl and had the other class serve as the control group. this method would have strengthened our research design. if we had a control class, we would have been able to assess whether the changes that occurred could be attributed to using mpbl. we didn’t use a control class because both of us thought mpbl was a better way to teach this course. our commitment to offering students what we considered the best teaching and learning method meant that, ethically, we could not utilize a control group. (even if the use of a control group had been possible, however, sample size was too small to demonstrate significant findings.) although most students showed good improvement from the pre-test to the post-test some students did not improve and some had less satisfactory answers on the post-test. there could be many reasons for this decline. we believe a significant reason was that the post-test was given on the last day of class when the students were anxious to get finished. some of their answers were very brief and seemed hurried. in the future we will not wait until the final class to do the post-test. finally, with more time we could have followed the students into their field practicum setting to assess whether what they learned using mpbl enhanced their ability to think like professional social workers in actual practice. although this would strengthen the research, it would be very complicated to control because each student is in a different setting with a different field instructor. even with a control group there would be many uncontrollable variables. given all of these complications, we still believe that meeting with the students at the williamson, s. and chang, v. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 8 end of their practicum semester to discuss their perceptions of the value of mpbl to their education would have been valuable. vii. conclusion. as indicated in preceding paragraphs, each author believed that utilizing mpbl in practice classes was the best way to teach social work students how to think like professional social workers. each certainly expected the method to show success. when the preand post-tests were analyzed and the evidence of success was inconclusive, all members of the research team were somewhat disappointed because the level of student improvement was less than what we had hoped. however, the lessons learned about the mpbl method and about conducting sotl research have been invaluable. just as there is a process of learning to “think like a professional,” there is, perhaps, a process of learning to “think like a sotl researcher” as well. it is our hope that other sotl researchers will continue this research using mpbl with students in other disciplines. as researchers in other disciplines have found (lundeberg, 1999; wolfer and scales, 2006), using cases can help students transfer learning from the classroom to the world of work. in mpbl the instructor models for students the questions that professionals use as they approach challenging case situations. working with these questions seems to be a logical way to help students in any discipline learn how to think about challenging case situations. references altshuler, s.j., and bosch, l.a. (2003). problem-based learning in social work education. journal of teaching in social work, 23(1/2), 201-215. angelo, t.a. and cross, k.p. (1993). classroom assessment techniques: a handbook for college teachers. san francisco: jossey-bass. baldwin, w., bankston, p., anderson, w.m., echtenkamp, s., haak, r., smith, p, iatridis, p.g. (2002). can students in a modified pbl curriculum exceed the national mean on usmle part 1? medical education, 36 (8), 791-795. chan, c.l.w. and ng, s.m. (2004). the social work practitioner-researcher-educator: encouraging innovations and empowerment in the 21 st century. international social work, 47(3), 312-320. chang, v.n. ,scott, s.t., and decker, c.l. (2009) developing helping skills: a step by step approach. belmont, ca: brooks/cole. edwards, s. and hammer, m. (2005). laura’s story: using problem-based learning in early childhood and primary teacher education. teaching and teacher education, 22 (4), 465-477. gelman, c.r. and mirabito, d.m. (2005). practicing what we teach: using case studies from 9/11 to teach crisis intervention from a generalist perspective. journal of social work education, 41(3), 479-494 williamson, s. and chang, v. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 9 goodnough, k. (2005). issues in modified problem-based learning: a self-study in pre-service science-teacher education. canadian journal of science, mathematics, and technology education, 5 (3), p289-306. herreid, c.f. (1994). case studies in science education. journal of college science teaching (23) 221–229. koerin, b. b., harrigan, m. p., and reeves, j. w. (1990). facilitating the transition from student to social worker: challenges of the younger student. journal of social work education. 26(2), 199-208. lager, p.b. and robbins, v.c. (2004). field education: exploring the future, expanding the vision. journal of social work education, 40 (1), 3-12. lam, d. (2004). problem-based learning: an integration of theory and field. journal of social work education, 40(3), 371-391. lundeberg, m. a. (1999). discovering teaching and learning through cases. in m. a. lundeberg, b. b. levin, and h. l. harrington (eds.), who learns what from cases and how? the research base for teaching with cases (pp. 3 – 23). mahwah, nj: lawrence erlbaum. margetson, d. (1991). why is problem-based learning a challenge? in d. boud and g. geletti (eds.), the challenge of problem based learning. new york: st. martin's press, pp. 42-50. middendorf, j., and pace, d. (2005). decoding the disciplines: a model for helping students learn disciplinary ways of thinking. new directions for teaching and learning, 98, 1-12. saatci, e. (2008). problem-based learning in an intercultural business communication course. journal of business and technical communication, 22 (2), 237-260. savery, j.r. (2006). overview of problem-based learning: definitions and distinctions. the interdisciplinary journal of problem-based learning, 1(1), 9-20. spencer, j. (2003). abc of learning and teaching in medicine: learning and teaching in the clinical environment. bmj, 326, 591-594. retrieved november 29, 2008 from http://www.bmj.com/cgi/reprint/326/7389/591 stemler, s. (2001). an overview of content analysis. practical assessment, research and evaluation, 7(17). retrieved january 9, 2007 from http://pareonline.net/getvn.asp?v=7andn=17 thompson, j., licklider, b., and jungst, s. (2003). learner-centered teaching: postsecondary strategies that promote ‘thinking like a professional’. theory into practice, 42(2), 133-141. wolfer, t.a and scales, t.l. (eds.) (2006). decision cases for generalist practice: thinking like a social worker. belmont, ca, brooks/cole. journal of the scholarship of teaching and learning, vol. 23, no. 1, april 2023, pp.14-30. doi: 10.14434/josotl.v23i1.32675 the impact of undergraduate research experience intensity on measures of student success donna chamely-wiik florida atlantic university dchamely@fau.edu anthony ambrosio florida atlantic university tracy baker lynn university amrita ghannes florida atlantic university jennie soberon florida atlantic university abstract: despite the growing interest to provide research engagement opportunities to undergraduate students, few studies have investigated how engagement “intensity” impacts measures of student success. a quasi-experimental, matched-subject design was employed to study differences between varying levels of research experience intensity (i.e., experienced, novice, control groups) on graduating gpa, time to graduate, and type of post-graduation experience. results indicated that experienced students had significantly higher graduating gpas than novice or control students, and both research groups had significantly lower time to graduate than the control group. findings also indicated experienced student researchers are significantly more likely to progress to graduate school than either novice research or control students. implications for implementing research initiatives are discussed. keywords: undergraduate research, student success, gpa, post-graduation activities, time to graduation undergraduate research (ugr) is one of ten high impact practices shown to enhance and improve college student performance and success (kuh, 2008). the council on undergraduate research (cur) defined undergraduate research as “an inquiry or investigation conducted by an undergraduate student to make an original intellectual or creative contribution to a discipline” (council on undergraduate research, 2018, para. 3). undergraduate research experiences can include both curriculum-based opportunities and co-curricular opportunities (beckman & hensel, 2009; batttaglia et al., 2022). course-based undergraduate research experiences (cures), such as capstone courses, integrate undergraduate research into course curriculum and materials (batttaglia et al., 2022, chamely-wiik et al., 2014). co-curricular undergraduate research involves research activities outside the classroom such as summer research internships or fellowships, participating in summer research programs, presenting at poster sessions, and contributing meaningfully to research projects (kardash, 2000; pender, 2010; thiry et al., 2012). at the individual skill level, curricular and co-curricular activities have been defined as those activities that involve “analyzing data, interpreting findings, and deliberating about next steps of an experiment” (thiry et al., 2012, p. 264). mailto:dchamely@fau.edu chamely-wiik, ambrosio, baker, ghannes, and soberon journal of the scholarship of teaching and learning, vol. 23, no. 1, april 2023. josotl.indiana.edu multiple studies have demonstrated the importance of curricular and co-curricular ugr experiences on promoting inquiry skills and learning outcomes such as critical thinking and communication (thiry et al., 2011; monarrez et al., 2020) student success metrics (baron et al., 2020; brown et al., 2020; simmons, 2018), student preparedness, and career clarification (craney et al., 2011; frederick et al., 2021). russell et al. (2007) found that undergraduate research opportunities increased students’ understanding of how to conduct a research project, confidence in their research skills, and awareness of what graduate school entails. students who participate in ugr are also more likely to develop an expansive skillset. mcdevitt et al. (2016) found that undergraduate students who worked with faculty mentors in research improved in the areas of analyzing, writing, and presenting research data. one study found that science undergraduates who participated in a research experience had an increased ability to perform skills at the end of the experience compared to the beginning (kardash, 2000). evidence suggests these skills persist through graduate school. graduate students who experienced ugr were better at finding existing literature, developing testable hypotheses, and selecting, analyzing, and presenting data (gilmore et al., 2015). participating in ugr also positively affects students' perceived skill competence. for example, students who participated in the nsf-funded research experiences for undergraduates (reu) program (nsf, 2017) reported higher gains in specific research-based skills (follmer et al., 2018). perceived increases in skills and knowledge from research engagement occur even with beginner researchers (bhattacharyya et al., 2018). research competence was not only reported by the students, but also noted by their faculty mentors (kardash, 2000). likewise, students who participated in ugr at a hispanic-majority institution were more likely to gain “knowledge and skills, institutional support, overall satisfaction, grade point average, and student-faculty interaction” (collins et al., 2017, p. 583). undergraduate research engagement has also led to increased institutional metrics for student success, especially for underrepresented populations including persistence (simmons, 2018) and higher gpas (whittinghill et al., 2019). bowman and homes (2018) observed freshmen participation in ugr to be positively related to fourth-year undergraduate gpas. another study using student selfreported research experiences in an empirical design found that students who participated in ugr had higher gpas when compared to those without ugr participation (baker, 2017). furthermore, it appears these experiences positively predict higher college gpas independent of how the research experience occurs (kim & sax, 2009). freshmen and sophomore students, particularly african american students and those with lower gpas who participated in a targeted ugr experience, were more likely to be retained within the institution compared to a control group (nagda et al., 1998). jones et al. (2010) found that hispanic and african american students who engaged in ugr were more likely to persist in their major and obtain their biology degrees, compared to peers without ugr. in addition, students who participated in ugr were more apt to pursue graduate education, engage in additional research activities, enter doctoral stem programs, and showed a higher interest in pursuing a research career (hathaway et al., 2002; pender et al., 2010; wolkow et al., 2019). students at a minority-serving institution were more likely to develop relationships with mentors as well as feel a part of a community through ugr programs, leading them to aspire for ph.d. and graduate programs (carpi et al., 2017). engineering alumni who participated in undergraduate research were four times as likely to pursue doctoral degrees than non-research students (zydney et al., 2002). barlow and villarejo (2004) found that underrepresented minority freshmen who participated in an undergraduate research program for biological sciences were 2.4 times more likely to graduate from the institution, independent of graduating major, and 4.1 times more likely to graduate in biological science. several more recent studies have evaluated the impact of undergraduate research experiences (ure) on academic student success metrics and post-graduation plans specifically at hispanic or minority-serving institutions (schneider et al., 2021; baron et al., 2020; battaglia et al., 2022). battaglia 15 chamely-wiik, ambrosio, baker, ghannes, and soberon journal of the scholarship of teaching and learning, vol. 23, no. 1, april 2023. josotl.indiana.edu et al. (2022) found that engagement in undergraduate research of different research modalities (all cocurricular) at a hispanic-serving institution (hsi) is associated with increased number of successfully completed semester credit hours, higher gpas, increased graduation rates, and higher enrollment in further education. schneider et al. (2021) showed similar trends of students engaged in ugr having demonstrated increased gpas as well as entry into graduate school. schneider et al. (2021) also identified a gap in the literature by stating “we have not looked at this data regarding how many ‘times’ students appeared in our database, but that would warrant further investigation to see if extended involvement impacted gpa (and other factors).” they also identified only a small difference between their institutional and ugr population in terms of years to degree completion, indicating that ugr students have not graduated in fewer semesters than the university undergraduate population, and further suggesting that “it would be interesting to know if other campuses see similar trends” (schneider et al., 2021). this paper aims to address these gaps in the literature. while many of these studies have examined the benefits of participating in ugr, few have explored how the level or intensity of the undergraduate research experiences affect measures of student success. zydney et al. (2002) used the number of semesters involved in research to compare student perceptions of research involvement benefits. their findings suggested that 34 out of 38 students who participated in four or more semesters perceived research involvement as “extremely important” (p.154). baron et al. (2020) evaluated the impact of the length of participation in their emerging scholars program on semester gpa, semester credits earned, persistence and graduation. they found no statistical significance for gpa between those who completed a single versus multiple semesters. they did find that the graduation rate for the single semester participant was significantly lower than the graduation rate for multiple participants, but only for those pursuing an associate degree. thiry et al. (2012) further defined intensity of engagement in undergraduate research based on the length of time students engaged in research by creating two levels: novice and experienced. novice researchers were defined as “students who had completed two or less semesters of undergraduate research,” and experienced researchers were defined as “students who had completed more than two semesters plus one summer of undergraduate research” (thiry et al., 2012, p. 264). findings indicated differences in the reported experiences between novice and experienced researchers and identified experienced researchers to be more likely to report perceived improvement in data collection techniques, data analysis engagement, and advanced scientific thinking skills. the term novice researcher has also been used to examine gains in skill and knowledge among undergraduates with little to no prior research experience who began research early in their academic careers (bhattarcharyya et al., 2018). although these studies demonstrate multiple positive effects ugr has on student success, these effects have often been measured through self-reports and indirect measures, and impact has been determined by correlations frequently without the benefit of an objective comparison group (haeger et al., 2020). additionally, most of these studies have focused solely on stem students (haeger et al., 2020). battaglia et al., (2022) looked at the effect on the type of ugr experience on different metrics, however it was mostly stem focused and did not factor in semesters involved. furthermore, the few studies measuring ugr impact empirically have not factored in the possible effect of research experience intensity. this study examines the effect of research experience intensity (i.e., the number of semesters involved in research) between stem and non-stem students on several student success metrics. research experiences consistent with existing literature will be objectively operationalized by participation in well-defined curricular and co-curricular initiatives. 16 chamely-wiik, ambrosio, baker, ghannes, and soberon journal of the scholarship of teaching and learning, vol. 23, no. 1, april 2023. josotl.indiana.edu institutional background this study was conducted at florida atlantic university, a hispanic-serving, doctoral granting, public, accredited southeastern university, with a classification of high research activity (carnegie classification of institutions of higher education, 2020). this classification identifies the university as a large public university with a high undergraduate enrollment of approximately 24,500. a 2011 analysis of the institution’s research activity revealed that most of the student research conducted was limited to graduate and upper-division undergraduate students, primarily through honors programs. given the extensive literature on the positive impacts of undergraduate research, a university-wide initiative was developed in 2012 to expand this practice to include more undergraduate students. florida atlantic university leveraged its re-accreditation with the southern association of colleges and universities: commission on colleges (sacscoc) to develop and implement a set of initiatives that would expand student engagement in undergraduate research through a quality enhancement plan (qep). the university’s qep, florida atlantic university, focused on improving student learning by expanding a culture of undergraduate research and inquiry across all disciplines at the university, and guided by a centralized office. the institution also included undergraduate research metrics as part of their 2015 strategic plan, and as part of faculty promotion and tenure portfolios in 2018. the goal of the florida atlantic university was to expand undergraduate research and inquiry (uri) at the university and included the following objectives: a) establish an undergraduate curriculum which provides students necessary uri intellectual skills; b) expand student co-curricular uri opportunities; c) increase support and recognition for engaged faculty and students; and d) enrich and strengthen an institutional uri climate (chamely-wiik, 2013). numerous initiatives were developed to meet these goals such as an undergraduate research grants program, an annual research symposium, a research journal, a summer research fellowship, course-based research intensive (ri) and directed independent research (dir) experiences, student peer mentoring and professional development programs, a faculty liaison program, and student uri clubs. data collected through several of these programs were examined for this study. defining undergraduate research for the purposes of this study, cur’s 2015 definition of undergraduate research was adopted and refined to include the following criteria to address the methodological gaps in previous studies and ensure a delineated definition of research: ● student research must be faculty-mentored ● the nature of the inquiry must be original ● students complete the entire cycle of research and inquiry ● the inquiry or research must generate tangible outcomes consistent with the discipline ● the inquiry or research must be communicated to an external audience and/or subject to external or peer review the uri programs that adhered to the above qualifications included the undergraduate research grants program (ugrg), the undergraduate research symposium (urs), the undergraduate research journal (urj), the summer undergraduate research fellowship (surf) program, the national science foundation (nsf) learning environment and academic research network (learn) program, and research intensive (ri) and directed independent research (dir) courses. an undergraduate research committee which includes one faculty member from each college, has 17 chamely-wiik, ambrosio, baker, ghannes, and soberon journal of the scholarship of teaching and learning, vol. 23, no. 1, april 2023. josotl.indiana.edu guided the establishment and implementation of all above initiatives to ensure that stem and nonstem experiences across the university adhere to the above delineated definition as appropriate to their discipline. below we provide additional details about the different programs for context of the research nature of each of the programs. undergraduate research grants program (ugrg). introduced the fall semester of 2012, the ugrg program supports undergraduate student projects where the research or creative activity conducted by the student generally takes diverse forms of directed research, such as independent study, work on an honors thesis or as a part of a larger ongoing study under the direct mentorship of a faculty member. eligible students must be undergraduates in good academic standing. to be included in this study, undergraduates must have been awarded a grant and received funding. undergraduate research symposium (urs). the urs, which was first hosted in spring 2012, showcases student undergraduate research and scholarship progress through oral, performing arts, poster, or visual arts presentations. submitted projects must be conducted as an undergraduate, and students whose abstracts were accepted for a presentation were included in this study. undergraduate research journal (urj). established in spring 2012, the urj is an interdisciplinary, double-blind peer-reviewed journal that is published annually, online and in print, and showcases high quality undergraduate research in all fields. only research projects conducted and written by an undergraduate are eligible for submission. students whose manuscripts were published were included in this study. summer undergraduate research fellowship (surf). surf projects, which were first awarded the summer of 2015, are ten-week, intensive summer immersion research experiences for faculty-undergraduate student teams. eligible students must be undergraduates in good academic standing and retain undergraduate status through the following fall semester. typically, surf was sought by faculty mentors for experienced undergraduates. to be included in this study, undergraduates must have been awarded a fellowship and received funding. nsf learn program. the nsf learn program, which had its first group of participants in the fall of 2016, seeks to increase retention and success of underrepresented students in science, technology, engineering, and mathematics (stem) disciplines, by engaging students in undergraduate research. to be eligible to participate in the year-long program, undergraduates must be either transfer students or first time in college (ftic) freshmen with a declared stem major. participants that completed the year-long learn program are included in the study. research intensive (ri). research intensive (ri) course designation was established in the spring of 2015 to expand the practice of curricular-based research opportunities for undergraduate students and to allow documentation of student engagement in research and scholarly activity on their transcripts. all ri courses include evidence of students completing the entire cycle of research and inquiry including learning gains in content knowledge, formulate questions, plan of action, critical thinking, ethics and communication (chamely-wiik, et al., 2014). courses were submitted and reviewed for ri designation by a university committee using a common rubric to ensure assignments satisfied criteria (author, 2017). undergraduate students enrolled in an ri course were included in the study. directed independent research (dir). directed independent research (dir) courses were established in the spring of 2015 to identify and track through the curriculum, undergraduates engaged in research or creative inquiry under the mentorship of a faculty member. undergraduate students enrolled in a dir were included in the study. 18 chamely-wiik, ambrosio, baker, ghannes, and soberon journal of the scholarship of teaching and learning, vol. 23, no. 1, april 2023. josotl.indiana.edu research questions a quasi-experimental, matched-subject design was employed to study differences between varying levels of research experience (e.g., novice and experienced compared to a control group) on various measures of student success. the following research guided the analysis: rq 1: are there differences in graduation gpa between students with varying levels of undergraduate research experience (experienced, novice, control)? rq 2: are there differences in time to graduation between students with varying levels of research experience (experienced, novice, control)? rq 3: are there differences in post-graduation activities between students with varying levels of research experience (experienced, novice, control)? method participants participants (n=344) were selected from all undergraduate students involved in undergraduate research (n=2,959) enrolled from spring 2012 through spring 2019 who graduated with a bachelor’s degree and did not transfer credits earned prior to this timeframe. students who had transfer credits prior to spring 2012 were eliminated from the study to reduce generational and historical effects. this period was chosen to correspond with the inception of the uri programs to allow 4-6-year graduation rates to be realized. students involved in the selected research programs within the timespan were divided into two groups representing different levels of uri research experience: novice or experienced. novice students engaged in a uri program for one or two semesters prior to graduation. experienced students engaged for at least three different semesters prior to graduation. with these specifications, the final list of program students (n= 2,959) was submitted to the university’s institutional research (ir) office to merge with the subset of participants identified above. after removing personal identification, ir securely returned a complete dataset which included all dependent and independent variables used in this study. students not previously coded as “novice” or “experienced” were identified as part of the control group, which represented those with no uri experiences. design a matched-subject design was used to equate these groups based on three factors: first semester gpa, stem/non-stem-related major, and number of transfer credits at the time of sampling. first semester gpa was used to equalize students on academic ability, serve as a pre-measure control to graduating gpa, and reduce differences between subjects on potential apriori research interest. transfer credits were not used in gpa scores. stem classification was determined by categorizing a student’s graduating major and used to ensure equal numbers in research-heavy disciplines that may place different emphases on the value, utility or definition of a research experience. the number of transfer credits was used to control the degree of exposure to this institution, and to equate groups on their distance to graduation, since time to graduate is a dependent variable in this study. the experienced group was the smallest of the three groups, so every viable case became a subject in the study (n=86). each novice subject was matched to every case in the experienced group. twice as many subjects were matched for the control group to maximize design sensitivity. a random selection procedure was used when multiple novice or control subjects were a viable match to a case. 19 chamely-wiik, ambrosio, baker, ghannes, and soberon journal of the scholarship of teaching and learning, vol. 23, no. 1, april 2023. josotl.indiana.edu the final selected group demographics and variable statistics are presented in table 1. the number of students in the experienced and novice groups who participated in each research initiative is presented in table 2 and these results can be used to gauge differences between the groups in terms of the relative intensity levels. table 1. frequencies and averages of demographic characteristics between experienced, novice and control groups. experienced novice control totals n 86 86 172 344 # stem 55 55 110 220 # non-stem 31 31 62 124 female 62 44 114 220 male 24 42 58 124 native american/alaskan 1 0 0 1 asian 6 7 10 23 black/african american 9 8 23 40 hispanic or latino 25 24 48 97 white 39 40 82 161 multiple 0 3 3 6 1st sem avg. gpa (sd) 3.60 (0.42) 3.58 (0.45) 3.59 (0.43) -- *avg transfer credits (sd) 22.25 (29.7) 22.30 (29.7) 22.24 (29.6) -- *median transfer credits were 3.0 for each group table 2. number of participants by initiative and group. dir florida atlantic universi tyrj learn ri surf urs ugr novice 19 2 2 22 0 46 30 experienced 29 6 6 22 24 78 77 legend: novice refers to students engaged in an undergraduate research program/course for one or two semesters prior to graduation and experienced refers to students engaged for at least three different semesters prior to graduation. 20 chamely-wiik, ambrosio, baker, ghannes, and soberon journal of the scholarship of teaching and learning, vol. 23, no. 1, april 2023. josotl.indiana.edu dir refers to directed independent research; faurj refers to florida atlantic undergraduate research journal; learn refers to the nsf learn program; ri refers to research intensive courses; surf refers to the summer undergraduate research fellowship program; urs refers to the annual undergraduate research and inquiry symposium; and ugr refers to the undergraduate research grants program. measures this study used three dependent variables in group comparisons: the grade point average of students at the time of graduation (degree gpa); the number of years to earn a bachelor’s degree (time to graduate); and educational experience after graduation (post-grad experience). postgraduate experience was obtained through national clearinghouse data that was matched with student records. for postgraduate experience, students were classified into three experience groups: graduate/professional, 2nd-undergraduate/certificate, and no-post graduate experience. results research question 1 the first hypothesis sought to test differences between experienced, novice and control groups on their gpa at graduation. a one-way anova was used in this comparison. a levene’s test validated homogeneity (p = 0.231). results indicated a significant difference between groups (f (2, 341) = 8.41; p < 0.01). a bonferroni post-hoc analysis showed that the experienced group had a significantly higher degree gpa than the control group (p < 0.01), but not the novice group (p > 0.05). there was no significant difference between the novice and control groups (p > 0.05). group means and standard deviations are presented in figure 1. research question 2 the second hypothesis examined the difference between experienced, novice and control groups on time to graduate. a one-way anova was employed. a levene’s test indicated no violations in the homogeneity of variance (p = 0.47). results showed a significant difference between groups (f (2, 341) = 3.31; p < 0.05). a bonferroni post-hoc analysis indicated a significantly lower time to graduate between the experienced and control groups (p < 0.05), but not between the experienced and novice groups (p > 0.05). there is a significant difference between the novice and control groups (p > 0.05). figure 1 presents the group means and standard deviations. 21 chamely-wiik, ambrosio, baker, ghannes, and soberon journal of the scholarship of teaching and learning, vol. 23, no. 1, april 2023. josotl.indiana.edu figure 1. average graduating gpa and years to graduate by group. research question 3 a 3 x 3 chi-square test for independence was used to determine the relation between the level of undergraduate research involvement (experienced, novice, control) and type of post-graduate experiences (graduate/professional, 2nd-undergraduate/certificate, or no post-graduate experience). results showed a statistically significant difference between groups on post-graduate experiences (x2 (4, n=344) = 14.15, p < 0.05). a follow-up cramer’s v analysis identified a small overall effect size (φc = 0.14). post-hoc analysis of adjusted residuals with a bonferroni correction indicated the control group had a significantly higher observed versus expected frequency for “no post-graduate experience,” and a significantly lower observed versus expected frequency for “graduate/professional.” in contrast, the experienced group had a significantly lower “no postgraduate experience” count and a significantly higher “graduate/professional” count than expected. the novice group displayed no significant difference in observed and expected values for each postgraduate condition. the observed versus expected frequency for “2nd-bachelors/certificate” was comparable for all three groups. table 3 presents descriptive results of this analysis. table 3. frequency of post-graduate experience by group. observed frequency observed percent expected frequency control no post-graduate experience 108 62.8 93.5 2nd -undergraduate/certificate 24 14.0 24 graduate/professional 40 23.3 54.5 total 172 100.0 172 novice no post-graduate experience 44 51.2 46.8 2nd -undergraduate/certificate 11 12.8 12 graduate/professional 31 36.0 27.3 total 86 100.0 86 experienced no post-graduate experience 35 40.7 46.8 2nd-undergraduate/certificate 13 15.1 12 graduate/professional 38 44.2 27.3 total 86 100.0 86 22 chamely-wiik, ambrosio, baker, ghannes, and soberon journal of the scholarship of teaching and learning, vol. 23, no. 1, april 2023. josotl.indiana.edu discussion a quasi-experimental, matched-subject design was employed to examine differences between students with differing levels of experience with uri initiatives (experienced, novice, and control groups) on graduating gpa, time to graduation, and post-graduate activities. first-semester gpa, stem major distribution, and number of transfer credits were controlled to eliminate the possibility of confounding factors such as the perception that high-achieving students pursue undergraduate research more, or that uri is pursued more by students in stem. research question 1: gpa results indicated a significant difference between the experienced group compared to both novice and control groups when evaluating their graduating gpa. moreover, the difference between novice uri students and those with no uri experience was not significant, demonstrating that the intensity or duration of research experience affects student success on this metric. this builds on and provides empirical support for previous research on the positive relationship between gpa and students who participate in research (whittinghill et al., 2019, bowman & holmes, 2018; collins et al., 2017; russell et al., 2007, sell et al., 2018). since high school gpa varies from institution to institution (weighted, unweighted, etc.), we chose to utilize first-semester gpa within the exact-matched design to eliminate the possibility of confounding factors, further demonstrating a direct role of undergraduate research involvement on gains in gpa at graduation. experienced researchers may be better equipped to transfer research skills to other courses, resulting in improvements in gpa and other measures of student success. alternatively, it is possible that students who participate as novices in research, may be lacking in certain academic skills. since our study did not evaluate these factors or others such as motivation, further studies would need to be conducted to better understand these differences. an additional analysis was conducted to examine the nature of the relationship between gpa at graduation and the pattern of research involvement in total uri terms. the scatterplot results in figure 2 reinforce the idea of promoting student participation in multiple semesters of uri for a more significant impact. student graduation gpas have higher variances among students with a single uri experience while students engaging in three or four terms have lower graduation gpa variances. additional experience beyond four terms appears to have less effect on graduation gpa. 23 chamely-wiik, ambrosio, baker, ghannes, and soberon journal of the scholarship of teaching and learning, vol. 23, no. 1, april 2023. josotl.indiana.edu figure 2. scatterplot of relationship between total uri terms and gpa at graduation. while we controlled for stem/non-stem students in our sample through a matched design, a comparison between these subgroups was warranted to underscore the importance of recruitment of a diverse population of participants. a one-way anova was conducted to compare degree gpa between stem and non-stem students. a levene’s test validated homogeneity (p = 0.771). results indicated no significant difference between the groups (f (1, 342) = 1.04; p > 0.05). these findings are consistent with previous research in this area (collins et al., 2017; nagda et al., 1998), suggesting the lift in gpa is beneficial for any student pursuing research opportunities independent of discipline. this has implications for centralized offices to facilitate scaffolded and sustained involvement through/across multiple programs for up to at least three terms and expand their offerings to include stem and non-stem research experiences. to facilitate student engagement in multi-year research opportunities, colleges and universities should consider focusing their efforts and resources on early engagement in undergraduate research (ishiyama, 2002). additionally, offices should consider showcasing opportunities such as an annual symposium or a research journal, to support sustained involvement and encourage students to communicate the findings of their inquiry. departments could also scaffold research skills by offering multiple course-based undergraduate research experiences (cures) or other research intensive (ri) course experiences within their programs of study, leading to a capstone research experience. finally, participation in symposia and publication opportunities can be included within cures or considered a supplemental requirement for receiving funding from a centralized office. research question 2: time to graduation the second hypothesis of this study examined the impact of uri involvement on student time to graduation. results indicate a significant difference between the control group and both experienced and novice groups. even novice uri experiences (two terms or less) had a positive impact on a participant’s time to graduation. these findings support previous research in this area that found ugr 24 chamely-wiik, ambrosio, baker, ghannes, and soberon journal of the scholarship of teaching and learning, vol. 23, no. 1, april 2023. josotl.indiana.edu participation to be associated with higher retention rates (nagda et al., 2002), and lower time to graduate when comparing novice and experienced groups to a control group (thiry et al., 2012). from a student perspective, a lower time to graduation while engaging in undergraduate research could be financially beneficial, resulting in a quicker transition to the workforce or graduate school. since colleges and universities are being held to more metric-driven standards related to student success efforts such as time to graduation, engagement in high-impact practices like ugr could support student success. finally, these findings quell the perception that engagement in uri activities may slow a student down academically and supports the idea that even limited uri experiences support student success. the results demonstrate that even novice researchers who engage in two semesters or less of undergraduate research are more likely to graduate in less time than non-research active students. this could be beneficial for institutions with large transfer student populations. institutions could encourage these students to get involved, despite the shortened time at the university. additionally, for smaller institutions or those with limited resources, these findings suggest offering some research experience is still valuable and does positively impact a student’s time to graduation. since even limited engagement in undergraduate research demonstrates positive outcomes, this also bodes well for supporting small programs with limited opportunities, funding, and time. research question 3: post-graduation activities the third research question explored the relation between intensity of undergraduate research involvement and post-graduate experiences. results indicated that experienced research students were more likely to continue to post-graduate experiences than students who did not participate in uri research initiatives. these findings are consistent with previous studies, indicating that research involvement is highly and significantly correlated with higher degree aspirations (baker & dedonno, 2020; bauer & bennett, 2003; kim & sax, 2009; zydney et al., 2002). while these correlations exist, there is still ongoing research related to causal factors. for example, according to bowman and holmes (2018), students who engage in research activities later in their college years may have already made the decision to attend graduate school, and as a result, have decided to participate in these activities to have a better chance of attaining acceptance to a graduate program. this later involvement may lead to a confirmation of a previously decided upon career decision (seymour et al., 2004). although it is difficult to know which came first, research involvement or interest in graduate school, participation in undergraduate research activities is positively related to graduate school attendance. while the current sample was matched on stem/non-stem participation, a supplemental analysis was conducted to determine the relationship between major and post-graduation activities of the undergraduate researchers. an additional chi-square analysis found a significant relationship between stem/non-stem students and post-graduate experiences (x2 (2, n= 344) = 7.82, p< 0.05). post-hoc residual analysis with bonferroni corrections indicated that stem students were more likely to seek undergraduate, graduate, and professional post-graduation experiences than non-stem students (p< 0.05), although cramer’s v indicated the effect size was weak (φc = 0.15). in this analysis, graduate and professional pursuits were combined. many students pursuing stem majors often enter this field with intentions of going to professional programs such as medical or dental school. perhaps students pursuing stem disciplines come in with those career aspirations which could influence their interest in engagement in research. since undergraduates involved in research tend to pursue graduate degrees at a higher rate (and even more so in stem), centralized offices should consider offering graduate education professional development for undergraduates to support efforts of undergraduates who pursue external research experiences for undergraduates (reus) and introduce them to fellowship 25 chamely-wiik, ambrosio, baker, ghannes, and soberon journal of the scholarship of teaching and learning, vol. 23, no. 1, april 2023. josotl.indiana.edu opportunities like the national science foundation graduate research fellowship program (nsf grfp) to further optimize their acceptance into graduate programs. overall, these findings could be used to argue that students engaging in ugr gain cross-competency skills which positively impact their performance in other courses, leading to a higher gpa. however, more research would need to be done to verify this assumption. for example, a longitudinal study where direct measures of student learning outcomes are assessed throughout their educational career could yield which skills are most impactful to student success. limitations and delimitations although a rigorous matching design eliminated many confounding factors in the analysis, there are several limitations of the study that should be noted for subsequent research and when generalizing results. these include potential control group contamination, the varied student experiences within the uri programs, and student motivation or career aspirations. the degree of control group contamination was not known since the degree and type of research experiences or other high impact practices these students had in their other coursework or co-curricular engagements could not be determined. students in the novice and experienced groups had a known degree of research within ri courses and dir by virtue of a formalized process of course designation. however, there may have been students who completed volunteer research experiences or research internships which would not have been recorded. this could become a factor in reducing significance between the groups, especially when sampling at a research institution. future studies should attempt to clarify the purity of their control groups in the selection process via student surveys employed on a smaller control group. students did not engage in equal numbers across the uri initiatives. the symposium and ugr grant experiences accounted for a large percentage of the participation (see table 2). part of this was due to these programs being offered in earlier years when compared to other programs which were launched later in the 2012-2019 sampling period. additionally, research activities such as the institutional symposia, attract larger numbers of students due to the nature of the event. typically, surf was sought by faculty mentors for experienced undergraduates, which may account for their superior performance over novice and control students. while all these initiatives met the criteria for an intensive research experience, the effect of participation duration and type on the results is unknown. the operational definition of research intensity may need to be further refined. future studies could delve deeper to analyze the types of uri experiences (i.e., academic versus summer experiences, grant participation versus publishing in a journal) and their specific impact on student success. the third hypothesis, which examined post-graduate experiences, found significant differences between experienced and control subjects. however, the causal connection is not clear. it is not certain if students are involved because of their prior intentions of attending graduate/professional post baccalaureate programs, or if the research experiences promoted or motivated their choice to pursue these aspirations. future studies should consider measuring motivational choice within a pre-post experimental design; with this design, the effect of students’ initial intent could be isolated, and institutions can apply this information into improving their programs by targeting students not internally motivated to do research. additionally, and as discussed previously, stem students may have higher post-graduate expectations. despite controlling for stem/non-stem majors in the study, future studies should consider separating graduate and professional post-graduate activities to determine if differences exist. stem students’ aspirations to attend professional programs, such as medical school, may affect the percentage of students going to graduate school. 26 chamely-wiik, ambrosio, baker, ghannes, and soberon journal of the scholarship of teaching and learning, vol. 23, no. 1, april 2023. josotl.indiana.edu conclusions there continues to be growing interest and research on engagement of students in undergraduate research, yet few have studied the impact of “time” or “intensity” on student engagement in research. the primary purpose of this study was to evaluate the impact of the intensity of student engagement in undergraduate research on student success metrics. the metrics evaluated included student gpa at graduation, time to graduation and post-graduation activities. research findings demonstrate that experienced undergraduate research students (i.e., those who participated in more than two semesters of undergraduate research activities) had significantly higher gpas at graduation when compared to novice and control group students who were matched on first term gpa, stem/non-stem related major, and number of transfer credits at the time of sampling. no difference was identified between stem and non-stem groups. additionally, our findings indicate that experienced researchers are significantly more likely to progress to graduate school than either novice research or non-research students. experienced stem research students are also significantly more likely to progress to graduate school compared to non-stem counterparts. finally, both experienced and novice researchers demonstrated a significantly lower time to graduation when compared to the control group. acknowledgements the authors would like to acknowledge ben silva from florida atlantic university’s institutional effectiveness and analysis office, faculty mentors, and the staff from florida atlantic university’s office of undergraduate research and inquiry. references baker, t. n. (2017). the impact of undergraduate research participation on research self-efficacy. (publication no. 10642999). [doctoral dissertation. florida atlantic university]. pqdt openproquest. baker, t. n. & dedonno, m. a. (2020). the influence of research participation, field of study, and gender on research self-efficacy. the international journal of learning in higher education, 27(2), 11-26. doi:10.18848/2327-7955/cgp/v27i02/11-26. barlow, a. e. l. & villarejo, m. (2004). making a difference for minorities: evaluation of an educational enrichment program. journal of research in science teaching, 41(9), 861-881. doi: https://doi.org/10.1002/tea.20029. baron, s. i., brown, p., cumming, t., & mengeling, m. (2020). the impact of undergraduate research and student characteristics on student success metrics at an urban, minority serving, commuter, public institution. journal of the scholarship of teaching and learning, 20(1), 85-104. doi: 10.14434/josotl.v20i1.25423. battaglia, s. j., echegoyen, l. e., & diaz-martinez, l. a. (2022). institution-wide analysis of academic outcomes associated with participation in ugr: comparison of different research modalities at a hispanic-serving institution. scholarship and practice of undergraduate research, 5(3), 8-24. bauer, k. & bennett, j. (2003). alumni perceptions used to assess undergraduate research experience. journal of higher education, 72(2), 210-230. https://doi.org/10.1080/00221546.2003.11777197. beckman, m. & hensel, n. (2009). making explicit the implicit. council on undergraduate research quarterly, 29(4), 40-44. bhattacharyya, p., chan, c. w. m., & waraczynski, m. (2018). how novice researchers see 27 chamely-wiik, ambrosio, baker, ghannes, and soberon journal of the scholarship of teaching and learning, vol. 23, no. 1, april 2023. josotl.indiana.edu themselves grow. international journal for the scholarship of teaching & learning, 12(2). doi: https://doi.org/10.20429/ijsotl.2018.120203. bowman, n. a. & holmes, j. m. (2018). getting off to a good start? first-year undergraduate research experiences and student outcomes. higher education, 76, 17-33. doi: http://dx.doi.org/10.1007/s10734-017-0191-4. brown, d. a., wright, n. b., gonzales, s. t., weimer, n. e., & soto, j. g. (2020). an undergraduate research approach that increased student success at a hispanic-serving institution (hsi): the sure program at texas state university. scholarship and practice of undergraduate research, 4(1), 52-62. carnegie classification of institutions of higher education (n.d.). florida atlantic university. retrieved july 28, 2020, from http://carnegieclassifications.iu.edu/. carpi, a., ronan, d., falconer, h., & lents, n. (2017). cultivating minority scientists: undergraduate research increases self-efficacy and career ambitions for underrepresented students in stem. journal of research in science teaching, 54(2), 169-194. doi: https://doi.org/10.1002/tea.21341. chamely-wiik, d. (2013). distinction through discovery: quality enhancement plan. florida atlantic university. retrieved from https://www.fau.edu/ouri/documents/qualityenhancement-plan-1-11-13.pdf. chamely-wiik, d., holman, m., kirsch, p., meeroff, d. & peluso, j. (2014). scaffolding the development of student research skills for capstone experiences: a multi-disciplinary approach. council on undergraduate research quarterly, 34(4), 18-25 chamely-wiik d. & usher, b. (2017). research-intensive course designation. council on undergraduate research quarterly, 37(4), 13-15. collins, t. w., grineski, s. e., shenberger, j., morales, x., morera, o. f., & echegoyen, l. e. (2017). undergraduate research participation is associated with improved student outcomes at a hispanic-serving institution. journal of college student development, 58(4), 583-600. doi: 10.1353/csd.2017.0044. council on undergraduate research (cur). n.d.a “mission and vision.” accessed april 21, 2021. https://www.cur.org/who/organization/mission_and_vision/. craney, c., mckay, t., mazzeo, a., morris, j., prigodich, c., & de groot r. (2011). cross discipline perceptions of the undergraduate research experience. the journal of higher education 82(1), 92–113. doi: 10.1353/jhe.2011.0000. follmer, d. j., zappe, s., gomez, e., & kumar, m. (2018). student outcomes from undergraduate research programs: comparing models of research experiences for undergraduates (reus). scholarship and practice of undergraduate research, 1(1), 20-27. frederick, a., grineski, s., collins, t. w., daniels, h.a., & morales, d.x. (2021). the emerging stem paths and science identities of hispanic/latinx college students: examining the impact of multiple undergraduate research experiences. ed. stanley m. lo. cbe—life sciences education 20(2), ar18. doi: 10.1187/cbe.20-08-0191. gilmore, j., vieyra, m., timmerman, b., feldon, d., & maher, m. (2015). the relationship between undergraduate research participation and subsequent research performance of early career stem graduate students. the journal of higher education, 86(6), 834-863. doi: https://doi.org/10.1080/00221546.2015.11777386. haeger, h., banks, j. e., smith, c., & armstrong-land, m. (2020). what we know and what we need to know about undergraduate research. scholarship and practice of undergraduate research, 3(4), 62-69. hathaway, r., nagda, b., & gregerman, s. (2002). the relationship of undergraduate research participation to graduate and professional education pursuit: an empirical study. 28 https://www.fau.edu/ouri/documents/quality-enhancement-plan-1-11-13.pdf https://www.fau.edu/ouri/documents/quality-enhancement-plan-1-11-13.pdf chamely-wiik, ambrosio, baker, ghannes, and soberon journal of the scholarship of teaching and learning, vol. 23, no. 1, april 2023. josotl.indiana.edu journal of college student development, 43(5), 614-631. jones, m. t., barlow, a. e. l., & villarejo, m. (2010). importance of undergraduate research for minority persistence and achievement in biology. the journal of higher education, 81(1), 82-115. doi: 10.1353/jhe.0.0082. kardash, c. (2000). evaluation of an undergraduate research experience: perceptions of undergraduate interns and their faculty mentors. journal of educational psychology, 92(1), 191-201.doi: https://doi.org/10.1037/0022-0663.92.1.191. kim, y. k., & sax, l. j. (2009). student-faculty interaction in research universities: differences by student gender, race, social class, and first-generation status. research higher education, 50, 437-459. doi: 10.1007/s11162-009-9127-x. kuh, g. d. (2008). high-impact educational practices: what they are, who has access to them, and why they matter. washington dc: association of american colleges and universities. mcdevitt, a. l., patel, m. v., rose, b., & ellison, a. m. (2016). insights into student gains from undergraduate research using preand post-assessments. bioscience, 66(12), 1070-1078.doi: https://doi.org/10.1093/biosci/biw141. monarrez, a., morales, d., echegoyen, l. e., seira, d., & wagler, a. e. (2020). the moderating effect of faculty mentorship on undergraduate students' summer research outcomes. cbe life sciences education, 19(4), ar56. https://doi.org/10.1187/cbe.20-04-0081 nagda, b., gregerman, s., jonides, j., von hippel, w., & lerner, s. (1998). undergraduate student-faculty research partnerships affect student retention. the review of higher education, 22(1), 55-72. national science foundation (2017). research experiences for undergraduates (reu) sites and supplements: program solicitation. retrieved from https://www.nsf.gov/pubs/2005/nsf05592/nsf05592.htm pender, m., marcotte, d. e., domingo, m. r., & maton, k. i. (2010). the stem pipeline: the role of summer research experience in minority students’ ph.d. aspirations. educational policy analysis archives, 18(30), 1-36. russell, s., hancock, m., & mccullough, j. (2007). benefits of undergraduate research. science, 316, 548-549. doi: 10.1126/science.1140384. schneider, k., kuperman, a., watt, a., barulich, d., & campbell, t. (2021). tracking and assessing undergraduate research campus-wide: demographics, academic success, and postgraduation plans. journal of the scholarship of teaching and learning, 21(1), 107-119. sell, a. j., naginey, a., & stanton, c.a. (2018). the impact of undergraduate research on academic success. scholarship and practice of undergraduate research, 1(3), 19-29. seymour, e., hunter, a. b., laursen, s. l., & deantoni, t. (2004). establishing the benefits of research experiences for undergraduates in the sciences: first findings from a three‐year study. science education, 88(4), 493-534. doi: https://doi.org/10.1002/sce.10131. simmons, kiyoko n. (2018). effect of undergraduate research programs on retention of hispanic students. https://digitalrepository.unm.edu/oils_etds/50 thiry, h., weston, t. j., laursen, s. l., & hunter, a. (2012). the benefits of multi-year research experiences: differences in novice and experienced students’ reported gains from undergraduate research. cbe-life sciences education, 11, 260-272. doi: 10.1187/cbe.11-11-0098. whittinghill, j. c., slovacek, s. p., flenoury, l. p., & miu, v. (2019). a 10-year study on the efficacy of biomedical research support programs at a public university. scholarship and practice of undergraduate research, 3(1), 30-38. 29 chamely-wiik, ambrosio, baker, ghannes, and soberon journal of the scholarship of teaching and learning, vol. 23, no. 1, april 2023. josotl.indiana.edu wolkow, t. d., jenkins, j., durrenberger, l., swanson-hoyle, k., & hines, l. m. (2019). one early course-based undergraduate research experience produces sustainable knowledge gains, but only transient perception gains. journal of microbiology & biology education, 20(2). doi: 10.1128/jmbe.v20i2.1679. zydney, a., bennett, j., shahid, a., & bauer, k. (2002). impact of undergraduate research experience in engineering, journal of engineering education, 91(2), 151-157. doi: 10.1002/j.2168-9830.2002.tb00687.x 30 5031-31896-1-ce-2 journal of the scholarship of teaching and learning, vol. 14, no. 3, august 2014, pp. 132 145. doi: 10.14434/josotl.v14i3.5031 college textbook reading assignments and class time activity lola aagaard1, timothy w. conner ii2, and ronald l. skidmore3 abstract: a convenient cluster sample of 105 undergraduate students at a regional university in the midsouth completed a survey regarding their use of college textbooks, what strategies might increase the likelihood of their reading textbook assignments, and their preference for how class time was used. descriptive analysis was conducted on the results and chi-square was run on 25 selected comparisons, with a bonferroni correction of the resulting alphas. about half the students reported that they do read the assigned textbook readings. freshmen were significantly more likely to report that outside reading should not be required of students prior to coming to class, and less likely to report having used or known about e-textbooks. strategies reported to most likely prompt reading the textbook included in-class quizzes over text material, assigning graded study-guides to complete while reading; testing over material found in the textbook but not covered in class; and assigning shorter reading assignments. preferences for use of class time varied by experience in college, but the majority of students preferred group discussion and application of material to real life rather than just lecture over the textbook content. keywords: higher education; college teaching; textbook use; instructional strategies introduction regardless of the fact that professors often assign textbook readings to students as part of their course requirements, reports have clearly articulated that many students (perhaps most of them), do not actually complete the course readings as instructed (lei, barlett, gorney, & herschbach, 2010; pecorari, shaw, irvine, malstrom, & mezek, 2012; sikorski et al., 2002). this lack of student engagement in reading for the courses not only includes assigned readings for traditional class meetings, but also in terms of preparing for course examinations (aagaard & skidmore, 2004; clump, bauer, & bradley, 2004; sikorski et al., 2002). the implications of these findings would seem to suggest that most of what students glean from course study comes directly from the learning experiences conducted within the classroom, with the in-class presentation of material serving as the primary means by which students gain understanding of important course concepts. this model of engagement with courses changes the course pedagogical arrangement, as students who might otherwise benefit from discussion and application of course concepts must instead be introduced to concepts and terminology during class. in other words, students’ lack of engagement in independent reading of class assignments compels professors to make a different set of lecture plans when thinking about what to do with students during class. 1 associate professor, morehead state university, foundational and graduate studies in education, l.aagaard@moreheadstate.edu 2 assistant professor, morehead state university, foundational and graduate studies in education, t.conner@moreheadstate.edu 3 professor, morehead state university, foundational and graduate studies in education, r.skidmore@moreheadstate.edu aagaard, l., conner ii, t.w., & skidmore, r.l. journal of the scholarship of teaching and learning, vol. 14, no. 3, august 2014. josotl.indiana.edu 133 as lack of student reading prior to class time necessarily limits the depth to which professors can engage learners with important course concepts, studies of methods by which to compel students to read the course text and reading materials have been conducted. suggested strategies that have emerged from such investigations have included quizzes over course materials (johnson & kiviniemi, 2009; ruscio, 2001; ryan, 2006), worksheets with which students study (aagaard & skidmore, 2009; ryan, 2006), chunking the reading tasks into smaller segments, and using the textbook during class instructional activities rather than only asking students to read the materials outside of class (aagaard & skidmore, 2009). the efficacy of each of these strategies will need to be properly explored should instructors in the academy wish to promote independent student engagement in course readings outside of class. although we seem to understand the basic nature of college students and the use (or lack thereof) of course texts and other readings outside of the classroom, related questions left somewhat unaddressed deal not only with what professors might do to compel student reading outside of class, but also with what students might prefer professors do with the time they share together learning within the classroom context. that is to say, some great attention has been given to the exploration of student use of reading materials, but has there been sufficient support to understand what professors do with instruction to promote learning from the perspective of students? after all, education should be a dynamic process that involves engagement not only from the perspective of the educators, but also from the students, as is the case with methods such as inquiry learning (oliver, 2007). the question that remains, however, is not only one of what methods of instruction might be effective when instituted by instructors, but also one of what kind of research has been conducted to ascertain students’ preferences with instructional time during class activities at the college level? as stated above, students have expressed a desire for professors to use a variety of inclass activities to promote their independent reading of course materials, including testing over the textbook readings, shortening the reading assignments, providing advanced organizers to guide note-taking while they read (and offering credit for them), and using the textbooks in class (aagaard & skidmore, 2009). it would seem, as suggested by aagaard and skidmore, (2009), and others (berry, cook, hill, & stevens, 2011; johnson & kiviniemi, 2009; oliver, 2007), that undergraduate students are quite dependent on their professors in terms of promoting the independent use of course readings outside of class, -if the professors require it, give credit for doing it, give specific guidance and direction as to what is relevant and essential, and provide scaffolds to teach students how to do it, then students will probably read on their own outside of class. this sentiment of dependence on the professor as a model is echoed in the work of m. p. ryan (2001) whose study of conceptual models of lecture learning and note taking practices revealed that, in most of the models offered, students’ metaphors for the role of the instructor during lecture ultimately impacted the ways in which they engage in learning behaviors in class, such as the ways in which and purposes for which they engage in note taking behaviors. an implication from this work would be that, if professors should want more appropriate learning behaviors to be demonstrated by students during class, then they (the professors) should model such behaviors during instruction in ways that promote particular note taking practices appropriate for the subject matter being taught (ryan, 2001). as indicated by ryan (2001) “efforts to improve lecture learning may only begin to produce substantial and pervasive benefits for college learners when they focus as much on conceptual change as they do on behavioral change” (p. 307). aagaard, l., conner ii, t.w., & skidmore, r.l. journal of the scholarship of teaching and learning, vol. 14, no. 3, august 2014. josotl.indiana.edu 134 with regard to the use of lecture-based instruction, one factor that should be taken into account when considering students’ preferences for professors’ teaching strategies in class is the degree to which students endorse particular lecture-based styles (kane, 1990; mji & kalashe, 1998; struyven, dochy, & janssens, 2012). for example, when asking students to comment on their preferences concerning the degree to which professors used structured assignments during lecture (such as following lectures with worksheets to complete after each lesson) as opposed to less structured lecture formats, students indicated strongly that they preferred more structure to rather than less during lecture; however, the continual use of only one strategy repeatedly was also reported to be a disadvantage as students perceived a relative loss of autonomy in the classroom (kane, 1990). in other words, although the students reported that implementing within-class assignments was beneficial, what students really wanted was for the professor to alternate modes of delivery based upon the material being presented (kane, 1990). other than traditional lecture-based formats of instruction, another method found in the literature about student preference and student learning outcomes includes problem-based or inquiry learning (oliver, 2007; struyven, dochy, & janssens, 2012). in the case of inquiry learning, oliver (2007) revealed that, when the course materials were structured around solving authentic problems within the discipline rather than merely lecturing over course concepts and content, more students reported enjoying the method of delivery and over 90% perceived positive outcomes as a result of this approach. however, students still expressed problems with the inquiry learning approach, including the relevance of the problems offered, its organization (with particular focus on the directions and instructions), and the difficulty of some of the problems (to which the author attributed the lack of clarity in directions) (oliver, 2007). zhang’s (2008) work focused upon not only the preferred teaching styles of students, but also the relationship between students’ teaching style preferences and their achievement in academic tasks. although this study utilized boys in secondary school in hong kong, it was revealed that the participants’ preferences for teachers that promoted creative thinking, collaboration among students and the use of higher-order reasoning in class over monotone, traditional teacher-centered approaches was reflective of those practices favored by university level students as well. further, this study indicated that, in most areas asked about, students’ preferred teacher practices related positively to their academic achievement while traditional and least-favored practices had an adverse relationship with student achievement (zhang, 2008). as can be seen, there appears to be a whole host of ways in which classes can be structured to support learning for undergraduate students most in need for guided support from their college professors. however, in addition to considering the learning outcomes or instructor preferences alone, it is critically important that researchers seek to understand the degree to which students’ preference for learning contexts plays a role in their ability to achieve academically in the courses being offered (struyven, dochy, & janssens, 2012; zhang, 2008). rather than worry about the amount of content we cover in classes we teach, perhaps professors should consider the notion that teaching less content in ways students endorse might be more advantageous to their students’ overall understanding of the course material (locher, 2004). perhaps professors should take more time to consider the multitude of ways that the content might be approached in class, offering students variety so that they may be better engaged in the various course topics (kane, 1990).   these concerns over student engagement with textbooks and classroom activities led to the research questions for this study: what types of professor behaviors encourage students to aagaard, l., conner ii, t.w., & skidmore, r.l. journal of the scholarship of teaching and learning, vol. 14, no. 3, august 2014. josotl.indiana.edu 135 engage with their textbooks? what are students’ preferences for learning content in the classroom? method participants this study employed a convenient cluster sample of 105 students from eight summer classes at a regional university in the mid-south (see table 1). table 1 frequency of students surveyed in each course course n % soils (agriculture) 13 12.9 geology 9 8.6 philosophy 8 7.6 english 18 17.1 history 13 12.4 foundations (education) 10 9.5 biology 16 15.2 approximately 36% of the students were in natural science courses and 64% were in humanities or education courses. sixty-one percent of respondents were female and nearly 100% were caucasian. they reported 29 different majors, with the highest concentrations being education (17%), biologyrelated (13%), and agriculture-related (10%). the 101 students who reported majors could be classified into two basic categories of science/technology/math (n=43) and humanities/social science/business/education (n=58). the distribution across year in college is shown in table 2. table 2 sample distribution across year in college year n % freshman 26 25 sophomore 19 18 junior 26 25 senior 33 31 graduate 1 1 students were asked to self-report their gpa range. a large majority (63%) claimed a b average, while 30% reported a c average. the remaining 7% were split between a and d average grade point averages. aagaard, l., conner ii, t.w., & skidmore, r.l. journal of the scholarship of teaching and learning, vol. 14, no. 3, august 2014. josotl.indiana.edu 136 instrumentation participants were administered a 25-item researcher-designed survey (see appendix a) that included 11 items regarding use of course textbooks, 11 items about preferences for use of class time, and four demographic items. all items were multiple-choice. textbook items asked whether students read their textbooks when assigned to do so, as well as whether particular strategies by the professor would get students to read their textbooks or not. each class time use preference item was forced choice between two options (for instance, between professor lecture and group activities). procedure researchers requested permission from course instructors to administer the survey to their students in the last 15 minutes of a regularly scheduled class period. analysis frequency analyses were performed on every item for descriptive purposes and cross-tabulations were created of every item with each of three of the demographic variables. the cross-tabs were visually inspected and chi-square analyses were run on 25 selected comparisons, with a bonferroni correction of the resulting alphas. as a result, chi-square tests with p-values of less than or equal to 0.002 were considered significant. as a follow-up analysis, cross-tabulations were created of every item with the two broad categories of student majors. only one of the items showed a frequency difference of more than 10% between the responses of the majors and a chi-square analysis was run on that item. results textbook use a majority of students (52%) reported that they do read the assigned textbook readings, but 48% replied that whether or not they read the assignment depended on other factors. freshmen were significantly more likely to report that reading the textbook before class should not be required, while seniors were more likely to say that whether they should have to read the text depended on other things (see table 3). table 3 chi-square analysis of year in college by “should be required to read text?” freshman sophomore junior senior/grad item response n (row %) n (row %) n (row %) n (row %) yes 5(16) 11(34) 8(25) 8(25) no 13(62) 1(5) 5(24) 2(9) it depends 8(15) 7(13) 13(25) 24(46) note: chi-square = 28.98; df=6; p < 0.0001 aagaard, l., conner ii, t.w., & skidmore, r.l. journal of the scholarship of teaching and learning, vol. 14, no. 3, august 2014. josotl.indiana.edu 137 freshmen were significantly less likely to report having used or known about e-textbooks (see table 4). table 4 chi-square analysis of year in college by “ever used an e-text?” freshman sophomore junior senior/grad item response n (row %) n (row %) n (row %) n (row %) yes 1(4) 8(32) 9(36) 7(28) no 11(21) 6(11) 14(26) 22(42) it depends 14(56) 5(20) 3(12) 3(12) note: chi-square = 26.02; df=6; p < 0.0002 strategies reported to most likely prompt reading the textbook included in-class quizzes over text material, assigning graded study-guides to complete while reading; testing over material found in the textbook but not covered in class; and assigning shorter reading assignments (see table 5). table 5 response to suggested strategies to get students to read textbooks not read might read most likely read strategy n (row %) n (row %) n (row %) in-class quiz 0(0) 28(27) 76(73) online open-book quiz 20(19) 35(34) 49(47) study guide for credit 4(4) 20(19) 81(77) discuss in class 14(14) 39(38) 50(48) test material not discussed 5(5) 14(14) 84(81) shorter reading assignments 1(1) 23(22) 79(77) explain instructional features of textbook 35(34) 39(37) 30(29) the lowest rated strategy was having the use of the textbook’s instructional features (glossary, chapter summaries, etc.) explained to students. females were significantly more likely than males to read the textbook if there were going to be in-class quizzes, as reported in table 6. aagaard, l., conner ii, t.w., & skidmore, r.l. journal of the scholarship of teaching and learning, vol. 14, no. 3, august 2014. josotl.indiana.edu 138 table 6 chi-square analysis of male and female responses to “read text if there is an in-class quiz?” male female item response n (row %) n (row %) might read 19(68) 9(32) would read 22(29) 54(71) note: chi-square = 12.97; df=1; p < 0.0003 preference for use of class time students reported preferring the use of powerpoint lectures to notes on the chalkboard and generally preferred the use of group discussion and application of material to real-life rather than just lecture over textbook content. they were in favor of group presentations in contrast to individual presentations, but there was more division over in-class group activities, with quite a number choosing the other alternatives when they were presented (see tables 7 and 8). table 7 preferences for use of class time (lecture-related) i would prefer the professor: n % lectured only over material that was in the textbook. 19 18 lectured over the textbook, but also some material that was not in the textbook. 84 81 both choices marked 1 1 have powerpoint slides to present basic notes for the lecture. 74 71 use the chalkboard to present basic notes for the lecture. 25 24 both choices marked 5 5 lectured only over material that will be tested. 45 43 lectured over tested material, but also over some material that is interesting but not going to be tested. 59 57 lectured only over the textbook material. 7 7 covered the content; gave examples of how the material applied to real life. 96 92 both choices marked 1 1 just lectured over the textbook material in some way. 27 26 encouraged group discussion of the material. 75 72 both choices marked 2 2 just lectured over the textbook material in some way. 43 41 had students do group activities related to the material. 60 58 both choices marked 1 1 lectured over content in class. 73 69 put the lectures podcasts online; did other things related to content during class. 28 27 both choices marked 4 4 aagaard, l., conner ii, t.w., & skidmore, r.l. journal of the scholarship of teaching and learning, vol. 14, no. 3, august 2014. josotl.indiana.edu 139 table 8 preferences for use of class time (not lecture-related) i would prefer the professor: n % encouraged group discussion of the textbook material. 46 45 had students do in-class group activities related to the textbook material. 54 53 both choices marked 2 2 had students do in-class group activities related to the textbook material. 76 72 had students do individual presentations of projects related to the text material. 28 27 both choices marked 1 1 had students do individual presentations of projects related to the text material. 36 35 had students do group presentations of projects related to the textbook material. 66 64 both choices marked 1 1 gave in-class closed-book quizzes over textbook content. 36 35 gave online open-book quizzes over textbook content prior to the class period. 68 65 although 64% of respondents overall preferred group presentations, there was a significant relationship of this preference with year in college. sophomores and seniors were more likely than expected to opt for individual work, while freshmen and juniors were more likely to prefer group presentations (see table 9). table 9 chi-square analysis of preference for group vs. individual presentations by year in college freshman sophomore junior senior/grad item response n (row %) n (row %) n (row %) n (row %) individual 6(17) 11(31) 3(8) 16(44) group 20(30) 8(12) 22(33) 22(24) note: chi-square = 14.92; df=3; p < 0.0019 for the follow-up analysis by category of major, the only item where student responses showed much difference dealt with whether they believed they should be required to read the textbook prior to coming to class. as shown in table 10, students who were not in science, technology, or math majors were more likely to answer “it depends” to this question. because this analysis was a follow-up of the main research questions, it was not subjected to the bonferroni adjustment of alpha and thus was statistically significant at p < 0.05. discussion the purpose of this study was to ascertain how students at this institution use textbooks assigned for their respective classes, as well as the reactions and preferences that they have regarding instructor-directed presentation methods and other activities. results of this investigation indicate substantial variation in how text materials are used and preferences for what takes place aagaard, l., conner ii, t.w., & skidmore, r.l. journal of the scholarship of teaching and learning, vol. 14, no. 3, august 2014. josotl.indiana.edu 140 table 10 chi-square analysis of category of college major by “should be required to read text?” science/technology/math humanities/education/ business/social science item response n (row %) n (row %) yes 16(53) 14(47) no 11(58) 8(42) it depends 16(31) 36(69) note: chi-square = 6.21; df=2; p < 0.045 during class time. whether students read text materials seems dependent upon other factors, and not just simply whether the materials are assigned by the instructor (e.g., table 5). typically, students indicated that they would read the text materials if there were some associated ‘for credit’ activity, if the text was used during class time, and if the size of reading assignments was “shorter.” interestingly, first-year (i.e., freshmen) students felt that reading assigned text materials before class time should not be required, with seniors acknowledging that reading the text materials depends upon “other” factors. we speculate that these perspectives and attitudes are the result of their prior academic experiences. college freshmen likely utilize the same techniques during their post-secondary experiences. for example, due to economic reasons, text materials may only be available to students for review during a given class period during their secondary school experience. concrete presentation and the opportunity for the development of effective independent learning strategies necessary for life-long learning are not likely promoted or practiced. college seniors, however, have likely ‘learned the ropes’ with respect to the general requirements and expectations of post-secondary experiences, and are able to adjust and adapt their behaviors accordingly. it would seem that students prefer a “structured with variation” model when it comes to the use of class time. participants preferred a lecture format, but with related and authentic nontext material / content included. in addition, the preference was expressed for in-class group discussion of the text and other materials as well as for prepared organizers (i.e., powerpoint slides). not surprisingly, online open-book quizzes were preferred to in-class closed-book quizzes, yet respondents indicated that they would read the textbook more frequently if quizzes were given in-class in contrast to online. freshmen and juniors would prefer a group presentation format if this were a component of a course. this might be due to a preference for “safety-innumbers” (i.e., preservation of self-esteem) in open-forum evaluative situations. interestingly, and somewhat surprisingly, sophomores and seniors preferred to work independently. perhaps they have been frustrated with prior group projects / presentation experiences where one’s evaluation is at least partially determined by the cooperative participation of others (i.e., “once bitten, twice shy”). senior students may be focused upon finishing course work necessary for attainment of the degree, and therefore are reluctant to expend the socio-emotional time and effort necessary to engage in effective group activities. the responses of the students participating in this study give indications to instructors regarding how to encourage student reading of the textbook prior to its discussion during a class period. if the top-rated strategies were combined it would mean that students would be given a short reading assignment with a study guide to complete (for credit) during their reading. then, aagaard, l., conner ii, t.w., & skidmore, r.l. journal of the scholarship of teaching and learning, vol. 14, no. 3, august 2014. josotl.indiana.edu 141 after an in-class quiz over the reading assignment, the instructor would have students use the textbook in some way during the class period. when it came time for the unit test, students also would be responsible for material that was in the textbook but not discussed in class. limitations of study although the results offer much for any post-secondary educator to consider, limitations to this particular study emerge with respect to sampling techniques, time of acquisition of data, and weaknesses in understanding differences across students in particular fields. cluster sampling techniques were used to collect data primarily out of necessity. perhaps if the study is replicated, researchers could randomly select students within disciplines. secondly, the survey was administered to students during a four-week summer term. the number of participants would have been higher if the survey had been administered during a typical sixteen-week academic term. an increase in student participation rates might offer results that are different from the findings of this study. it is also possible that there is something different about students who take classes in the summer terms, so these results may not apply equally to students who never participate in summer school. additionally, the distribution of student majors in this study’s sample was not sufficient for any meaningful comparisons of students’ preferences of textbook reading and class activity across different programs of study. future researchers in this area may want to intentionally administer surveys to students in particular fields of study to allow for analysis of differences across student majors. a more nuanced analysis like this might reveal differences in preferences between major programs, particularly if the results show consistencies in majors across multiple colleges/universities. finally, it should be noted that the instrument used for this study was researcher-created. in future replication of this work, researchers would be advised to not only establish the validity of the items in the current instrument (see appendix), but also to include additional items that relate to other proximal and / or personal / dispositional factors that may also influence students’ use of textbooks and their preferences for various classroom strategies. appendix appendix a textbook and use of class time survey (with overall survey results included as percentages and significance notes where appropriate.) 1. do you think you should be required to read material in the textbook before coming to class? yes no it depends 30% 20% 50% ***signif. chi-square for year in school (p=0.0001): freshmen more likely to say no; sophomores more likely to say yes; seniors more likely to think it depends. aagaard, l., conner ii, t.w., & skidmore, r.l. journal of the scholarship of teaching and learning, vol. 14, no. 3, august 2014. josotl.indiana.edu 142 2. do you actually read the textbook material when it is assigned? yes no it depends 52% 6% 48% 3. what could the professor do to get you to read the textbook assignments? (mark the most appropriate column for each strategy.) professor’s strategy i would still not read the textbook. i might read the textbook. i would most likely read the textbook. a. give me an in-class quiz over material from the textbook assignment. ***sig. chi-square for gender*** p=.0003 0% 27% m: 46% f: 14% 73% m: 54% f: 86% b. give me an online open-book quiz over the textbook assignment. 19% 34% 47% c. have a study guide for me to fill out while reading the assignment, then give me credit for turning it in. 4% 19% 77% d. actually discuss the content of the textbook assignment in class. 14% 38% 48% e. test me over material that was in the textbook but not discussed in class. 5% 14% 81% f. make shorter reading assignments. 1% 22% 77% g. use the textbook in class in some way. 5% 43% 52% h. teach me how to use the textbook’s instructional features (glossary, summaries, etc.). 34% 37% 29% 4. have you ever used an e-textbook for one of your classes? yes no what is an e-textbook? 24% 51% 24% ***signif. chi-square for year in school (p=0.0001): freshmen less likely to say yes and more likely to say “what is it?”; sophomores more likely to say yes; seniors less likely to say “what is it?” 5. i would prefer the professor lectured: 18% a. only over material that was in the textbook. 81% b. over the textbook, but also some material that was not in the textbook. 1% (both) 6. i would prefer the professor used: 71% a. powerpoint slides to present basic notes for the lecture. 24% b. the chalkboard to present basic notes for the lecture. 5% (both) 7. i would prefer the professor lectured: 43% a. only over material that will be tested. 57% b. over tested material, but also over some material that is interesting but not going to be tested. aagaard, l., conner ii, t.w., & skidmore, r.l. journal of the scholarship of teaching and learning, vol. 14, no. 3, august 2014. josotl.indiana.edu 143 8. i would prefer the professor: 7% a. lectured only over the textbook material. 92% b. covered the content, but also gave examples of how the material applied to real life. 1% (both) 9. i would prefer the professor: 26% a. just lectured over the textbook material in some way. 72% b. encouraged group discussion of the material. 2% (both) 10. i would prefer the professor: 41% a. just lectured over the textbook material in some way. 58% b. had students do group activities related to the material. 1% (both) 11. i would prefer the professor: 45% a. encouraged group discussion of the textbook material. 53% b. had students do in-class group activities related to the textbook material. 2% (both) 12. i would prefer the professor: 72% a. had students do in-class group activities related to the textbook material. 27% b. had students do individual presentations of projects related to the textbook material. 1% (both) 13. i would prefer the professor: 35% a. had students do individual presentations of projects related to the textbook material. 64% b. had students do group presentations of projects related to the textbook material. 1% (both) ***signif. chi-square for year in school (0.0019): sophomores and seniors more likely to prefer individual presentations; juniors more likely to prefer group presentations. 14. i would prefer the professor: 69% a. lectured over content in class. 27% b. put the lectures in audio files online to be listened to prior to class, then did other interesting things related to the content during class. 4% (both) 15. i would prefer the professor gave: 35% a. in-class closed-book quizzes over textbook content. 65% b. online open-book quizzes over textbook content prior to the class period. 17. what is your gender? 39% a. male 61% b. female aagaard, l., conner ii, t.w., & skidmore, r.l. journal of the scholarship of teaching and learning, vol. 14, no. 3, august 2014. josotl.indiana.edu 144 18. what year of college are you in? 25% a. freshman 18% b. sophomore 25% c. junior 31% d. senior 1% (graduate student) 20. what is your overall gpa? 0% a. 0-0.99 3% b. 1.0-1.99 30% c. 2.0-2.99 60% d. 3.0-3.99 4% e. 4.0 references aagaard, l., & skidmore, r. (2004, november). student study habits and the relationship to test scores in an undergraduate course. paper presented at the annual meeting of the midsouth educational research association, gatlinburg, tn. aagaard, l., & skidmore, r. (2009, november). college student use of textbooks. paper presented at the annual meeting of the midsouth educational research association, baton rouge, la. berry, t., cook, l., hill, n., & stevens, k. (2011). an exploratory analysis of textbook usage and study habits: misperceptions and barriers to success. college teaching, 59, 31-39. doi: 10.1080/87567555.2010.509376 clump, m.a., bauer, h., & bradley, c. (2004). the extent to which psychology students read textbooks: a multiple class analysis of reading across psychology. journal of instructional psychology, 31, 227-232. retrieved october 16, 2009, from ebscohost academic search premier database. johnson, b. c., & kiviniemi, m. t. (2009). the effect of online chapter quizzes on exam performance in an undergraduate social psychology course. teaching of psychology, 36, 33–37. doi: 10.1080/00986280802528972 kane, b. (1990). undergraduate students’ concerns and preferences regarding structured and unstructured lecture-discussions. education, 111(1), 117-125. retrieved october 26, 2010, from ebscohost academic search premier database. lei, s.a., bartlett, k.a., gorney, s.e., & herschbach, t.r. (2010). resistance to reading compliance among college students: instructors’ perspectives. college student journal, 44(2), 219-229. retrieved october 29, 2010, from ebscohost academic search premier database. aagaard, l., conner ii, t.w., & skidmore, r.l. journal of the scholarship of teaching and learning, vol. 14, no. 3, august 2014. josotl.indiana.edu 145 locher, d. (2004). when teaching less is more. the teaching professor, 18(9), 2-6. retrieved october 26, 2010, from ebscohost academic search premier database. mji, a., & kalashe, l. (1998). exploring the lecturer-student relationship: preferences of a sample of undergraduate students in business management. psychological reports, 83, 12971298. doi: 10.2466/pr0.83.7.1297-1298 oliver, r. (2007). exploring an inquiry-based learning approach with first-year students in a large undergraduate class. innovations in education and teaching international, 44(1), 3-15. retrieved october 26, 2010, from ebscohost academic search premier database. doi: 10.1080/14703290601090317 pecorari, d., shaw, p., irvine, a., malstrom, h., & mezek, s. (2012). reading in tertiary education: undergraduate student practices and attitudes. quality in higher education, 18(2), 235-256. doi: 10.1080/13538322.2012.706464 ruscio, j. (2001). administering quizzes at random to increase students' reading. teaching of psychoglogy, 28(3), 204-206. retrieved october, 30, 2010, from ebscohost academic search premier database. doi: 10.1207/s15328023top2803_08 ryan, m.p. (2001). conceptual models of lecture-learning: guiding metaphors and modelappropriate notetaking practices. reading psychology, 22, 289-312. retrieved october 26, 2010, from ebscohost academic search premier database. doi: 10.1080/02702710127638 ryan, t. e. (2006). motivating novice students to read their textbooks. journal of instructional psychology, 33(2), 136-140. retrieved october 30, 2010, from ebscohost academic search premier database. sikorski, j. f., rich, k., saville, b. k., buskist, w., drogan, o., & davis, s. f. (2002). student use of introductory texts: comparative survey findings from two universities. teaching of psychology, 29, 312-313. retrieved october 16, 2009, from ebscohost academic search premier database. doi: 10.1207/s15328023top2904_13 struyven, k., dochy, f., & janssens, s. (2012). explaining student’s appraisal of lectures and student-activating teaching: perceived context and student characteristics. interactive learning environments, 20(5), 391-422. doi: 10.1080/10494820.2010.500084 zhang, l.f. (2008). preferences for teaching styles matter in academic achievement: scientific and practical implications. educational psychology, 28(6), 615-625. doi:10.1080/01443410802004634 3147rooks journal of the scholarship of teaching and learning, vol. 13, no. 2, may 2013, pp. 84 – 100.   enhancing curriculum through service learning in the social determinants of health course ronica nicole rooks1 and christine tagliaferri rael abstract: service learning bridges classroom learning and community volunteerism and is anchored in the curriculum, classroom discussion, and community. we incorporated service learning projects (slp) into three social determinants of health courses (2008-2010) to promote: experiential learning; undergraduate scholarship; faculty career development through the scholarship of teaching and learning; and collaborative university-community research to reduce social inequalities in health. we examined whether slp facilitated student learning of course concepts. we used mixed methods analyzing students’ (n=25) pre-/post-test surveys, research papers, and site supervisors’ (n=17) interviews. despite positive survey ratings, results showed decreased student agreement about slp facilitating student learning. content analysis revealed specific student themes: finding slp rewarding for future public health careers; aligning student interests with community-based organizations (cbos)’ goals; and valuing interactive experiences with cbos’ clients. students gained beneficial career development skills with cbos but needed better preparation for their slp by increased discussion of their and cbos’ expectations. keywords: service learning, social determinants of health, social inequalities, undergraduates, public health i. introduction. service learning bridges classroom learning and community volunteerism, where service is anchored in the curriculum, classroom discussion, and community (ballantine & phelps, 2002), i.e. the laboratory to apply public health and social science knowledge and skills to social issues. it also promotes undergraduate scholarship through research, professional networking, and service. in recognition of these benefits the institute of medicine recommends that all undergraduates should have access to public health education, including an understanding of how the social and physical environments shape health through an ecological model (cashman & seifer, 2008; gebbie, rosenstock, & hernandez, 2003). service learning can be a vehicle to fulfill this mandate by applying public health and social and behavioral sciences to healthfocused social problems. we are particularly interested in the benefit of improving student outcomes through problem-based learning with a community-based organization (cbo). the literature on healthfocused service learning identifies various benefits to students and cbos. students’ benefits include: improved academic outcomes such as better grades, critical thinking, and problemsolving skills; connecting theory to practice; increased capacity to view phenomenon from multiple perspectives and a deepened sensitivity to diversity; increased social awareness and                                                                                                                 1 department of health and behavioral sciences, ronica.rooks@ucdenver.edu rooks, r.n., and rael, c.t. journal of the scholarship of teaching and learning, vol. 13, no. 2, may 2013. josotl.indiana.edu 85 social justice beliefs; increased confidence to make community service contributions (e.g., selfefficacy); leadership; and enhanced understanding of their professional development and career paths (cashman & seifer, 2008; peterson & yockey, 2006; reeb, 2010). cbos’ benefits include: volunteer hours to meet the organization’s goals addressing social problems; freeing the organization’s time and resources for other projects; gaining new ideas from outsider perspectives; programmatic and infrastructure improvements; opportunities to contribute to students’ education; and developing positive relationships with the university (blouin & perry, 2009; brown et al., 2006; cashman & seifer, 2008; peterson & yockey, 2006). however, little research exists on whether universities are successful in improving student learning outcomes as they address health-focused social problems and social inequalities through community outreach (kenny & gallagher, 2002; mobley, 2007). as a public health student toboada (2011) critiqued community service learning models designed as charityoriented activities, traditionally pairing white, middle-class students with low income and communities of color. based on her experiences, students need proper curriculum training in racism, power, and privilege before and during their interactions with communities so that their experiences do not reinforce bias. addressing the prior issues loewenson and hunt (2011) used social determinants of health (sdoh), health disparities, and social justice frameworks, as well as theory on working with underserved populations, to positively transform public health nursing students’ attitudes toward the homeless. by the end of the semester students reported stronger beliefs related to structural causes for homelessness and more comfort associating with homeless people. this research shares the theme of adequate curriculum preparation on social inequalities in health prior to and concurrent with students’ service learning experiences. thus, as instructors a critical first step to enhancing the health of vulnerable populations is improving students’ attitudes towards working with them throughout the semester (daiski, 2007). to facilitate student learning, theory on student engagement supports using service learning projects (slp) to foster community social change. giles and eyler (1994) developed service-learning theory from dewey’s principles of experience, inquiry, and reflection within his theory of knowing. dewey’s educational philosophy connects theory to practice in communities for social justice. he presents problems that awaken students’ curiosity and demand for information. students build on their prior experiences through interaction and reflective thinking, resulting in learning. similarly, kolb’s (1984) experiential learning theory links education, paid work, and personal development to a lifelong active process of learning through direct interactions with phenomena. kolb believes that learning is an active process where knowledge is created through transforming experiences. considering these guidelines, we incorporated slp into three sdoh courses (fall 20082010) with the goals of promoting: experiential learning at the intersection of public health and the social and behavioral sciences; undergraduate scholarship and professional development; faculty career development through the scholarship of teaching and learning; and collaborative university-community research to reduce social inequalities in health. the major objective of this course was for students to analyze the relationships between the individual pursuit of health and the social structural contexts in which this happens in our society, with some cross-national discussions. we examined issues related to the social, psychological, behavioral, economic, political, cultural, and environmental variations in health and disease, particularly focusing on social inequalities in health, social stress, health behaviors, illness experiences, relations between rooks, r.n., and rael, c.t. journal of the scholarship of teaching and learning, vol. 13, no. 2, may 2013. josotl.indiana.edu 86 providers and patients, the structure and processes of healthcare organizations, financial and other barriers to accessing healthcare, health policy, and social change. students were required to do slp, engaging in civic responsibility while connecting course concepts to their service. the assignment specified that students do their slp with a local health or healthcare-focused cbo. the specific sdoh topics to focus on in their service learning experiences and research papers were left to students, but we suggested the following general topics: disease (e.g., biomedical perspective) versus illness (e.g., psychosocial perspective) in society, health behaviors, social stressors, experiencing illness and disability, interactions with healthcare professionals, the healthcare system, healthcare delivery, treatment, healthcare policy, etc. from a patient’s and/or an organization’s perspective. students had to: 1) provide a service learning contract describing their slp. 2) work at least 20 hours in a cbo. 3) keep journal notes on their experiences, observations, and reflections. 4) participate in two in-class discussions, reflecting on their slp. 5) complete a final research and evaluation paper of their slp. 6) provide a letter verifying their completion of their slp and hours signed by their site supervisor. we provided students with slp learning goals, a rubric, and on-going course discussions about the slp throughout the semester to model successful student learning. we included the rubric in appendix 1. this paper addresses whether slp facilitate applied student learning of sdoh course concepts using mixed methods. our intent was to help students gain an awareness of social inequalities in health, understand their own biases, and illustrate how these prior issues could affect health-related problems. ii. methods. a. study design. with institutional review board approval from the university of colorado denver, we examined the service learning experiences of sdoh students and their site supervisors. the sample included university students from three semesters in 2008 (undergraduate), 2009 (undergraduate and master’s level), and 2010 (undergraduate and co-taught course) and their site supervisors, primarily working in non-profit, health-related cbos in denver, colorado. we used mixed methods, descriptively analyzing students’ preand post-test quantitative surveys and their research papers and site supervisors’ interviews using qualitative analysis. we divided students (n=25) and supervisors (n=17) into two mutually exclusive groups and asked them to respond to surveys or interviews unique to their group. student sample. a research assistant proctored student consent forms 4-5 weeks into each course. students either immediately turned their consents in or by semester’s end. students’ involvement in service learning data collection was voluntary, and they were able to refuse participation at any time, for any reason. we asked students to respond to preand post-test surveys to assess their service-learning experiences, with open-ended comments at the end (gelmon, holland, driscoll, spring, & kerrigan, 2001). gelmon and colleagues present a long and revised, shorter version of this survey in their research, developed for use with undergraduate students. we used questions from the longer version, but substituted in a few rooks, r.n., and rael, c.t. journal of the scholarship of teaching and learning, vol. 13, no. 2, may 2013. josotl.indiana.edu 87 questions from the revised version. for the pre-test, students were asked to respond to whether they believed their slp would benefit their learning of sdoh course material (prior to beginning their slp). the post-test was the same as the pre-test, except we changed questions to past tense (post completing their slp). we administered the pre-test within the second week and the post-test during the last week of the course. we instructed students to choose a unique, random number to anonymously match their preand post-tests. additionally, we examined students’ slp papers with reflections about their experiences. site supervisors/cbo sample. we conducted follow-up interviews after final grade submissions so that site supervisors’ comments did not factor into students’ grades. prior to conducting mostly telephone interviews a research assistant obtained signed consents from site supervisors via email or fax. site supervisors’ survey participation was completely voluntary; they were allowed to stop the interview at any time. the interview guide included questions about students’ responsibilities, whether students had sufficient knowledge to volunteer at the site, the quality of their work, whether students had professional characteristics (e.g., showed initiative, communicated effectively, and reliability), and whether students’ work at the sites was beneficial to the cbo, other positive or negative aspects associated with this experience, and suggestions for improvement to the slp. dependent variables. to assess students’ learning we used quantitative surveys and qualitative paper reflections focused on the value of service learning as an applied tool for learning course concepts. student surveys included twenty-four items on whether their attitudes toward community service in the course changed during their slp experiences (shown in table 2). each item indicated students’ level of agreement via a 5-point likert scale (1=strongly disagree to 5=strongly agree). survey items reflected a cronbach’s alpha of 0.884, indicating high internal question consistency. students were also asked to qualitatively reflect on their slp experiences in their papers’ research/evaluation and conclusion sections, particularly focusing on any applied learning gained in the course or for their future professional development. independent variables. we included five socio-demographic survey items to understand any variation among our students. race/ethnicity included the following categories: caucasian/white, african american/black, hispanic/latino, native american, asian/asian american, or other. age included: 18-24, 25-34, 35-44, or 45-54. sex included: male or female. class level included: freshman, sophomore, junior, senior, post-baccalaureate, graduate, or other. and, paid work/hours per week included: 1-10, 11-20, 21-30, 31-40, 41+ hours, or “i do not have a job.” statistical analysis. for our quantitative analysis we used a paired sample t-test to determine whether students’ mean level of agreement regarding their learning through community service varied significantly between the preand post-tests. due to the small sample size, we set significance at 0.1 to increase the likelihood of attaining meaningful results. for our qualitative analysis we used content analysis to examine students’ survey comments and research papers and site supervisors’ interviews in microsoft excel (swallow, newton, & van lottum, 2003). we developed a priori codes based on the surveys and interview questions, as well as deductive codes via emerging themes. to increase inter-rater reliability, both co-authors reviewed all documents. combining both types of analyses as mixed methods is beneficial for data triangulation, providing generalizability and contextual-depth to our analyses (borkan, 2004). rooks, r.n., and rael, c.t. journal of the scholarship of teaching and learning, vol. 13, no. 2, may 2013. josotl.indiana.edu 88 iii. results. a. quantitative analyses. table 1. shows summary statistics of students’ socio-demographic characteristics from the pre and post-test surveys. the majority of students were female, caucasian/white, seniors in class level, and working between 21-30 hours a week. table 3 shows paired sample t-test mean responses for students’ preand post-test surveys. students generally rated the pre-test items between neutral to agree, with the majority of items slightly decreasing in agreement, a negative change, by the time of their post-tests. the only significant items included: q.1, q.8, q.20, and q.22, showing that students felt less likely to believe: their service learning work could be used in their everyday lives; it benefitted the community; it made them aware of their own biases and prejudices; and it helped them enhance their leadership skills. students’ ratings on eight questions (e.g., q.4, q.6, q.10-11, q.17-19, and q.21) slightly increased, ranging from neutral to agree or remaining the same over time, but none of these findings were significant. while the final question (q.3) did not show a significant change over time, students slightly increased their disagreement that they could have learned more from this course if more time was spent in the classroom instead of doing community work. the highest student agreement items at the beginning and end of the course focused on them being comfortable working with other cultures and being able to make a difference in their communities. we also showed significant change items by course year (table 3), where most significant items did not overlap in each year. in 2008, two items showed improvement, with students changing from neutral to agree regarding their interactions with their community partner enhancing their course learning and the community work assisting them in defining their profession. but, the remaining two items decreased from agree to neutral, regarding the community work making students more aware of their own biases and feeling they could make a difference in their communities. in 2009, students increased their disagreement with the item that they could learn more from the course by spending more time in the classroom rather than the community and increased their agreement with the item that performing community work helped them clarify their majors. students decreased their agreement with the items concerning community work helping them become more aware of their communities’ needs; it assisting them in defining their profession; it making them more marketable in their profession; and it helping them develop problem-solving skills. in 2010, students were less likely to believe that community participation helped them to see how the subject matter could be used in everyday life; more likely to agree about volunteering before the course; and more likely to agree that they have a responsibility to serve their community. despite the items remaining positive, about half of the significant changes across each year occurred in a negative direction. b. qualitative analyses. rooks, r.n., and rael, c.t. journal of the scholarship of teaching and learning, vol. 13, no. 2, may 2013. josotl.indiana.edu 89 students’ comments revealed that those who intended to pursue a public health career found this experience especially relevant to their course learning and career development. students stated: “…the most valuable thing this course did for me was to classify the practical need our individual people, counties, communities, states, and nations as a whole all have for public health.” “i think that the service learning project is so great because it gives us real-world experience and allows us to see the class-content outside of the classroom.” “i would rate this experience as substantial in regards to a research or advocacy position i might employ one day. i feel this way because i assume there will be many jobs within the human and health services sector i.e. public health that require a large amount of research…” “this project also helped solidify that my future educational endeavors will be focused around hiv/aids.” “the lasting effect this project has is that i know how much more research needs to be put into prenatal epidemiology and how important public health issues in maternal and child health need to be addressed… this experience impacted me on a greater level than simply scholastic achievement. i felt like i gained real world knowledge…” table 1. summary statistics of students’ demographic characteristics from the preand posttest surveys. pre-test post-test percent percent gender male 9.4 12.5 female 84.4 68.8 race/ethnicity caucasian/white 53.1 50.0 african american/black 12.5 3.1 hispanic/latino 15.6 12.5 asian/asian american 3.1 3.1 other 6.2 6.2 multiracial 6.2 6.2 class level sophomore 9.4 3.1 junior 28.1 31.2 senior 50.0 37.5 graduate 9.4 9.4 hours worked per week 1-10 hrs/wk 9.4 15.6 11-20 hrs/wk 6.2 3.1 21-30 hrs/wk 28.1 31.2 31-40 hrs/wk 28.1 21.9 41+ hrs/wk 6.2 0 i do not have a job 18.8 6.2 source: social determinants of health students in 2008-2010. notes. our original pre-test (n=31) and post-test (n=27) sample sizes varied due to: 1) differences in those students who began the course, prior to the drop/add period, and those who finished the course and 2) absences on the days the surveys were administered. after we matched the preand post-test surveys a sample size of 25 students remained. we coded the unmatched rooks, r.n., and rael, c.t. journal of the scholarship of teaching and learning, vol. 13, no. 2, may 2013. josotl.indiana.edu 90 surveys as missing data. rooks, r.n., and rael, c.t. journal of the scholarship of teaching and learning, vol. 13, no. 2, may 2013. josotl.indiana.edu 91 table 3. paired sample t-test mean results for students’ preand post-test surveys. item statement pre-test mean post-test mean sign. (2-tailed) q.1) the community participation aspect of this course will help me to see how the subject matter i learn can be used in everyday life. 4.24 3.44 0.011+ (q.2) the community work i will do through this course will help me to better understand the lectures and readings in this course. 3.76 3.48 0.283 (q.3) i feel i could learned more from this course if more time was spent in the classroom instead of doing community work. 2.76 2.60 0.491 (q.4) the idea of combining work in the community with university coursework should be practiced in more classes at this university. 3.88 3.96 0.723 (q.5) i am responsible for the quantity and the quality of knowledge that i obtain from this course. 4.28 4.00 0.183 (q.6) i was already volunteering in my community before taking this course. 3.32 3.60 0.230 (q.7) the community participation aspect of this course will show me how i can become more involved in my community. 4.08 3.96 0.560 (q.8) i feel that the community work i will do through this course will benefit the community. 4.16 3.56 0.036+ (q.9) the community work involved in this course will help me to become more aware of the needs in my community. 4.04 3.84 0.446 (q.10) i have a responsibility to serve my community. 4.1 4.29 0.382 (q.11) my interactions with the community partner will enhance my learning for this course. 3.72 3.72 1.000 (q.12) doing work in the community will help me to define my personal strengths and weaknesses. 3.76 3.64 0.641 (q.13) performing work in the community will help me clarify which major i will pursue. 3.46 3.17 0.418 (q.14) the community work in this course will assist me in defining which profession i want to enter. 3.71 3.38 0.224 (q.15) the work i will accomplish in this course will make me more marketable in my chosen profession when i graduate. 4.04 3.92 0.450 (q.16) the community aspect of this course will help me to develop my problem-solving skills. 3.68 3.56 0.574 (q.17) the syllabus provided for this course outlined the objectives of the community work in relation to the course objectives. 3.96 4.00 0.857 (q.18) most people can make a difference in their community. 4.36 4.36 1.000 (q.19) i am comfortable working with cultures other than my own. 4.48 4.48 1.000 (q.20) the community work involved in this course will make me aware of some of my own biases and 4.04 3.60 0.013 + rooks, r.n., and rael, c.t. journal of the scholarship of teaching and learning, vol. 13, no. 2, may 2013. josotl.indiana.edu 92 prejudices. (q.21) the work i will perform in this course will help me learn how to plan and complete a project. 3.80 3.80 1.000 (q.22) participating in the community will help me enhance my leadership skills. 4.04 3.64 0.047 + (q.23) the work i will perform in the community will enhance my ability to communicate my ideas in a real world context. 4.08 3.84 0.110 (q.24) i can make a difference in my community. 4.56 4.40 0.256 source: survey questions from gelmon et al. (2001) for the social determinants of health students in 2008-2010. + our significance level is set at α=0.1 table 3. significant mean results for students’ pre-and post-test surveys by course year. item # 2008 2009 2010 n mean t-test sig (2tailed) n mean t-test sig (2tailed) n mean t-test sig (2tailed) 1 14 4.29 2.242 .043 14 3.50 3 6 2.17 2.236 .076 6 2.67 6 14 3.00 2.223 .045 14 3.64 9 6 4.33 2.712 .042 6 3.50 10 14 3.86 2.876 .013 14 4.36 11 5 3.60 2.138 .099 5 4.40 13 6 4.00 3.841 .012 6 4.17 14 5 3.40 2.138 .099 6 4.00 2.697 .043 5 4.20 6 2.67 15 6 4.17 2.236 .076 6 3.67 16 6 4.00 2.236 .076 6 3.00 20 5 4.20 2.449 .070 5 3.60 24 5 4.80 4.000 .016 5 4.00 source: social determinants of health students in 2008-2010. + our significance level is set at α=0.1. “the project was a great addition to the social determinants of health course as well as to my personal exposure and experience…i gained valuable knowledge…including skills like data collection methods…i plan to continue my career path focusing on public health, sociology, and demography so obtaining these skills hands on has been a great asset to my academic and professional life…this was my first time [in college] with the opportunity to rooks, r.n., and rael, c.t. journal of the scholarship of teaching and learning, vol. 13, no. 2, may 2013. josotl.indiana.edu 93 apply my knowledge from inside the classroom, to something that is occurring right now, in the real world.” a second theme was the importance of aligning individual student interests with the organization’s goals. students who provided negative feedback often indicated that had their personal objectives been more compatible with those of their cbos, they would have had a more enriching learning experience. one student commented, “if i had chosen a better site – many of my ‘disagrees’ [in the post-test] would have probably been ‘agrees.’” participants in volunteer opportunities stressing labor-oriented tasks, such as restocking shelves, cleaning, or running errands, were less likely to positively rate their experience than students who participated in research or outreach activities. in addition, students who strictly performed labor jobs were more likely to perceive their volunteering as a burden rather than as learning. a third theme was some students’ desires to be more interactive with their cbos’ clients. these students initiated their slp believing they would have more “hands-on” learning experiences with people, but some cbos worked at a macro-structural level needing students to do background research for educational materials, designing surveys, and influencing policy. one student who worked with a preventive health center cbo stated, “the weakness of the project was the lack of direct interaction with the students. i never interviewed the transgender students, therefore my poster could have been off or the flyer might have been irrelevant to the table 4. major themes identified from students’ and cbos’ perceptions. student themes 1. slp provided career development. 2. slp were more beneficial when students’ interests aligned with cbos’ goals. 3. slp were more beneficial when there was interactive, “hands-on” learning with cbos’ clients. cbos’ themes 4. students should understand the cbo’s mission and have a clear idea of what they want to accomplish with their slp. 5. student slp volunteers were beneficial to cbos, who often had limited budgets. 6. having student volunteers provided publicity for cbos, raising public awareness about their causes. source: social determinants of health students and their site supervisors in 2008-2010. needs of those particular students.” a second student found that her initial expectations of working on child abuse and neglect issues were disappointingly different than her experiences stating, “i thought that by working with organizations whose emphasis was on child wellbeing, that a person-to-person interaction with the children was part of my description…i thought it would be more a community center type atmosphere and not offices pushing out paperwork?…i wanted a more hands on experience...however, when taking a step back and analyzing the situation objectively i felt that what i was trying to accomplish…was exactly what they needed.” while some students expressed these concerns, many others were satisfied with their direct, interactive learning and experienced positive improvement in their early expectations to real experiences by semester’s end. one student stated, “while it was an accident that i ended up in the mobile clinic, i think that it was the best place i could have been because of the direct contact with patients and because that is where the need was and continues to be.” a student rooks, r.n., and rael, c.t. journal of the scholarship of teaching and learning, vol. 13, no. 2, may 2013. josotl.indiana.edu 94 working in a long-term care facility was initially terrified, “…expecting to find men and women who were on their death beds and getting ready to die…instead my views and expectations were altered after my time at [anonymous cbo], and i now have a renewed understanding and appreciation for quality long term healthcare.” another student stated, “i would recommend to future students who want to volunteer with this organization to interact more with those who are homeless and spend as much time as you can with this organization because there are a lot of wonderful people to meet and interesting things to learn about the homeless in denver.” students’ themes were consistent with many made by site supervisors, where the first supervisor theme was that students and cbos could experience greater benefits if students had an introductory-level background in their cbos’ mission topic and a clear idea of what they wanted to accomplish with their slp. site supervisors were willing to train students in all relevant technical skills, but they agreed that incoming students should have some familiarity with the cbo. one supervisor reported, “students should learn about the background of the organizations…checking the website, being knowledgeable about what the organization does. they should come with a few ideas about what they would like to learn.” once students became proficient with their acquired skills, demonstrating a positive learning transition, supervisors relied on students to provide their cbos with meaningful assistance. despite the prior concern, at least two slp turned into part-time paid employment for students. these service learners were described as “enthusiastic” and “self-starters” by their site supervisors. both students forged strong bonds with the populations in which they worked. positive responses by site supervisors were overwhelming. this second theme was nearly uniform across all sites. cbos reported that working with students was a great experience and provided lasting benefits to their organizations. most cbos operate on a limited budget and many supervisors reported that having volunteers was extremely helpful. a third theme was publicity. many supervisors viewed slp as a way to raise awareness about their causes. one supervisor required students to shadow her before beginning their individual projects on homelessness. she stated, “a lot of people don’t have a background on this and it’s nice to be able to educate the community and to get our name and message out there.” often university students had no previous contact with the populations in which they worked. by exposing students to these groups, cbos were able to promote greater public understanding of their clients. iv. discussion. perhaps the most striking finding on whether slp facilitated student learning was the mean decrease in student agreement levels on all significant measures from the preto post-tests, despite ratings still occurring between neutral and strongly agree. these findings were not consistent with most service-learning research, where students’ attitudes generally showed improvement in social responsibility, awareness, and social justice (bach & weinzimmer, 2011; long et al., 2011; ottenritter, 2004; reeb, 2010). however, some researchers have encountered difficulties with decreasing student ratings (parker-gwin & mabry, 1998; villanueva, hovinga, & cass, 2011). while gelmon, et al. (2001) state that few students will demonstrate dramatic changes in their preand post-tests during a one semester course, we offer possible reasons for students’ decreased agreement about their slp. two explanations relate to earlier themes of some students not having compatible interests with their cbos and differences in students’ initial expectations rooks, r.n., and rael, c.t. journal of the scholarship of teaching and learning, vol. 13, no. 2, may 2013. josotl.indiana.edu 95 versus later realities regarding client interactions. third, the higher percentage of students working greater than 20 hours a week while taking this course and doing their slp could have been an added stressor, dampening their experiences. from post-course student interviews, madsen and turnbull (2006) found that students’ paid work was the biggest challenge for their service learning experiences. fourth, while students were advised to avoid organizations that did not include applied research as a part of their slp, some students did administrative tasks not relevant to their slp. one future strategy to fix this problem is to channel students toward the faculty pre-approved cbos who present their possible slp in the course. fifth, students’ selfselection into this course and their initial naiveté or idealism about doing applied research and helping others could have negatively changed by semester’s end. students’ attitude changes were possibly prompted by slp difficulties, including: site selection, site interactions, or meeting cbos’ project goals. to address the prior difficulties, we believe that instructor awareness of challenges is warranted to better prepare students and faculty for interaction with cbos. various authors reference challenges associated with students’ service learning interactions with cbos (blouin & perry, 2009; cashman & seifer, 2008; peterson & yockey, 2006). they mention unprofessional and/or unreliable students who risk cbos’ invested resources and the populations they serve. sometimes students misrepresent cbos in their writing assignments, when cbos’ missions are not learned and/or understood. another problem is poor communication between faculty and cbos, when cbos’ roles are not clearly defined in assignments. these challenges can reduce cbos’ enthusiasm to work with university students and faculty and must be addressed to sustain university-community partnerships for service learning. we addressed these issues by becoming more proactive and inviting cbos and the university’s experiential learning center to present their potential projects to help students find sites. we reminded students early in the semester about being professionals as they interact with cbos, making sure they dress appropriately, show up on time, and learn the missions of their cbos. in addition, for cbos that we recommended to students, we emailed them the assignment and goals within the first two weeks of the semester so that they could ask us any questions if needed. similarly, when students chose their own cbos we told them to disclose the assignment, its goals, and instructor contact information at the beginning of their service. but, we were not able to resolve all difficulties. in some cbos students worked with the coordinator of volunteers rather than direct supervisors, who were often unaware of details pertaining to students’ later volunteer activities. subsequently, these direct supervisors were unable to provide feedback beyond their initial students’ training experiences or they gave rooks, r.n., and rael, c.t. journal of the scholarship of teaching and learning, vol. 13, no. 2, may 2013. josotl.indiana.edu 96 table 5. challenges identified by faculty. student challenges 1. some students lacked compatible interests with their cbos. 2. there were differences in some students’ initial expectations versus their later realities regarding client interactions. 3. some students worked more than 20 hours a week, creating an additional stressor on their service learning experiences. 4. some students did administrative tasks not relevant to their slp. 5. some students’ initial naivete or idealism about doing applied research and helping others could have negatively changed by semester’s end. cbo challenges 6. some students worked with volunteer coordinators rather than direct supervisors, where the latter were often unaware of details pertaining to these students’ volunteer activities. other research challenges 7. we had a small sample size. 8. selection bias may exist in the students who took the course as an elective, prior to it becoming a requirement for undergraduate public health majors. 9. the survey we used measured attitudinal changes due to students’ slp over a shortterm, but we do not know if these changes were maintained in the long-term. 10. we did not adjust for social desirability. source: social determinants of health courses in 2008-2010. secondary reports of student performance. this phenomenon was more common in larger organizations. this research had other challenges. first, we had a small sample size from a single institution, despite collecting data from three courses over three years. our sample size was smaller than expected in part due to difficulties with matching preand post-test surveys. we asked students to write a three-digit number on their pre-test and write it in their notes to use for the later post-test and anonymous matching. however, many students did not keep or remember the correct numbers to match the surveys. second, selection bias was a concern in the type of students taking this elective course, prior to it becoming a requirement for undergraduate public health majors in our last course. because the course work load was high, it likely attracted above average, upper-class level students at the university (villanueva, hovinga, & cass, 2011). third, the survey design measured short-term attitudinal changes (gelmon et al., 2001); however, some research has shown long-term changes in student attitudes over years (fenzel & peyrot, 2005). finally, this study did not adjust for social desirability; thus, students’ responses could be artificially more positive than negative about their slp. despite students’ reduced, but still positive, interest in some aspects of community service over time and our prior limitations, we view service learning as a valuable learning experience for undergraduate public health students. by pairing service learning with undergraduate public health education students gain public health literacy through the application of social determinants to individual and community health (cashman & seifer, 2008). service learning emphasizes reciprocal learning between students, faculty, and cbos, as rooks, r.n., and rael, c.t. journal of the scholarship of teaching and learning, vol. 13, no. 2, may 2013. josotl.indiana.edu 97 well as reflection that connects practice to theory and critical thinking. it also develops citizenship skills to achieve social change. our undergraduate public health program, situated within a college of liberal arts and sciences, is an interdisciplinary, pre-professional degree that teaches critical analysis, information synthesis, and problem solving (cashman & seifer, 2008; riegelman, teitelbaum, & persily, 2002). this combination of knowledge and skills will better prepare undergraduate students to meet the challenges of and contribute to our nation’s health. in addition, service learning is valuable to undergraduate public health students because it provides practical field experience in a discipline where there are few internship opportunities for undergraduates. schools of public health have typically favored research institute structures for student development rather than the work-based learning models in professional schools, but even these opportunities are limited for undergraduates (madsen & turnbull, 2006; potter & eggleston, 2003). consequently, undergraduate students may lack exposure to important community issues. but, by creating service learning opportunities with cbos as part of undergraduate public health matriculation, students can gain access to organizations, populations, and problems of interest to their career development. based on our experiences with this course we realize that future improvements to students’ experiences means that we better prepare them for their slp through early and ongoing class discussions about their and cbos’ expectations. the following student comment continues to inspire us to improve students’ service learning experiences—“at the end of my time with [anonymous cbo] for this project i can rate it as one of, if not the, best experiences that i have had as a student so far. it expanded my knowledge of the subject matter in such a way that i’m afraid any future class experiences may fall a little short.” appendix 1. service learning project (slp) assignment learning objectives and grading rubric. 1) have knowledge of the social determinants of health perspective (sdoh), an interdisciplinary conceptual model for understanding health and healthcare problems. a list of possible issues to address within this model include: social inequalities in health by race, ethnicity, social class, gender, sexuality, disability, etc.; social stress; health behaviors; the life course (in utero to older adults); social and physical environments (social support, social networks, neighborhoods, housing, transportation, working conditions, etc.); doctor-patient relationships; access to and quality of healthcare; etc. inadequate adequate advanced student demonstrated poor knowledge of the sdoh perspective on exams, in journal and final paper discussions with no examples from the sdoh conceptual model, and no change in knowledge on pre to post-test surveys. student demonstrated good knowledge of the sdoh perspective on exams, in journal and final paper discussions with multiple examples from the sdoh conceptual model, and change in knowledge on pre to post-test surveys. student demonstrated excellent knowledge of the sdoh perspective on exams, in journal and final paper discussions with multiple examples from the sdoh conceptual model and high initial knowledge or significant change in knowledge from preto post test surveys. rooks, r.n., and rael, c.t. journal of the scholarship of teaching and learning, vol. 13, no. 2, may 2013. josotl.indiana.edu 98 2) be able to explain and provide examples of macroand micro-level contributions to health and healthcare problems and offer solutions to address them. inadequate adequate advanced student did not discuss or provide examples for macro & micro-level contributions to health and healthcare through his/her journal and final paper. student described macro & micro-level contributions to health and healthcare problems with multiple examples through his/her journal and final paper. student described macro & micro-level contributions to health and healthcare problems with multiple examples and discussed innovative solutions to health and healthcare problems through his/her journal and final paper. 3) demonstrate their awareness of social inequalities related to health and healthcare, whether in terms of culture, race, ethnicity, gender, socioeconomic status, sexual orientation, immigration status, age, etc., and understand their own biases and how these could affect health and healthcare problems. inadequate adequate advanced student demonstrated poor knowledge of diversity/ social inequality issues in journal and final paper discussions with no examples of how these affect health and healthcare and little knowledge of other cultures or own biases on preto posttest surveys. student demonstrated good knowledge of diversity/social inequality issues in journal and final paper discussions with multiple examples of how these affect health and healthcare and knowledge of other cultures or own biases on preto post-test surveys. student demonstrated excellent knowledge of diversity/social inequality issues in journal and final paper discussions with multiple examples of how these affect health and healthcare and knowledge of other cultures or own biases on preto post-test surveys. student demonstrated a deep understanding of structural barriers based on social inequalities in health and healthcare through their journals, final papers, site supervisor’s interview report, and having high initial knowledge or significant change in knowledge from preto post-test surveys. rooks, r.n., and rael, c.t. journal of the scholarship of teaching and learning, vol. 13, no. 2, may 2013. josotl.indiana.edu 99 4) demonstrate how their service learning projects can contribute to the organization’s growth, meet community needs, and enhance community members’ and students’ lives. inadequate adequate advanced student did not demonstrate both a descriptive and analytical knowledge of the organization’s or community’s need, how this project meaningfully affected or changed community members’ lives, and self-reflection on what was learned from this experience to enhance his/her education or career skills in his/her journal, final paper, and employer’s interview report. student demonstrated good descriptive and analytical knowledge of the organization’s or community’s need, how this project meaningfully affected or changed community members’ lives, and selfreflection on what was learned from this experience to enhance his/her education or career skills, with one or two examples in each of the prior categories from his/her journal, final paper, and site supervisor’s interview report. student demonstrated excellent descriptive and analytical knowledge of the organization’s or community’s need, how this project meaningfully affected or changed community members’ lives, and self reflection on what was learned from this experience to enhance his/her education or career skills, with multiple examples in each of the prior categories from his/her journal, final paper, and site supervisor’s interview report. references bach, r., & weinzimmer, j. (2011). exploring the benefits of community-based research in a sociology of sexualities course. teaching sociology, 39, 57-72. ballantine, j., & phelps, k. (2002). what can i do? applying classroom knowledge to service work, or doing practice sociology. sociological practice, 4, 41-52. blouin, d. d., & perry, e. m. (2009). whom does service learning really serve? communitybased organizations’ perspectives on service learning. teaching sociology, 37, 120-135. borkan, j. m. (2004). mixed methods studies: a foundation for primary care research. annals of family medicine, 2, 4-6. brown, j. d., bone, l., gillis, l., treherne, l., lindamood, k., & marsden, l. (2006). service learning to impact homelessness: the result of academic and community collaboration. public health reports, 121, 343-348. cashman, s. b., & seifer, s. d. (2008). service-learning: an integral part of undergraduate public health. american journal of preventive medicine, 35, 273–278. daiski, i. (2007). perspectives of homeless people on their health and health needs priorities. journal of advanced nursing, 58, 273–281. rooks, r.n., and rael, c.t. journal of the scholarship of teaching and learning, vol. 13, no. 2, may 2013. josotl.indiana.edu 100 fenzel, l. m., & peyrot, m. (2005). comparing college community participation and future service behaviors and attitudes. michigan journal of community service learning, 12, 23-31. gebbie, k., rosenstock, l., & hernandez, l. m. (eds). (2003). who will keep the public healthy? educating public health professionals for the 21st century. washington, d.c.: institute of medicine of the national academies, national academy press. gelmon, s. b., holland, b. a., driscoll, a., spring, a., & kerrigan, s. (2001). assessing service-learning and civic engagement: principles and techniques. campus compact. providence, ri: brown university. giles, d. e., & eyler, j. (1994). the theoretical roots of service-learning in john dewey: toward a theory of service-learning. michigan journal of community service learning, 1, 77-85. kenny, m. e., & gallagher, l. a. (2002). service learning: a history of systems. in kenny, m. e., simon, l. a. k., kiley-brabeck, k., & lerner, r. m. (eds), learning to serve: promoting civil society through service learning. boston, ma: kluwer academic publishers. kolb, d. a. (1984). experiential learning: experience as the source of learning and development. englewood cliffs, nj: prentice-hall, inc. loewenson, k. m., & hunt, r. j. (2011). transforming attitudes of nursing students: evaluating a service-learning experience. journal of nursing education, 50, 345-349. long, j. a., lee, r. s., federico, s., battaglia, c., wong, s., & earnest, m. (2011). developing leadership and advocacy skills in medical students through service learning. journal of public health management & practice, 17, 369-372. madsen, s. r., & turnbull, o. (2006). the academic service learning experiences of students in a compensation and benefits course. journal of management education, 30, 724-742. mobley, c. (2007). breaking ground: engaging undergraduates in social change through service learning. teaching sociology, 35, 125-137. ottenritter, n. w. (2004). service learning, social justice, and campus health. journal of american college health, 52, 189-191. parker-gwin, r., & mabry, j. b. (1998). service learning as pedagogy and civic education: comparing outcomes for three models. teaching sociology, 26, 276-291. peterson, b. a., & yockey, j. (2006). service-learning projects: meeting community needs. home health care management & practice, 18, 315-322. potter, m. a., & eggleston, m. m. (2003). supporting academic public health practice: a survey of organizational structures in public health schools. journal of public health management & practice, 9, 165-170. rooks, r.n., and rael, c.t. journal of the scholarship of teaching and learning, vol. 13, no. 2, may 2013. josotl.indiana.edu 101 reeb, r. n. (2010). service-learning in community action research: introduction to the special section. american journal of community psychology, 46, 413-417. riegelman, r. k., teitelbaum, j. b., & persily, n. a. (2002). public health degrees—not just for graduate students. public health reports, 117, 485-488. swallow, v., newton, j., & van lottum, c. (2003). research in brief: how to manage and display qualitative data using ‘framework’ and microsoft excel. journal of clinical nursing, 12, 610-612. toboada, a. (2011). privilege, power, and public health programs: a student perspective on deconstructing institutional racism in community service learning. journal of public health management practice, 17, 376-380. villanueva, a. m., hovinga, m. e., & cass, j. l. (2011). master of public health communitybased practicum: students’ and preceptors’ experiences. journal of public health management & practice, 17, 337-343. journal of the scholarship of teaching and learning, vol. 23, no. 2, june 2023, pp.1-14. doi: 10.14434/josotl.v23i2.33543 from antagonist to protagonist: shifting the stories to support gen z students eileen kogl camfield university of california at merced ecamfield@ucmerced.edu leslie bayers university of the pacific abstract: this article explores prevalent stories about “gen z” students that unintentionally undercut both their success and learner-centered pedagogies. the authors consider how those beliefs might be reframed to serve all learners more effectively. we also explore how the racial reckoning, health pandemics, social unrest, and additional compounded traumatic events of 2020 complicated stereotypes about college-aged youth and magnified the ever-present need for more inclusive, flexible, and compassionate teaching approaches. we now have an opportunity to build on the lessons of 2020 and expand the lenses through which we consider our students’ visible behaviors and invisible experiences. we offer a rationale for and concrete pathways toward crafting more empathetic and productive stories about gen z students, which in turn allow us to develop teaching and assessment strategies that better align with our student-centered missions. keywords: gen z, narratives, equity, compassion, pedagogy, assessment imagine you are at a party and the host comes up to you and says, “i’d like to introduce you to someone. just be prepared: they’re narcissistic, entitled, and may spend the conversation distracted, glancing at their phone. oh, and don’t say anything about it because they’re really fragile and might collapse if they feel criticized.” how excited would you be to meet that individual? now imagine that instead of a party, you are entering your classroom, and instead of an individual, you are facing a roomful of gen z students (those born after 1996). how enthusiastic do you feel about your new semester? put another way, how have the critiques of gen z, as privileged (twenge, 2018), unempathetic, electronic device-tethered (turkle, 2017), snowflakes (lukianoff & haidt, 2018) primed us to dislike and distrust our students? as educators and faculty developers, in this article we hope to offer a more complex and compassionate view of gen z students and suggest some pedagogical strategies that can promote mutual flourishing. we acknowledge that many of our colleagues do not hold negative biases against gen z, but the stereotypes are pervasive and show up on our campuses. most of us will have overheard colleagues ranting against “students these days.” this article offers strategies to reframe and respond to those perceptions. disrupting the stereotypes the points disgruntled instructors make about some of the problematic gen z attitudes and behaviors are not necessarily false, but they fail to capture the full picture. as chimamanda adichie (2009) points out in her much-viewed ted talk the danger of a single story, we all are composed of multiple stories: all of these stories make me who i am. but to insist on only these negative stories is to flatten my experience and to overlook the many other stories that formed me. the single about:blank camfield and bayers journal of the scholarship of teaching and learning, vol. 23, no. 2, june 2023. josotl.indiana.edu story creates stereotypes, and the problem with stereotypes is not that they are untrue, but that they are incomplete. they make one story become the only story. so, too, when we tell only a single (negative) story about gen z, we have an incomplete picture of our students, which means we struggle to develop the empathy (bayers & camfield, 2018) and authentic relationships that lead to deep learning (schwartz, 2019; felton & lambert, 2020). gen z is not monolithic, and a deeper dive into the attributes of this group reveals occasional contradictions as well as opportunities to re-design curriculum to better support today’s college students. after unpacking many of the stereotypes attributed to gen z, we will then turn to an exploration of how to leverage their assets to support growth. of privilege and passivity despite accusations of helicopter parenting causing a rise in student entitlement (allen, 2019), unnuanced gen z stories were called into question as covid-19 and social unrest spread across the us in 2020. higher education’s abrupt shift to remote teaching and learning shone a light on and magnified profound educational inequities. the pandemic and racial reckoning brought systemic injustices, including police brutality against unarmed black people and unequal access to medical care, to the fore. meanwhile, amid great uncertainty, compounded trauma, and any number of barriers, many of our students have shown up to learn. they have also helped create political change. in person and on social media, they turned out in record numbers to protest systemic racism and police violence against black americans, and then to vote: one study estimates that 52% of protestors at black lives matters demonstrations in the early months of the pandemic were between the ages of 18-29 (kaiser family foundation, 2020), while an estimated 52-55% of eligible voters under age thirty cast ballots in the 2020 election (tufts circle, 2020). our gen z students are not all retreating into privileged and protected pockets; not unlike young idealists from past counter-culture movements, they are an activist and connected generation. gen z is also more racially and ethnically diverse than previous generations and is pursuing higher-education at greater rates (parker & igielnik, 2020). at many institutions generally, and camfield’s in particular, students are increasingly anything-but entitled and privileged. often dubbed “the university of the future,” -where more than 80% of students are members of historically underrepresented groups, 60% are pell eligible, and 75% are the first in their families to attend college -the university of california at merced has received national recognitions: # 1 among public universities in outperforming expected graduation rates and # 5 among all universities in social mobility (uc merced “accolades,” 2021). this has not been achieved by insisting students assimilate and conform to our visions of the ‘typical’ college student. instead, uc merced has become a “student-ready college” by making room for those learners who have had to work two jobs and support family obligations, as well as study. these students have no family helicoptering around, monitoring their success. often it is quite the opposite: establishing boundaries with family and needing to defend time spent studying takes up cognitive and emotional bandwidth for many students. further, their demographics have brought them added challenges. hammond (2015) describes how “underserved english learners, poor students, and students of color routinely receive less instruction in higher order skills development than other students...as a result a disproportionate number of culturally and linguistically diverse students are dependent learners” (p. 12-13). this means they have come to rely on their teachers to define and pre-determine their learning, often through worksheets that call on the most basic of rote memorization or comprehension. when students arrive in college, many professors expect them to be self-regulating, independent learners. this disconnect between past schooling and college expectations creates an added academic burden for students 2 camfield and bayers journal of the scholarship of teaching and learning, vol. 23, no. 2, june 2023. josotl.indiana.edu already managing a whole host of other issues. to overcome this and foster learning independence, instructors may try to design opportunities for students to engage with “productive struggle that actually grows our brainpower” (p. 12-13). unfortunately, such struggle can also be overwhelming for some students, especially when accompanied by an “amygdala hijack” (p. 40) brought on by threats such as imposter syndrome or stereotype threat. therefore, as verschelden (2017) states, we may need to spend more time helping our students manage these cognitive loads, rather than dismissing their inattention or late assignments as the byproducts of entitlement and privilege. how should we design all courses to foster this growth and allow for these demands? of snowflakes gannon (2020) eloquently challenges the accusation that gen-z-ers are ‘snowflakes’—fragile and overly-sensitive—, calling “the argument that today’s students are somehow less capable of dealing with controversial ideas or being intellectually challenged... fundamentally disingenuous” (p. 126). he further elaborates a compelling counter-narrative: far from being a generation of entitled snowflakes, today’s college students are under siege. they have less funding, less support, learn in more dysfunctional institutions, and live in an environment that is more fractured and polarized than ever. they work more hours at more jobs than any previous generation of students, deal with more issues related to anxiety and mental health than any of their forebears, and face a postgraduate economy so bleak that the baby boom generation is in full-on denial of its very existence. all this, and they are mocked, by generations that had it twice as good, about how it’s all their fault (p. 126). unfortunately, this economic uncertainty is compounded by capitalist pressure for a college education to reap primarily monetary rewards (tretina, 2021), which threatens to reduce educational interactions to the transactional. how, then, might we create a college curriculum that acknowledges future uncertainty, builds tolerance for ambiguity, and emphasizes transformation over transaction? of electronics addiction despite compelling research on social media addition, sometimes called idisorder (mitus, 2021), disrupting the myth of the distracted, social-media-saturated gen z-er necessitates unpacking a couple of biases and blindspots. the first is the fundamental attribution error instructors may engage in where an interior monologue might sound like this: “when i check my e-mail during a zoom meeting, i am being efficient with my time and am paying enough attention to the meeting to get the gist of it. whereas when my students chat over discord during my class, they are disrespectful and miss out on important elements of my lesson.” recognizing ways attention works for all humans can help rectify this fallacy. college instructors may also carry outdated knowledge about today’s internet users. for example, observations about “facebook depression” (o’keeffe, clarke-pearson & ccm, 2011) are over a decade old. to think today’s teens are as vulnerable to those effects might be inaccurate, especially when even at that time the conclusions were disputed with counterclaims that social media actually be useful in combating stigma surrounding mental illness (moreno et al, 2011). secondly, we might need to look at how our students are engaging with the internet and social media, not just noting that they are doing so. for example, our students are far more likely to verify the factual basis for a claim or pursue a topic of idle curiosity using a quick internet search than previous generations were who had to rely on the library-housed, multi-volume encyclopedia britannica. 3 camfield and bayers journal of the scholarship of teaching and learning, vol. 23, no. 2, june 2023. josotl.indiana.edu we can expect that some students may use google superficially or trust on-line sources too implicitly. it is our job as educators to teach critical digital literacy, not to assume they come to college with it already intact. we might also note that concerns over potential social problems caused by the pursuits of youth started long before electronic entertainment came around (best, 1998). moreover, gen z itself appears to be somewhat divided on the impact of social media on their lives. in a recent pew survey (parker & igielnik, 2020), roughly one-third says the effect is mostly positive, about one-quarter say it has been mostly negative, and just under half say it has been neutral. those who see it positively tend to say it helps them feel connected with friends and family, whereas those who say the effect is negative worry about bullying and rumor spreading. this, again, illustrates the danger of perceiving gen z as monolithic and incapable of awareness and self-analysis. moreover, the 20-year span gen z has been around has also witnessed a re/evolution in the use of the internet as a base for creative pursuits. far from passive users, today’s college students are making art, composing new music, performing original poetry, and hosting podcasts at never-beforeseen rates. a survey (rapp, 2019) of over 1,200 gen z-ers revealed more than half engaged in creative pursuits through platforms like youtube, snapchat, and tiktok; however, this did not mean that their off-line lives lacked artistic engagement: 77% of the respondents reported that they spend their free time offline drawing, journaling, or playing an instrument. this has also translated into an interest in homemade crafts and do-it-yourself interior design tips, which are often freely shared on-line. how can we harness the potential of media engagement while also fostering digital self-regulation, civility, and information literacy? of cancel culture and conformity despite legitimate concerns over cancel culture, “an intolerance of opposing views, a vogue for public shaming and ostracism, and the tendency to dissolve complex policy issues in a blinding moral certainty” (letter on justice and open debate, 2020), gen z also exhibits a great deal of creativity and acceptance of ambiguity. far from the merely superficial or decorative, gen z creativity leverages the wealth of information available on the internet to ‘crowd source’ and to perfect the ‘mash-up,’ a “melting of borders, both aesthetic and cultural” (rapp, 2019). additionally, they report that while such borrowing, mixing, and complication allows for greater expression, it also tends to be grounded in an individual’s personal truth. increasingly, this truth is built around: social causes like lgbtq issues, body positivity, and mental health… the internet can be a judgmental place, but gen zers are unabashedly using their reach to create communities and dialogue around their fluidity. they’re redefining gender identities and breaking beauty norms, going beyond tutorials and selfies to establish more multifaceted personas (p. 67). if our students are making videos to promote sustainability and/or body positivity, how might we leverage some of what is happening organically in our students’ spare time and harness that creativity in our classrooms? gen z-ers also seem more comfortable with ambiguity, prefer to blur boundaries, and embrace change (parker & igielnik, 2020). for example, many do not identify with a political party but rather support specific causes and engage in grassroots efforts to promote them. they are more apt to self-monitor, suspend judgement, and stand-up. according to a study published by the annie e. casey foundation (aecf, 2021), gen z members tend to be more open-minded, liberal-leaning and actively engaged in advocating for the fair and equal treatment of others. their strong presence 4 camfield and bayers journal of the scholarship of teaching and learning, vol. 23, no. 2, june 2023. josotl.indiana.edu at protests and the election polls in 2020 demonstrated this propensity. true, this tendency can sometimes be taken to a “cancel culture” extreme, but it also signals an unwillingness to passively accept the status quo – what in our nation’s founders was praised as a heroic revolutionary spirit. it also points to an educational opportunity: how can we work with our students to identify effective ways of challenging authority/dominant norms/objectionable behavior and stimulating discourse? revealing and re-prioritizing hidden narratives along with the incomplete narratives discussed previously, an insistence on the most unflattering and negative interpretations of student behaviors risks ‘blaming the victim,’ masks more nuanced realities, and impedes more complex understanding of our students’ experiences. even if we accept the “tethered (to electronic devices) and privileged” gen z labels, we may miss out on two others. as discussed by camfield, moore, & allen (2020), gen z students also labor under the ‘pressure to be perfect,’ to perform for the approval of parents, coaches, teachers, and peers. this claim is borne out by studies that show a linear increase in perfectionism for gen z when compared to previous generations of college students (curran & hill, 2017). they also feel profoundly ‘unsafe:’ from school shootings, to uncertain economic forecasts, not to mention global pandemics; it is no surprise we are seeing epidemic levels of depression and anxiety (twenge, 2018), arguably not entirely due to dependence on their digital devices. sadly, many of the stereotypes in circulation about gen z have at root the same impact on students: each undermine resilience by either removing any adversity from students’ paths or by creating unbearable burdens. neither allows students to develop learning muscles necessary for perseverant engagement with tough issues. camfield, moore, and allen (2020) asked: how might we reframe our conceptualization of ‘rigor’ to create just the right amount of difficulty so that students develop the skills to ‘bounce back’ in the face of academic challenge? it also suggested that such a reconceptualization requires “critical empathy” from instructors where we reconsider “our perception of what students ‘should’ be retaining from certain lessons, providing us with the opportunity to revisit lessons, revise our curriculum for effectiveness, and model a mode of continuous learning that brings collaborative discovery back to our classrooms” (p. 130). now we add to that story. as co-authors, in our teaching and in presenting on our past research, we have had the opportunity to talk with many gen z students. not only have they expressed appreciation for our capacity to see the fullness and complexity of their identities, but they have also added to our insights. one point often made is that, perhaps because of the constant self-comparison encouraged through social media or because of being told they are ‘privileged and entitled,’ some students do not feel they have the right to own their own stories. as one young woman put it, “i keep seeing these stories about people starving in the third world and about others being forced into sex trafficking. somehow that makes my own problems with managing my schedule seem trivial.” on one hand, such a statement might be lauded as valuable perspective-taking; on the other hand, minimizing our own experiences is an unproductive form of self-gaslighting. how might we help our students contextualize their knowledge and feelings, balancing what they learn about the experiences of others with their own? disrupt inherited pedagogical narratives unfortunately, instead of enhancing our students’ positive identities, many of our inherited pedagogical practices are shaped by motifs of subjugation and radical individualism. more specifically, our campuses are still fundamentally unwelcoming to women and students of color. this is not surprising as academic institutions in this country are rooted in white-centered and patriarchal structures (hill et al. 2020). the norms of hierarchy (e.g., uncritically deferring to the sage on the stage) 5 camfield and bayers journal of the scholarship of teaching and learning, vol. 23, no. 2, june 2023. josotl.indiana.edu and expectations about productivity and professionalism (e.g., that academic work is separable from one’s personal obligations and unaffected by emotions) that we often replicate in our classrooms reinforce this oppression. additionally, inherited instructional and assessment practices that are teacher-centric, that promote competition among students, and that use punishment as incentive recast and maintain persistent patterns of domination and marginalization. we have seen this recently as institutions have addressed concerns about student cheating in remote environments by doublingdown on surveillance technology, which tells students that they are not to be trusted, further undermines their self-efficacy and resilience, and maintains a dependency model of education. myths about alternative approaches being less “rigorous” (perhaps code for less white and male) impede change. finally, over-applications of the work of scholars like carol dweck (2007) and angela duckworth (2018) can do the same: while the frameworks of growth mindset and grit can potentiate self-efficacy and learning, neither mindset nor grit alone can overcome systemic oppression, or a fundamental lack of access to resources and opportunity. no amount of resilience can make up for historical educational inequities, not to mention immediate barriers to learning like no access to the internet, an empty gas tank , or too little money for textbooks. embrace relational narratives another motif that has emerged in our recent work is the hunger of gen z for meaningful relationships. some have expressed this as feeling like they “have to ‘curate’ their lives on social media and ‘watch their every word’ for fear of ‘cancel culture.’” however, others simply say they want to connect with other people in “real” and “deep” ways. here, twenge’s (2018) work and her observation that “igen’ers spend less time interacting with their peers face-to-face than any previous generation” (p. 71) and her conclusions about a rise in loneliness resonates. however, she attributes this almost exclusively to internet use, not perhaps to overscheduling or other conflicting demands on time. a different picture of reality, as revealed by the recent pandemic and the shift to emergency remote instruction, is that students crave face-to-face connection with each other. for example, in the uc merced new student survey from fall 2020, despite the fact that 60% of the 1,019 first-year students completing the survey said they had a strong sense of belonging to the campus community, 54% identified “missing connection with family or friends,” with 60% also noting “difficulty making new friends,” as barriers to their schoolwork or academic success (uc merced irds, 2021). campuses can help students build more meaningful connections if they prioritize relationship-building through activities like collaborative learning and team science. we also know relationships with instructors matter as well. camfield’s 2016 study of developmental writing and writers proposed the theory of mediated-efficacy which described the role of a teacher as that of an intermediary that helps students “reconcile negative self-beliefs developed in the past with newly-forged positive identities that could impact their future performance” (p. 9). it acknowledges instructor power and suggests we use this as a force for good, to help “students dismantle learned helplessness, dispute pessimism, and develop optimism” (p. 10). in this context: specific course content or skills become the tools of that mediation. useful yes, but not unlike the utility of knowing how to use a hammer, only truly valuable when used to build something. mediated-efficacy requires a balance between helping students wield tools on their own and creating the environment in which they believe they have something worthwhile to construct (p. 10). felton and lambert (2020) describe relationships as the “beating heart of the undergraduate experience” (p. 1), and while they vaunt their importance, they simultaneously caution that “they 6 camfield and bayers journal of the scholarship of teaching and learning, vol. 23, no. 2, june 2023. josotl.indiana.edu should not occur by happenstance or only for some students” (p. 1). their robust study, involving 400 interviews with students, faculty and staff at 29 higher-education institutions across the country, lead them to conclude “the future is relationship-rich” (p. 147). relationships can be transformative, positively shaping student self-esteem, sense of meaning, and world view. changing the narratives harkening back to the previously-cited ted talk, adichie (2009) notes that “stories matter. many stories matter. stories have been used to dispossess and to malign, but stories can also be used to empower and to humanize. stories can break the dignity of a people, but stories can also repair that broken dignity.” figuring out strategies to humanize our perceptions of gen z students is imperative. to do so we must build our curriculum and develop our pedagogical practice around narratives that help students manage cognitive load, utilize their abundant creativity, promote tolerance for ambiguity, experience academic rigor that builds resilience, activate critical empathy, own their own stories, and build relationships. while no one would have asked for these circumstances, the events of 2020 did create greater potential for higher education to think outside of the box, or even to re-examine why that ‘box’ exists – where did it come from? who is being asked to perform contortions to fit into it? as educators had to abruptly shift to teaching remotely amid compounded traumatic events and further visibility of systemic injustice, we witnessed an expansion of empathy and creativity: instructors tirelessly led with care to stay connected with students and sustain learning. many radically reimagined teaching and assessment strategies to support students as everyone in the classroom navigated distress and uncertainty. we now have an opportunity to build on the lessons of 2020 and expand the lenses through which we consider our students’ visible behaviors and invisible experiences. shifting perspective or point-of-view we can start by shifting our perspective or points-of-view as instructors. how we imagine our students, even before meeting them, matters. if we picture gen z students as a monolithic group of privileged, protected, and distracted young adults, we will be more apt to misjudge and misinterpret their actions. if we replace that story with one of a remarkably diverse, creative, engaged, and connected generation, we will be more open to seeing the deeper dimensions of their work and behaviors, as well as identifying our own blindspots in perspective. once we are in the classroom together, we can assume the best of our students and interpret their actions with empathy (bayers & camfield, 2018). at workshops, we often engage faculty in an exercise designed to reveal how quickly instructors can leap to negative judgments on the basis of incomplete information. we invite participants to write down a common behavior they see manifest in their gen z college students, cite a stereotypical explanation of that behavior, then imagine a more generous interpretation of what might be going on for the student under the surface. to cite one example, “students don’t do the reading” often comes up as a common problem. initial and oversimplified interpretations might point to learner laziness, disinterest, or attention deficits, among other possibilities, as the cause. however, instead of judging or blaming the student, a deeper consideration can reveal a lack of instructor transparency around why and exactly how to approach a reading assignment or can uncover a lack of alignment between readings and course goals or in-class activities. such insights lead to productive pedagogical interventions. to the extent being unprepared for class is the student’s fault, replacing the dismissive assumption of laziness with a more generous interpretation -lack of access to the reading material, or competing demands (other classes, work, and family, for example) -activates trust and 7 camfield and bayers journal of the scholarship of teaching and learning, vol. 23, no. 2, june 2023. josotl.indiana.edu builds relationships. if we can compassionately imagine the barriers to our gen z students’ learning, we can design better pathways to success. alongside instructor empathy and compassion for gen z student needs must be an active commitment to fostering hope in those students who may feel overwhelmed or jaded. equity-minded educators should not minimize those things many of our gen z students experience and oppose: oppression and social injustice. an everincreasing number of our students live these realities. so, we agree with hammond (2015) that “instead [of amplifying a sense of victimization], we should focus on highlighting a community’s resiliency and vision for social change” (p. 92). redesigning the setting begin with cultural frameworks. given that higher education was historically written by and to further empower white men, a substantial portion of our gen z students do not see themselves or feel safe and supported within traditional academic narratives, even if they are unaware of the source of this unease (and may unfairly blame themselves for their own discomfort). creating welcoming and affirming spaces for all of our students will require educators to question not only the foundations of knowledge and standard practice in each and every one of our fields, but also our own explicit and implicit biases. we need to rewrite our teaching stories to include our current students as central figures and become vigilant in interrogating if and how our instructional assumptions, policies, practices, and content draw from, affirm, and speak to the various identities and cultural frameworks our students hold. as we elaborate below, strategies like anti-racist pedagogy, universal design for learning, and other approaches grounded in collaborative learning and transparent teaching support this rewrite. antiracist pedagogy requires that we work to see, make visible, and actively counter the ways in which our academic practices and curricula are built upon primarily eurocentric voices and views; this includes interrogating disciplinary foundations that have come to be seen as inherent and devoid of a particular cultural perspective (blakeney, 2011). in the classroom, antiracist pedagogy requires us to question not just what we teach, but also how we do so (we explore equitable, power-sharing practices below). those among us who hold white privilege must commit to the lifelong work of antiracism on personal levels in order to engage it systematically in our pedagogy. in me and white supremacy, layla f. saad (2020) expresses the societal impact that this inner work can have: “since systems and institutions are created and held in place by many individual people, it is my hope that as more people do the personal inner work here, there will be a ripple effect of actionable change of how white supremacy is upheld out there” (p. 12). “out there” includes in our classrooms, offices, and every other aspect of campus culture. as we expand our own knowledge and resiliency for engaging in uncomfortable yet necessary conversations about race and power, we can model for and better support our gen z students in doing the same. such conversations necessitate careful course and classroom architecture. alongside establishing cultural frameworks, build enhanced access. the mindset fostered by universal design for learning (udl) can also help us better imagine, prepare for, and support our highly diverse gen z students. udl acknowledges that all learners come to our classes with varying strategies for and barriers to learning—differences that may or may not be visible—and prompts us to build in flexibility as a design and operating principle. such adaptability can help us better serve not only the estimated 20% of gen z students with learning disabilities (nces, 2019), but also those from diverse linguistic, cultural, and academic backgrounds, those juggling competing demands in their lives, and others marginalized by traditional academic structures and practices. udl guidelines, outlined by cast (2018), encourage instructors to provide multiple means of: 1) engagement, or ways of stimulating learner interest and motivation; 2) representation, or ways of presenting information 8 camfield and bayers journal of the scholarship of teaching and learning, vol. 23, no. 2, june 2023. josotl.indiana.edu and content; and, 3) action and expression, or ways of allowing learners to demonstrate what they know. we encourage colleagues to consider their go-to course design, instructional, and assessment approaches and, in each case, ask: “who is best served by this approach? who might be excluded by it? what barriers to learning might i unintentionally be creating with this approach? what additional options might i build-in to account for learner variability?” for example, if my assessment is only based on written work, students with dysgraphia or dyslexia will struggle to demonstrate what they have learned. providing alternative ways for students to demonstrate their learning (e.g., through voice recordings), aligned with the same learning outcomes, will offer more equitable access. if my go-to instructional practice is small-group work, introverts may struggle to participate. assigning specific roles or tasks within those groups and balancing team-based learning with independent work makes space for all students to engage. in addition to empowering all students with more pathways to, though, and beyond the achievement of course learning outcomes, the flexible mindset fostered by udl can help us better adapt as we continue to navigate the impact of the global pandemic and further disruptions to teaching and learning. all students thrive in a welcoming and collaborative classroom environment. learning invitations and gestures that “summon students cordially” into our learning spaces (purkey, 1992) should ground our courses. yet as bali et al. (2018) note, it is not enough to “assume that saying ‘welcome’ will mean people feel welcome;” rather, we need to craft and support “intentionally equitable hospitality.” before our classes begin, we can set the tone for engagement through an asynchronous opportunity to interact, such as quick video introductions or short narratives that allow them to begin telling their stories and making connections. this will lay the foundation to grow and sustain community during the semester. equity unbound (n.d.) has curated a rich collection of structured synchronous and asynchronous activities, each with detailed instructions, for building and sustaining “intentionally equitable hospitality.” far from being mere “ice breakers,” activities that allow us to get to know our gen z students and that create multiple and meaningful opportunities for them to interact with one another will allow us to not only connect our course content to their interests and aspirations, but also affirm their identities. to reframe our own stories about gen z students, we need to hear theirs. something as simple as a daily check-in helps students feel seen and valued and can create ripple effects, such as more robust participation in instructors’ office hours. inviting students to help shape discourse agreements also sets the tone for community and collaboration while affirming existing gen z values of advocating for the fair and equal treatment of others (aecf, 2019). when students become active agents in shaping the learning environment, they become more invested in and accountable to the class community (brookfield & preskill 2005). this could have an added benefit of disrupting a view of learning as individualistic and transactional. the syllabi that house these agreements, alongside additional expectations and roadmaps to course success, can also be characterized by welcoming, collaborative, and supportive language; for example, we can replace cold, dry, and punitive syllabi statements with invitations and strategies to learn, connect, and succeed. as an illustration, statements about university honor codes are often inherited from a boilerplate document that assumes the worst and offers little educational support: students must observe the university honor code. this professor strictly enforces this code and its stipulated penalties. if you have any questions about plagiarism and other departures from the right way of conducting yourself in academic situations, speak with me. ignorance of policy is not an excuse to violate policy. (camfield syllabus: 2013) how different a more intentionally designed and educationally focused statement can sound: plagiarism is an issue that is as complicated as linguistic expression is nuanced. for our 9 camfield and bayers journal of the scholarship of teaching and learning, vol. 23, no. 2, june 2023. josotl.indiana.edu purposes, plagiarism entails representing another’s work as your own. note that plagiarism includes: • submitting work that is done in part or wholly by someone else (or done by you for a different class/context) • paraphrasing or summarizing any source without referencing it • copying any source without using quotation marks or block indentation. in sum, if you submit your own work with all outside sources or ideas carefully and correctly documented, you will have maintained academic honesty. remember that writing is a thinking process, so you should engage with resources as though you were in a conversation. the integrity of your ideas rests on maintaining scholarly habits while in this dialogue with experts; ask questions and research actively with detailed notes. (camfield syllabus: 2021) in addition to sounding harsh, the first statement also exemplifies the hidden curriculum, which is a product of generational-privilege; it assumes student knowledge about “the right way” of conducting oneself in academic situations. such exclusionary assumptions can add to students’ cognitive loads (verschelden, 2017) and activate anxiety, whereas more transparent and inviting statements can ease and welcome. co-authoring the design of learning activities when possible—for example, by soliciting student input on content or assignments—can simultaneously support class community and learner autonomy by shifting traditional classroom roles from teacherto learning-centered dynamics. such activities need not be time-consuming and may result in better student understanding of expectations. for example, students can analyze an assignment prompt or exemplary model to identify salient components and steps necessary for successful completion to co-generate the rubric by which they will subsequently be evaluated. this can offer students’ a sense of both agency and connectedness. it can also counter the sense of helplessness and isolation fostered by the pandemic and recent social upheaval. a place to begin is with discussion of course outcomes: we can invite students to consider how the class aligns with their own learning, professional, and life goals. we can explore the “emergent outcomes” students discover as the course unfolds (stommel 2017). when we provide these kinds of opportunities for students to weigh in on and take stock of how our teaching, class activities, and their approaches to learning support their values, we make space for them to better ‘own their own stories’ and see their education as a mechanism for fleshing out their autobiographies. we can further empower gen z students by transparently sharing scripts for how to succeed in course activities, from class interactions to assignments and assessments. transparent teaching has been demonstrated to boost academic confidence, motivation, learning strategies, skill transfer, and success for all students, and to be particularly supportive of students who are the first in their families to go to college and/or from groups historically excluded from college campuses (tilt higher ed). when we explicitly and straightforwardly clarify the purpose, task, and criteria for success in our assignment instructions, we decode the unspoken rules for how to succeed in academia. helping our gen z students understand the meaning, merit, and relevance of coursework to their academic success and future careers can generate buy-in, while meaningful feedback can help them identify concrete ways to improve their performance and overall learning. transparent teaching provides a scaffolding that allows students to explore and own their identities as scholars, or become active authors of their learning stories. hanstedt (2018) describes this negotiation as key to developing the skills to confront “wicked,” 21st-century problems: 10 camfield and bayers journal of the scholarship of teaching and learning, vol. 23, no. 2, june 2023. josotl.indiana.edu authority in this context implies authorship, the ability to write and rewrite, shape, and create. at the same time, this ability comes from something or someone. authority is granted, given, and earned. the content and skills students acquire during their years in college are crucial; they are part of what creates a sense of authority in students (p. 5). this process entails a shifting of traditional and transactional classroom roles and expectations. our classes can give students spaces to practice assuming the kind of authority hanstedt describes, sometimes getting it wrong, and trying again. to cite one example, identifying and even scripting roles in small group work can allow students to intentionally listen to and support one another, to try on different conversational gestures, to build their academic confidence, and to find their scholarly voices. structured collaboration lays the groundwork for organic interaction, civil discourse, and creative problem solving. when students are clear on what the task is and know that everyone will not only be expected to but also have the opportunity to participate equitably, their anxiety is reduced, and they can engage more productively (eddy & hogan 2017; kernahan 2019). transparency and structure, from scaffolding frameworks and roles for activities to clear and accessible roadmaps through our courses and assignments, can also provide a crucial sense of safety for students returning to campus following the traumatic events of 2020 and beyond. it has always been important to reduce cognitive load and keep our students’ focus on learning through clear course pathways. the extraordinary events our gen z students have witnessed and experienced in their recent lives--a global pandemic, repeated police brutality against black americans, extreme political divisiveness, environmental catastrophes (hurricanes, wildfires, floods, and other disasters), a record number of mass shootings, and more--heighten the need for clarity and structure in our classrooms. as we co-create this new narrative, instructors must also become more mindful of how we speak to and with students, as language can convey either power-over or power-with others. this does not mean a sacrifice of rigor. hammond (2014) describes how teachers as “warm demanders” (p. 97) find the right balance between a firm insistence on academic standards and a nurturance that allows students to feel safe to approach with questions. engaging in conversations that communicate our support as well as provide an entrée into academia can also activate student creativity. if we are open to hearing students’ suggestions, we can harness some of that gen z inventiveness. one recent example of this occurred during the shift to emergency remote instruction in spring 2020 in camfield’s first-year seminar course intended to introduce students to life at a research university. as such, it had a required component that was originally assigned as a group-written research paper with an accompanying in-class oral presentation. the pandemic made this impossible, so camfield asked her students for ideas on alternatives. one student suggested they each compose and record a researchbased podcast to be uploaded to the course discussion board. not only did these podcasts fulfill all of the learning outcomes of the traditional assignment, but the quality of the production was also far higher than the previous in-class presentations. moreover, students expressed far more enjoyment and satisfaction with the assignment, as well as feelings of having gained relevant or ‘real world’ skills. such a reimagined assignment points to the value of tapping into gen z creativity and devising new ways of assessing their learning. while the previous example demonstrates the way we can change how we approach assessment to better support learning, another component of the dialogue we develop with students has to do with the language of our feedback. camfield and bayers (2019) describe traditional assessment protocols as “mindless,” in that they can be overly fixated on perceptions of objectivity and scientific validity and rely on isolated, summative, and high-stakes tests, making students feel dehumanized and disconnected from the learning process. we have called for more “dialectical evaluative practices that invite students into, guide students through, and take students beyond learning in the classroom in ways that honor their agency as whole persons” (p. 123). this can involve 11 camfield and bayers journal of the scholarship of teaching and learning, vol. 23, no. 2, june 2023. josotl.indiana.edu figuring out ways, like student self-annotation, to access invisible learning (camfield, moore, & allen, 2020) or substituting punitive reading quizzes, which reinforce dependent learning and instructor power, with more creative approaches, like the “one-pager,” where a reading response is built around a student-selected image that they feel typifies the dominant theme or take-away from the assigned reading. subsequent class discussion can then be structured around the sharing of these images. not only does this assessment approach encourage completion of assigned work through peer-based accountability, but it also fosters visual literacy and other 21st-century skills (minor, 2021). in addition to the examples previously shared, other forms of more mindful assessment might include: outcomes framed around integrated concepts; more emphasis on frequent, lower-stakes assessments (versus less frequent, high-stakes tests); asset(instead of deficit-) based evaluation methodologies and rubrics; an emphasis on meaningful, targeted feedback over grades; selfand peerassessment; and opportunities for post-assessment reflection and metacognition. each of these approaches can counter the grade-driven, competitive culture our gen z students were raised in and help spark significant, creative, collaborative, and effortful learning that honors error and even discomfort as part of the process. the global pandemic prompted many instructors to experiment with these and other alternatives to traditional assessment practices. our hope is that this rewriting of old assessment narratives continues to evolve as we move forward. conclusion by inviting educators to shift their perspective on how they imagine their students (even before meeting them), to redesign the settings in their classroom, to help students craft scripts for success, to build dialogues with them that flatten hierarchy and foster learning relationships, we hope to revise the narrative of how we teach gen z students. with this new framing, students can better manage cognitive load, ambiguity, and the anxiety brought about by living in these complex times. we have also suggested ways to leverage gen z’s assets to help them develop the resilience to withstand whatever the future may bring. yet in the end, much as we have attempted to disrupt some of the stereotypes and “repair broken dignity” (adichie, 2009), we do not mean to imply we have told the whole story. instead, we hope to encourage instructors to enter their classrooms more enthusiastic about their teaching, empathetic toward gen z, and inspired to ask: who are you? – with the reminder that a story does not tell itself and is highly influenced by how it is prompted. references adichie, c. n., (2009). ted talks: chimamanda adichie – the danger of a single story. films media group. https://www.ted.com/talks/chimamanda_ngozi_adichie_the_danger_of_a_single_story/tra nscript?language=en allen, v. (26 march 2019). controlling 'helicopter parents' are to blame for snowflake 'gen-z' youngsters struggling at university, psychologists say. the daily mail. https://www.dailymail.co.uk/sciencetech/article-6852541/helicopter-parents-blamesnowflake-gen-z-youngsters-struggling-university.html annie e. casey foundation (aecf) (2021, may 11). social issues that matter to gen z. https://www.aecf.org/blog/generation-z-social-issues/ ash, a., hill, r., risdon, s.n., & jun, a. (2020). anti-racism in higher education: a model for change. race and pedagogy journal, 4(3), 1-35. https://soundideas.pugetsound.edu/cgi/viewcontent.cgi?article=1071&context=rpj 12 about:blank about:blank about:blank about:blank camfield and bayers journal of the scholarship of teaching and learning, vol. 23, no. 2, june 2023. josotl.indiana.edu bali, m., caines, a., hogue, r.j., dewaard, h.j., & friedrich, c. (2019, may). “intentionally equitable hospitality in hybrid video dialogue: the context of virtually connecting.” elearn magazine. https://elearnmag.acm.org/archive.cfm?aid=3331173 bayers, l. & camfield, e.k. (2018, march). student shaming and the need for academic empathy. hybrid pedagogy. https://hybridpedagogy.org/student-shaming-academic-empathy/ best, j. (1998). too much fun: toys as social problems and the interpretation of culture. symbolic interaction, 21(2), 197-212. https://www.jstor.org/stable/10.1525/si.1998.21.2.1 blakeney, a.m. (2005). antiracist pedagogy: definition, theory, purpose and professional development. journal of curriculum and pedagogy, 2 (1), 119-132. camfield, e k. (2016). mediated-efficacy: hope for “helpless” writers. journal of developmental education, 39(3), 2-11. https://files.eric.ed.gov/fulltext/ej1130186.pdf camfield, e.k. & bayers, l (2019). mindful assessment in support of student learning. the journal of contemplative inquiry. 6(1), 121-144. camfield, e. k., moore, j. d., & allen, j. (2020). sifting through gen z stereotypes: using critical empathy to assess invisible learning. journal of higher education theory and practice, 20(6). doi: https://doi.org/10.33423/jhetp.v20i6 https://articlegateway.com/index.php/jhetp/article/view/3136/2981 cast (2018). universal design for learning guidelines. https://udlguidelines.cast.org/?utm_source=castsite&lutm_medium=web&utm_campaign= none&utm_content=aboutudl. curran, t. & hill, a. p. ((2017). perfectionism is increasing over time: a meta-analysis of birth cohort differences from 1989-2016. american psychological association psychological bulletin 145(4), 410-429. http://dx.doi.org/10.1037/bul0000138 eddy, s. l. & hogan, k. a. (2017). getting under the hood: how and for whom does increasing course structure work? cbe--life sciences education., 13(3). doi: https://www.lifescied.org/doi/full/10.1187/cbe.14-03-0050 equity unbound. (n.d.) community building activities. retrieved may 18, 2021, from https://onehe.org/equity-unbound/ lukianoff, g. & haidt, j. (2019) the coddling of the american mind: how good intentions and bad ideas are setting up a generation for failure. penguin. felton, p. & lambert, l. (2020). relationship-rich education: how human connections drive success in college. johns hopkins up. gannon, k. (2020). radical hope: a teaching manifesto. west virginia up. hammond, z. (2014) culturally responsive teaching and the brain: promoting authentic engagement and rigor among culturally and linguistically diverse students. corwin press. hanstedt, p. (2018). creating wicked students: designing courses for a complex world. stylus publishing. kernahan, c. (2019). teaching about race and racism in the college classroom. west virginia up. kaiser family foundation. health tracking poll (june 2020). https://www.kff.org/racial-equity-andhealth-policy/report/kff-health-tracking-poll-june-2020/ letter on justice and open debate (7 july 2020). harper’s magazine. https://harpers.org/a-letter-onjustice-and-open-debate/. minor, d. (2021, january 1). visual literacy is critical for 21st century learners. national council of teachers of english literacy blog. https://ncte.org/blog/2021/01/visual-literacy-critical21st-century-learners/ mitus, j. s. (14 january 2021). idisorder: the emerging trend of tech addiction among generation z. hofstra university department of counseling and mental health online workshop. https://youtu.be/r_-pg1dn7ce 13 about:blank about:blank about:blank about:blank about:blank about:blank about:blank about:blank about:blank about:blank about:blank about:blank about:blank about:blank about:blank about:blank about:blank about:blank about:blank camfield and bayers journal of the scholarship of teaching and learning, vol. 23, no. 2, june 2023. josotl.indiana.edu moreno, m.a., jelenchick, l. a. , egan, k. g., cox, e., young, h., gannon, k. e., & becker, t. (2011). feeling bad on facebook: depression disclosures by college students on a social networking site. journal of depression and anxiety, 28(6): 447–455. doi:10.1002/da.20805. national center for education statistics (nces) (2019). students with disabilities. https://nces.ed.gov/fastfacts/display.asp?id=60 o’keeffe, g. s., clarke-pearson, k., and council on communications and media (2011). clinical report: the impact of social media on children, adolescents. in pediatrics, american academy of pediatrics. doi: 10.1542/peds.2011-0054 parker. k. & igielnik, r. (2020). on the cusp of adulthood and facing and uncertain future: what we know about gen z so far. pew research center: social and demographic trends. https://www.pewresearch.org/social-trends/2020/05/14/on-the-cusp-of-adulthood-andfacing-an-uncertain-future-what-we-know-about-gen-z-so-far-2/ purkey, w. w. (1992). an introduction to invitational theory. journal of invitational theory and practice, 1(1), 5-15. rapp, j. (2019). into z future: meet generation z, the next generation of super creatives. j. walter thompson analytics. https://gertkoot.files.wordpress.com/2019/06/into-z-future_understanding-genz_the-next-generation-of-super-creatives-1.pdf saad, l.f. (2020). me and white supremacy: combat racism, change the world, and become a good ancestor: sourcebooks. schwartz, h. (2019). connected teaching: relationship, power, and mattering in higher education. stylus press. stommel, j. (2017). why i don’t grade. https://www.jessestommel.com/why-i-dont-grade/ tilt higher ed: transparency in learning and teaching (n.d.) retrieved 15 may 2021 from https://tilthighered.com/ tretina, k. (31 march 2021). is college worth the cost? forbes. https://www.forbes.com/advisor/student-loans/is-college-worth-it/ tufts university center for information & research on civic learning and engagement (circle) (2020, november 25) election week 2020: young people increase turnout, lead biden to victory. https://circle.tufts.edu/latest-research/election-week-2020#issues:-the-topconcerns-that-drove-youth-to-the-polls turkle, s. (2017). alone together: why we expect more from technology and less from each other. basic books. twenge, j. m. (2018) igen: why today's super-connected kids are growing up less rebellious, more tolerant, less happy--and completely unprepared for adulthood--and what that means for the rest of us. atria/simon & schuster books. uc merced accolades https://www.ucmerced.edu/accolades uc merced institutional research, decision-making and support (irds). (2021, february 28). new student survey dashboard. center of institutional effectiveness, https://cie.ucmerced.edu/new-student-survey-2020-dashboard verschelden, c. (2017). bandwidth recovery: helping students reclaim cognitive resources lost to poverty, racism, and social marginalization. stylus publishing. 14 about:blank about:blank about:blank about:blank about:blank about:blank about:blank https://www.forbes.com/advisor/student-loans/is-college-worth-it/ about:blank#issues:-the-top-concerns-that-drove-youth-to-the-polls about:blank#issues:-the-top-concerns-that-drove-youth-to-the-polls about:blank https://cie.ucmerced.edu/new-student-survey-2020-dashboard microsoft word bresciafinal journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006, pp. 56 – 74. connecting, making meaning, and learning in the electronic classroom: reflections on facilitating learning at a distance charlene johnson and william brescia, jr.1 abstract : the increasing use of technology to meet the vast educational needs of our expanding world has led to heightened concerns about learning experiences within educational environments that are removed from the immediate purview of instructors (duffy & kirkley, 2004). recent calls for use of more collaborative environments in which students interact with their instructors as well as with fellow students, have become more pronounced because of the purported learning benefits for students (bonk, 2002). constructivism, a theory of learning that is based on collaboration and interactions, provides such an environment (jonassen, 1991). we report on the experiences of one professor who maintained a constructivist approach while teaching a foundation course in education (classroom learning theory) in a distance education setting. emphasized are the challenges associated with creating the appropriate conditions for learning when moving from the face-to-face interactions of the regular classroom to the setting of compressed video. the implications of the medium for her role as facilitator, the establishment of a learning community, techniques of questioning and inquiry, and group collaboration are addressed. the impact of the medium and the greater cultural diversity of the distance education classes on how the tenets of constructivism are manifested and experienced by the students is also discussed. i. introduction there is considerable use and application of technology for instructional purposes in the electronic classroom. distance education including video conferencing, web-based courses, and compressed video, has greatly increased the breadth and scope of educational outreach (mangan, 2001). educators use distance education to span the distance between groups and to ensure equitable access to educational opportunities for those interested in receiving additional training and/or expanding their educational repertoire in a number of disciplines (mangan, 2001; raymond, 2000). accompanying the increased use of distance education and its technological tools is the clarion call for effective pedagogical strategies to ensure that the learning processes within this medium are as authentic and effective as those provided in traditional on-site, face-to-face (f2f) classrooms. to ensure that the learning experiences are equitable to those offered in f2f classrooms, bernard, derubacava & pierre (2000) recommend that courses offered via distance education be comparable in format and content to the f2f courses. given the structural and technological realities of teaching at a distance, adhering to this recommendation can be 1 assistant professor of preventive medicine and director of instructional technology, university of tennessee health science center, 930 madison avenue, room 836, memphis, tn 38163, 901/448-5506, bbrescia@utmem.edu. johnson, c. and brescia, jr., w. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 57 challenging. fostering and maintaining the interactions that are endemic to f2f classrooms further compounds the difficulty. interactive instructional approaches that complement and build upon students’ existing knowledge base and experiences are touted as being highly effective for enhancing the learning experiences for a wide range of students, and for fostering higher-order thinking and problemsolving skills (abdal-haqq, 1998; keiny, 1994). the student’s role in interactive instructional environments is to work with others to discover, construct, and participate in social collaborations that bring about meaning (crumpacker, 2001). this is true for courses offered f2f as well as those offered via distance (mcalpine, 2000). constructivism as a philosophy and approach to learning embodies these axioms. this article examines one instructor’s quest to remain true to her constructivist beliefs while leading a classroom of aspiring teachers via compressed video, an interactive video format. her experiences while teaching the course via distance to three groups of students over four years are compared across the three classes as well as with those of the students she taught f2f. the foci of the discussion are the contributions, challenges, cultural and logistical implications arising from maintaining an effective constructivist environment via distance; and, the influence these experiences have had on the professor’s beliefs regarding teaching and learning processes. the questions addressed in this article are: 1. can an instructor guided by constructivism remain true to the major precepts of this approach when physically distant from students? 2. how are learning community, collaboration, question/inquiry, and student voice manifested in distance-learning environments? 3. does distance differentially affect students’ reactions to a theoretical, foundational course that is taught based on constructivism? 4. how do students’ reactions to and understanding of material delivered via distance compare to that of students in the typical, f2f classroom setting? 5. at are the implications of cultural understanding and compatibilities for reactions to and learning within constructivist classrooms via distance? ii. constructivist theory the traditional transmission model of education views learning as an infusion of facts and information from one individual to another (abdal-haqq, 1998; keiny, 1994). constructivism diverges from this pedagogical model with its emphasis on collaboration and active participation of students as they seek to understand the material and resolve any inner conflicts it may cause. students play an active role in constructing meaning from material studied based on their experiences and background knowledge (scheme). their prior conceptions form a basis for determining the meaning of new knowledge. collaboration and interaction among peers to “test” and mediate the knowledge process is integral to the process of making meaning from new experiences. learning is considered a long-term phenomenon that requires discussion, debate, and opportunities to reconstruct ideas (brooks & brooks, 1999; watts & pope, 1989). because knowledge is generated in collaboration rather than transmitted from one (a teacher) to another (a student), the role of the instructor is one of facilitator as opposed to “transmitter” of knowledge (brescia, 2003, abdal-haqq, 1998). to effectuate these principles, collaborative learning is a major practice within constructivist approaches (bonk, 2002). these ideas concerning learning and its interactive nature are founded and endorsed by a number of theorists including jean piaget, lev vygotsky and paulo friere. johnson, c. and brescia, jr., w. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 58 for piaget, the development of scheme, the cognitive mental organization within the minds so fundamental to learning, hinges on experiences and the meanings individuals attribute to them (wadsworth, 1996). these meanings are mediated by interactions with others. vygotsky’s view on cognitive processes differed from piaget’s in that he did not see cognition as being influenced by interactions, but as being determined by them. the implications of culture for these interactions and the meanings given to them are an integral part of his ideas on learning (wertsch, 1985). freire (1970) brought another dimension to the interactive nature of learning by proposing the idea of critical pedagogy. rather than education serving as a mechanism to assimilate youth into the existing order of things, critical pedagogy seeks to provide young people with the mechanism to build anew and revise the existing order of ideas and concepts. freire was critical of the “banking” paradigm in education which asserts that we seek to “deposit” knowledge into children, with little to no recognition of their culture, i.e., what they bring to the educational experience, or the relevance of the new information for what they already know. this process of depositing is considered detrimental to the authentic, contextual learning which freire contended is necessary for the incorporation of information, lifelong learning, and praxis – action based on knowledge. for freire, teachers facilitate learning and do not determine or provide the learning. as with constructivism, the trend or tradition of teachers as transmitters of knowledge is antithetical to freire’s beliefs about learning and the transformative nature of the learning process. each of these theorists emphasized raising the level of cognitive processing by building and expanding on what students bring with them experientially, culturally, and educationally so that they become learners capable of analytical reasoning in any learning environment. these ideas/concepts are especially pertinent for distance education given the nature of the medium and the variety of contexts that converge in the virtual classroom. likewise, educators who believe in these principles should endeavor to ensure that authentic, transformative learning processes are germane to their instructional approaches. facilitating is an instructional technique which is essential to effectuate these principles. for preservice teachers, coursework that reflects these principles enhances their ability to make meaning of foundational theories and furthers their understanding, incorporation and subsequent use of these principles when they construct educational practices (keiny, 1994; abdal-haqq, 1998). to ensure that students/preservice teachers are exposed to these principles in action, instructional strategies such as group work and focusing on major ideas are regularly used (muirhead, 2001). in groups, ideas are discussed, debated, and negotiated with peers and instructors. these class dialogues are integral to students processing and formulating meaning of the material across varied perspectives. focusing on major concepts rather than facts in isolation and ideas out of context allows students to incorporate information in a more meaningful and contextualized manner (abdal-haqq, 1998; van looy, callaert, debackere, & verbeek, 2004). both practices reflect the belief that knowledge is constructed and predicated on the relevance or meaningfulness of the knowledge for the student. for distance learning these practices are even more pertinent given the nature of the medium with students who differ by location, experience, and educational background. negotiation of meaning is more complex and richer given these differences but its success hinges on the learning community that is developed. establishing a learning community is critical to the success derived from these interactive practices for student learning. the community ethos is an essential component of a constructivist classroom. it provides the safety net that frees students to share their experiences and ideas in an johnson, c. and brescia, jr., w. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 59 otherwise risky environment. establishing a distance environment in which students feel safe makes it possible for learners to share their knowledge across the various sites (brown, collins & duguid, 1989). these principles and practices are standard within classes taught by the authors whether they are f2f or via distance. in the next sections, the context, i.e., course, program and class activities; students and their reactions, including those on campus as well as those at a distance; and the instructor and her reactions are discussed. iii. constructivism f2f a. course information this course is part of a five-year, masters of arts in teaching (m.a.t.) licensure program. it is an upper-level (junior or senior) educational psychology/learning theory course required for all education majors. it is taken after a student has completed the introductory course in education. the instructor taught this course three years in a f2f setting; was among the inaugural group who developed and taught the course; and has been integrally involved in its development and evolution over the years. the instructors who collaborated on the design of the course ascribe to constructivist principles and agreed that the course would reflect these ideas. the activities were developed for f2f classes based on constructivist ideology. b. students the majority of students in the f2f classes are traditional in that they range in age from 19 – 23 years old, are single, have limited experiences outside high school, and have few responsibilities beyond themselves. most of the students are of european descent and grew up in ethnically homogenous areas of the state. the course is a core course for all students majoring in education, and, as a result, each class contains students from various concentrations: elementary, secondary, music, art, agricultural, and special education. c. classroom practices – student reactions over the years, it has become evident that students are basically unfamiliar with anything which approximates constructivism in an educational environment. to expeditiously introduce them to this approach, the following paragraph is included in the instructor’s syllabus for the course. course requirements: class attendance/participation: a constructivist approach is employed in class whereby students are encouraged to be actively involved in and responsible for their learning. students are encouraged to read the material, note any questions or areas where concepts were not fully understood, and raise these questions during the class. however, the readings will not be “rehashed” during class. the focus in class is on students’ interpretation and synthesis of the reading material; different dimensions of the issues/ideas will be explored. a variety of learning approaches are used within the class for interpreting and analyzing the material. johnson, c. and brescia, jr., w. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 60 during the first days of class, this passage is discussed including what is explicit in this paragraph and the implied expectations. to further familiarize students with the approach and its tenets, one of the topics assigned for group presentation is constructivism. peers (group members) give their interpretations of the approach and the underlying rationale for the instructor’s seemingly ambiguous responses and unstructured activities. although largely unfamiliar with the constructivist approach, some students do report they have “heard about” the instructor from other students who have taken the course from her (mixed reviews). based on these reports, the students have some preconceptions about the instructor. however, the instructor has no knowledge of them as students/learners. every class is novel with its own unique personality and needs; the ongoing challenge is to develop each distinctive personality into a community of learners whereby the principles of learning within constructivism’s parameters can be manifested. the development of a learning community is integral to the success of this approach because it is vital to elicit sincere, reflective thinking on issues (bonk, 2002). to facilitate this type of environment, a significant amount of attention is given to “reading” and knowing the students. the first and most basic way of becoming familiar with the students is to recognize them by name. becoming familiar with names and hearing students’ voices are initial steps to building community among the students. name cards are created (students write their names on folded index cards– 5” x 8”) the first day of class. these cards are placed on the students’ desks during the first few classes, until the instructor recognizes them by name. another practice that contributes to the association of names with faces is calling on everyone at least once during the course of a class session. in addition to gaining familiarity with the students’ names, this practice indicates that value is placed on hearing the different “voices” within the class. however, a student always has a right to “say no.” that is, if they do not wish to “share” when called upon, a student can say he or she does not wish to share their perspective on that issue. this norm is established at the beginning of the semester and adhered to throughout the term. groups are another means to build community and they also contribute greatly to the mediated process of learning. for each class, there is a major assignment to be part of a group presentation for the class. this assignment reflects collaborative learning’s principle of interdependence in that the same presentation grade is given to each member of a group. individually, students write a paper or an article critique related to the research done for their contribution to the presentation. this ensures some measure of individual accountability (johnson, holubec, & roy 1984). in addition to this assignment, groups are formed within class on a regular basis to discuss or explore the concepts being studied. the format and manner for responding within the groups varies. an example of an in-class group activity that is used during the study of piaget and his theoretical principles follows. students are placed into groups and directed to depict on newsprint (markers are provided), without using sentences or paragraphs (narrative), piaget’s major theoretical precepts concerning thinking and cognition (assimilation, accommodation, organization, scheme, equilibrium, and disequilibrium) (woolfolk, 2004). the idea is not to just restate a precept (e.g., “assimilation is fitting new information into existing schemes,”– repeating a statement from the book [woolfolk, 2004, p. 31]). students are required to somehow demonstrate/illustrate the meanings and relationships of piaget’s theoretical precepts for cognitive development. initially, most students express apprehension about this assignment and johnson, c. and brescia, jr., w. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 61 cite a lack of drawing ability and/or uncertainty about how to create a depiction as reasons why they lack the skill or expertise necessary to participate in the activity. however, with prodding and encouragement, they slowly begin to discuss the meaning of the terms. eventually the class is buzzing with ideas as students become engaged in the activity. discussions concerning the meanings of the terms/precepts and the best way to illustrate and present these meanings emerge and enable students to illustrate their interpretations of the precepts in insightful ways. this activity reflects constructivist principles in that students’ backgrounds, majors, interests and educational experiences are incorporated in these depictions. a wide range of motifs including animals, knitting, sports, farming, etc., become a backdrop for explaining these concepts and their relationships. group processes are monitored by checking with each group, asking clarifying questions about the depiction and its meaning, and addressing questions the group members could not resolve for themselves concerning the precepts and/or the appropriateness of their drawings. the activity and the illustrations allow the instructor to make a firsthand assessment of students’ understanding and interpretations of the material. peers also assist in addressing misconceptions. this activity also incorporates another supporting belief of constructivism: the importance of recognizing and valuing multiple intelligences, i.e., the varied ways in which students process information (gardner, 1983). often students who struggle with traditional learning activities which emphasize memorization and recitation excel in this activity. on the other hand, many students who excel in these traditional activities are challenged by this exercise. another area of angst for students is the inquiry nature of the course. students in general are initially uncomfortable with nature of the professor’s questions and her responses to their inquiries. questions that focus on the major concepts and how students incorporate them into their practices prevail in the classes. questions are asked concerning the relevance of the assigned readings (what idea or concept in the chapter “spoke” to you and why?); familiarity with the practices shown (have you ever experienced this practice? if so, what were your reactions?); and the implications of these ideas for educational practices (given the research on the effectiveness of this approach/idea/concept/practice, will you incorporate it into your practice?). follow-up questions are often asked to further explore students’ meanings, perceptions, and interpretations of the material. the intent of these questions is to get students to grapple with the ideas and assess the implications for them as educators. often, the professor responds to students’ questions with a question: “what do you think?” initially, students are frustrated with this response because they want the instructor, to “just tell me the correct answer.” they want to be imbued with the instructor’s understanding of a concept rather than develop their own. instead, they are told that “correct” is contextual and that, although they are expected to follow certain analytical processes when studying material, they are expected to arrive at their own conclusions concerning how effectively a model explains phenomenon. as facilitator, the instructor places the onus of responsibility for addressing their issues and concerns with students – which adds to their frustration. they are given assistance in the process of making decisions about the relevance and implications of the material studied, but this is not determined for them. the question and answer procedure allows the community – both instructor and students – to better recognize any confusion or misinterpretation of material. when a student explains his/her thoughts on an issue, others gain insight into how someone else interprets the concept. listeners can then assess that interpretation in light of their own perceptions of the concept. as johnson, c. and brescia, jr., w. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 62 the semester progresses, students tend to: become increasingly active in their learning process; address each other’s questions or concerns; and accept that answers are contextual and may differ based on perspectives. iv. constructivism at a distance a. program/course information the off-campus licensure program was initiated to support the training and local employment of new teachers, particularly those of african descent (because of the largely african american population of the area). reflecting national trends in education, the number of african american teachers in the state’s schools has declined (american council on education, 2000). rural, economically depressed regions such as the one targeted by this off-campus m.a.t. program, are usually the first to feel the effects of this decline. the region has difficulty attracting and creating employment opportunities to replace the agriculture-related jobs lost because of technological innovation. the program also contributed to the flagship university’s stated mission to meet the needs of all the citizens of the state. the program was set up so that students completed their first two years of coursework at the regional community college. the upper-level courses for the undergraduate degree leading to the m.a.t. program and the graduate courses in the m.a.t. program were offered via compressed video at a location 300-plus miles from the state’s flagship university which was the instructor’s home campus. initially, it was expected that the students would travel to the flagship university to complete the last two years of undergraduate coursework and courses for m.a.t. degree. however, most of the students were nontraditional (over 25 twenty-five years of age), had families, were employed full-time, and were not able or willing to leave their families and jobs to complete the degree program in residence. distance education (interactive video) was a viable alternative for the students to complete the degree program. the only degree/licensure program offered via distance education was elementary education. instructor the instructor shared a common ancestry, of african descent, with the majority of students within the program, but was not born or raised in the state. during her time at the university, she had worked as a consultant with the schools in the area where the m.a.t. program was directed. however, the distance education course was her inaugural instructional experience teaching a university course to a class composed entirely of students from the region. when first asked to teach this theoretical course using distance education, the instructor was hesitant, resistant, and apprehensive. could a constructivist approach be implemented effectively without physical presence and proximity? the opportunity to interact with a population of students from another part of the state who did not otherwise have access to the course was a strong incentive that overrode her initial hesitancy and trepidation. the challenge was to use the advantages provided by this medium without compromising the integrity of her beliefs about the roles of learner and teacher. the instructor’s initial impression of distance education and its learning expectations (lecture, transmission model) was based on the setup of the room from which the video was to originate. the setup resembled that of a principal’s office. there was a chair set up at a table with controls and a microphone. after a consultation with the technical-support people, the setup was changed to be more in line with the instructor’s preferred mode of instruction. the chair and a podium were placed next to the controls, which eliminated the appearance of a johnson, c. and brescia, jr., w. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 63 barrier separating the instructor from the students. it was a seemingly minor change, but it gave the instructor the freedom to move around during the class which conveyed a more interactive persona. a. students over the course of four years, the course was offered three times via distance education. although similar in some respects, each group has its own unique quality. the first class (group 1) was predominantly african american (12 of 15) students; nontraditional in that 80% were over 25 and had families and jobs. these students were selected based on their excellent academic records and high grade-point averages. they were the program’s pioneers who experienced the usual intricacies of a new program. these common experiences and frustrations bonded them to each other. the second class (group 2) was similar to the first in terms of ethnicity (16 of 19 students were of african descent) but differed in terms of age and experiences; less than 60% were under 25, had families and outside jobs. in addition, the educational backgrounds were less stellar than those of the first group. for the inaugural group, high performers were selected to ensure the success of the program. the program received a significant amount of publicity and public response which was very positive. people from the community, the region, and the state were impressed with the program’s intent–providing schools in the area with well-prepared teachers– and with the distance-education format. this support was a catalyst for continuing the program for subsequent cohorts of students. those who came after the first group were recruited less aggressively and with less stringent academic requirements. the third class (group 3) differed from group 1 and group 2 in terms of ethnicity. ethnically, the class was more diverse with only nine of the eighteen students being of african descent and nine of european descent). ages and experiences resembled those of groups 1 and 2; only three of the eighteen (17%) were traditional students. as with the f2f classes, each class had its own unique personality. the varied personalities affected reactions to the lessons and the climate that evolved in the class. b. classroom practices – student reactions the distance-education students were also unfamiliar with a constructivist approach to learning and initially had mixed reactions to it. most students responded amicably to being asked to bring their realities into the classroom and share their opinions and ideas about the material and its relevance for them (knowles, 1990). in general, there had been little (if any) attention given to their experiences or perspectives by other instructors (delpit, 1995; freire, 1970). they welcomed opportunities to relate the course material to their lives and varied realities. these opportunities albeit gratifying and somewhat unexpected, did not preclude the discomfort associated with the process. overall, these were high-caliber students whose high grade-point averages (pervasive in the first class, somewhat common in the second and third classes), were indicative of their success with the traditional transmission model of learning. similar to the students in the f2f classes, the students preferred the didactic approach to teaching and learning, which involved listening to lectures, taking notes, reading the assigned texts, studying the notes and readings for exams, and responding with memorized information to exam questions. tolerance of ambiguity was minimal. the focus was on completing assignments and getting correct answers. similar to johnson, c. and brescia, jr., w. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 64 the f2f students, the explore-and-find-meaning approach emphasized by the instructor was different than what they were accustomed and, therefore, a source of great frustration. for the distance-education students, this frustration was further compounded by obligations to families and employment. they wanted (sometimes demanded) to know the exact, “correct” answers so they could recite them on cue just as they had done in the past with great success. they were not inclined to analyze and grapple with issues. interacting with the material and analyzing concepts for relevance and meaning were not customary practices for most of the students and they were unsure of the steps in and/or the feasibility of the process. when confronted with this situation on campus, being physically present the instructor can note students’ reactions, more readily assuage any discomfort, and facilitate their thinking about and interacting with the material. being 300 miles away and viewing students via a monitor presented a significant challenge to this mode of accommodating students’ needs. establishing a learning community and nurturing risk-taking behavior within the classes via distance was a challenge that was met with mixed results. methods used successfully in the f2f environment were initially employed for the distance-education classes. name cards were constructed, but the faces and the cards were not always clearly visible on the monitor. as an accommodation, students were asked to state their names before speaking. this helped to familiarize the instructor with the names in a time comparable to that in f2f classes. for a short time, students were acknowledged via an attribute or characteristic – “the young man in the red shirt,” “the person sitting beside [student name],” or “the person sitting in the corner.” this method, due to its impersonal nature, was used only during the first class period for group 1. reactions to being called upon to share one’s perspectives were also mixed. although they welcomed the opportunity to relate the material to their experiences, culturally, they did not appreciate or accept the process for communicating their understanding of material to the instructor. african american response style influences how constructivist ideology and practice is perceived and implemented. delpit (1995) discusses the direct response style prevalent in african american communities and its incompatibility with obscure approaches to instruction, such as process approaches to literacy and constructivism. she posits that the ambiguousness of response that is endemic to constructivism is foreign to the direct/straight-forward style of communication and discussion prevalent in african american communities. this premise is mostly applicable to literacy approaches for children of color when trying to incorporate sounds and meanings of words that are incongruous with their backgrounds. the instructor was aware of and had reflected on this contradiction and its significance for her approach to teaching and learning. previous to teaching the distant education class, she had not had an opportunity to really grapple with this enigma given the majority european american student population she usually teaches. however, with the distant education students who embodied these realities, her beliefs on culture and its implications for meeting student needs were under scrutiny in conjunction with her constructivist philosophy. although somewhat in agreement with this premise, she had a different “take” on this seemingly contradiction. she believed that the constructivist approach to teaching and learning does not run counter to the response style prevalent in african american communities under certain conditions. students of color are constantly dealing with ambiguities in language and meaning as a matter of cultural style (delpit, 995). translating this into classrooms and using it to enhance student learning was the challenge. shared ancestral background helped in meeting this johnson, c. and brescia, jr., w. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 65 challenge. common understandings of words, interaction styles, and conventions, allowed the class and the instructor to interact directly as they capitalized on meanings and extended them – reflective of constructivist precepts. multiple responses that were recognized and valued to the seemingly vague questions (more than one “correct” answer) empowered and furthered students’ “voice.” they explored personal meanings and expressed their feelings about the course and its requirements. once the students were more familiar and comfortable with the instructor, they directly expressed their frustration with query and response in the expectation that she would be more forthcoming with the “correct answer.” when the instructor continued to probe and explore their meanings and interpretations of the materials, students addressed the questions and displayed their frustrations in varied ways. as in the f2f classes, many times when a student was asked a question, there was a seemingly long and uncomfortable silence while waiting for a response. using wait time, the question was not passed to another student until the first student indicated he or she did not know the answer or did not wish to answer. via distance, this uncomfortable silence hangs between the two locations and its manifestation is difficult to gauge with limited vision of students and their reactions. frequently, students attempted to address the discomfort by assisting each other in providing an answer to the question. they offered clues, hints, and, on some occasions, answers based on what they perceived the instructor wanted. the combined efforts of students in this situation were crucial in furthering the analysis and understanding of material by the distance classes. via distance, with the instructor limited to the capabilities of a monitor, students restating a question or providing cues/hints for a classmate greatly aided in the shared understanding of the material and its meaning. in addition, the instructor was able to hear how students collectively processed and interpreted the information. similar ancestry helped mediate the understanding process. however, given the differing backgrounds of instructor (midwestern, urban) and students (southern, rural), these clarifications were instrumental to understanding the context and meanings given to concepts by the students. the students’ common experiential knowledge helped them negotiate meaning in a context that was relevant for them (kim & hannafin, 2004). this process evolved in group 1 and was often observed in group 2, but not in group 3. the interdependence of group 1 based on their similar backgrounds and goal to successfully complete the program, intensified their sense of community and served to define a common purpose for the class as they sought to make collective meaning of the material. group 2 was not as solidified as a learning community, but their similar experiences within and outside the course served to mold them into more of a community than group 3. group 3 had little sense of community. it was more fractured on the basis of their experiences and backgrounds. additional effort was required to get them to collaborate on the meanings of the material. these experiences with the three classes underscored the benefits of establishing a learning community for the class if one is not already in place. the community serves the needs of the class in a number of ways. developing community is contingent on hearing and honoring student voice but more often than not, students resist this objective/focus. in group 1 a student requested that she not be asked questions in class. when the instructor explained the need to hear her voice and interpretations of the material, the student countered with she was shy and wanted to hear everyone else and learn from them. a compromise was reached, in which an attempt would be made to forewarn her before calling on her. this compromise had been used in f2f classes on rare occasions for students with similar issues and/or diagnosed special learning needs. over johnson, c. and brescia, jr., w. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 66 time, the interactions enabled the student to become more vocal in class, offering responses based on her interests. she realized that by speaking voluntarily on issues which held her interest, she was able to choose when and how she participated. for other students who expressed similar concerns, a compromise was struck whereby their voices were heard during class without (as they perceived it) being “put on the spot.” group presentations were required of the distance-education classes, but accompanied with additional challenges. the topics for the presentations were selected by the instructor and were broad in scope (vygotsky, multiple intelligences, constructivism, etc.) to allow each group considerable latitude in determining their focus and areas of interest within the category for their presentation. based on their collective interests, each group was to determine its focus; research the areas of interest; and, develop a presentation based on their collective research. maintenance and monitoring of groups to ensure they function effectively proved to be a major challenge given the distance and the available resources. in f2f classrooms, if a group had problems or concerns, students requested assistance after class. sometimes an intervention was made that same day or by the next class period. typically, issues were minor and pertained to differences in working styles or opinions concerning the appropriate manner of presentation. in the distance environment, students’ concerns were more difficult to address. students could not talk to the instructor after class because the audio and video feed connection between the locations was shut down at the ending time for the class. students used email, but in many cases that did not provide the desired level of immediate and personal interaction preferred in these situations. thus, for the distance learning classes, personality and working-style issues that were ameliorated early and easily in the f2f classes became major stumbling blocks to developing an effective presentation. an associated issue stemmed from the complex restraints on time and responsibility for the students with families and jobs. finding time to meet outside of class to decide on the focus and manner of a group presentation is problematic even for relatively unencumbered traditional college students. however, when one works all day and attends class in the evenings – distance education classes were scheduled from 4:00 to 9:00 pm, two or three days per week – meeting this requirement becomes even more complicated. it became obvious after the experiences with group 1 that more support from the instructor via email and phone was needed to ensure they were functioning adequately. subsequent to group 1, one or two persons in each group emerged as leaders and became the major contacts at the off-campus site through which the instructor monitored the process. additionally, individual group members were contacted to discuss any evident frustrations or concerns in order to support and encourage their involvement with the group. these measures helped to make the experience less stressful and more viable for the students. a major issue was access to library resources necessary to adequately investigate a presentation topic. technically the distance-education students were enrolled at the flagship university. however, for library access they were limited to the inadequate collection of reference books and periodicals at the local community college. theoretically, students had access to the ample resources, i.e., educational literature and periodicals, at the flagship university. realistically, students often did not have ready access to these collections due to problems with the local internet service through which they obtained online access to the university library. after several unsuccessful attempts to resolve library-access issues, the group presentations for group 1 were cancelled. instead, students wrote about their experiences johnson, c. and brescia, jr., w. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 67 working with their groups and trying to secure adequate literature for the presentation. for groups 2 and 3, library issues were addressed early via email and consistent updates in class. access to resources did improve in subsequent years, after the university library became involved and worked to ensure access, but the problem was never completely resolved. some students had access to more technologically updated servers and internet providers, while others remained on the wrong side of the digital divide. to address this challenge of equity and access, books were recommended by the instructor on the presentation topics which the group could consider reading and using as an alternative resource to research articles. these books were available at the local libraries, through purchase from the community campus bookstore and/or online sources. with these adaptations, the group presentations remained a requirement for groups 2 and 3. classroom activities requiring group collaboration, such as the one on piaget’s theoretical precepts, proceeded as with f2f classes but with some adaptations. displaying and presenting results for distance education classes were done via the document camera because the drawings were harder to decipher if just held up by students and shown on screen. moreover, additional description and explanation were often required. students explained their depictions with rich detail and defended them when questioned about whether there had been appropriate application of the theoretical precepts. reactions to questions about the non-written depictions and their meanings (how reflective of the precepts and their relationships) varied by class. in groups 1 and 2, students tended to display their drawings and have them questioned with little to no display of defensiveness or hard feelings. by this point in the term, the understood response style allowed students to take risks and directly state their perspectives on the instructor’s interpretations of their ideas. generally, the clarifications and explanations enhanced the shared understanding of the class on concepts being discussed. however, group 3 lacked a strong community ethos and generally were more defensive about any comments or questions that challenged their depictions and interpretations of concepts. eliciting the higher-order thinking required to address the questions raised in class was also a challenge to maintain and foster. when asked a question, students were hesitant to address it because of the possibility of follow-up questions requiring application and analysis. they feared they would not be able to address them adequately, thereby seeming incompetent to their peers. the community (peer) assistance was helpful, but as the class endeavored to explore implications of theoretical precepts for their communities and its students, it was obvious that many were unaccustomed to these types of questions and/or how to address them. according to freire (1970), the students were not accustomed to learning as a basis for praxis or action. in the f2f classes, the instructor walked around the classroom, stood next to students, probed and encouraged responses recognizing that students were not accustomed to investigative/analytical questions. they were most comfortable when repeating a concept or principle from the book than when thinking about its meanings for meeting differing students’ needs. with a sense of community established, students were more likely to move from their comfort zones and venture to answer questions and/or propose solutions to issues. when they shared, the instructor provided encouragement and assisted them in making connections between the theoretical underpinnings of the concepts and their practical experiences with them. distance-learning environments which by definition make physical proximity impossible changed the dynamics of these strategies resulting in them being less personal in format and function for the students. johnson, c. and brescia, jr., w. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 68 these impersonal realities of the distance education environment impact the role of facilitator and prove challenging when handling students’ frustration. similar world views and communication styles helped in finding the right degree of directness for addressing student concerns. this was not as successful with the f2f classes due to the ethnic and cultural differences between students (primarily european american) and the instructor (african american). with f2f classes, a more subtle approach was used to address student frustration. with the distance education students, frustration was discussed candidly. students openly complained to the instructor about the requirements and her “hard” grading. conversely, the instructor directly addressed the class regarding concerns she had about the level of response and their performance on assignments. the common understanding concerning directness of response without ill feelings served the instructor’s purposes with the class also. once, when concerned about the overall quality of their work (poor, inarticulate, and careless), the instructor discussed her frustration with this issue in the class. what came from the discussion was enlightening and reflected research findings for student populations in lower socioeconomic status areas. the students explained that the constructivist approach with its emphasis on collaborative learning, processing of information, and reaching higher levels of thinking, was new and unfamiliar to them. past educational experiences had emphasized the rudiments of learning and focused on lower levels of thinking – rote memorization, comprehension, etc., known to predominate in areas where the poor and lower socioeconomic levels of people preside (oakes, 1985). they were not “slacking” in their work but were finding it extremely difficult to engage in the analytical processes expected. their explanations helped the instructor better understand why their level of disequilibrium was high and somewhat debilitating. the class explored how and why these lower-level cognitive exercises prevailed in the locality as well as the implications of a link between low socioeconomic status and the predominance of these approaches to learning (apple, 2004). one student shared that she had recently observed these practices in her daughter’s classroom and had become more cognizant of them as a result of the class. she pondered the implications of these practices for area students’ future academic achievement and analytical abilities given that these higher-order thinking skills are necessary in higher education and in life. this was an extraordinary educational moment: the class had examined an issue and its implications for them making a difference as educators with their training – praxis (freire, 1970). based on this feedback, the instructor provided more opportunities in class for discussing concerns with the coursework. questions and issues were addressed as directly as possible, while still allowing students to analyze the material based on their own realities rather than those of the instructor. additional time for processing of information was recognized as necessary for the students to better fulfill required assignments. periodically conducting the class f2f with the distance-education students was significant for familiarizing the instructor with the students and the students with the instructor’s approach to teaching. during these visits the instructor walked around the classroom and asked questions just as she does in her classes on campus. students experienced constructivism in close proximity as opposed to having it modeled from the podium via distance education. they appreciated and understood the approach much better after those classes. they became more understanding and comfortable with the instructor and her approach after the visits. she tried to make at least three visits to the area: one to observe the group presentations, another to teach a concept, and the third for the presentation of the portfolio at the end of the semester. johnson, c. and brescia, jr., w. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 69 after the semester ended, the instructor continued her affiliation with some of the students. several emailed her to express gratitude and their thoughts about the value of the course. the most memorable email came from a student in group 1; the sentiments expressed embody her goals for the course and the students. she wrote: i just wanted to let you know how much i have enjoyed taking this class under your instruction. some of my fellow classmates feel that you expected too much from us. however, they fail to realize that through it all they met all deadlines and they fulfilled all requirements. therefore, they should realize something about themselves. i have really and truly enjoyed taking classroom learning theory. dr. johnson, you will never understand how much you have helped me this semester. you have helped me to explore avenues within myself that i never knew existed. you made me push myself to excel far beyond that of which i felt i was capable of. reflecting back over the semester, i realize more so now than ever before, the reason why you pushed us the way you did is because you want us to become the best educators that we possibly can. you want us to do more than just memorize answers, you want us to comprehend and fully understand the concepts and ideas. in my opinion, i honestly feel that you are the type of educator that i hope to become. an educator that cares about her students, both inside the classroom and outside of the classroom. even when i thought that no one noticed how i felt at the time, you noticed. you encouraged me to hold on. i wanted to say thank you. thank you for caring, thank you for pushing me to do the things of which you felt i was capable of doing. thank you for being “true” and thank you for being “real.” this email was received after grades were posted and the semester was done. other students have expressed similar sentiments, but not as eloquently. in subsequent years, some have emailed the instructor about the value of the course when preparing for the licensure exams, praxis ii. it seems that the skills gained from having their ideas, beliefs and perspectives challenged, and being prodded to articulate their rationales proved beneficial for analyzing issues that are a major emphasis of the test. additionally, case study is a part of the class, so it provides some exposure to this type of assessment. c. instructor’s reaction as a constructivist, the instructor’s questioning techniques were enhanced, and her commitment to groups remained steadfast in spite of its challenges. her initial reaction to the idea of distance education has changed. she sees the value and efficacy of distance education for reaching students who would not otherwise be able to fulfill degree-program requirements. the exposure to another reality–i.e., rural, economically depressed area – was enlightening and led to a careful reexamination of her perspectives on difference and its manifestations. she especially appreciated having an opportunity to become privy to and interact directly with the different contexts and realities of student learning. every class began with current events – this is true in f2f as well as distance classes. the instructor’s rationale, which is shared with the class, is that events worldwide have johnson, c. and brescia, jr., w. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 70 implications for classrooms. in both f2f and distance education, local and world news are discussed and evaluated for relevance to the classroom, students, and educational issues. however, for the distance-education students, this activity became a forum for assessing local news and events from their perspectives. local realities and histories brought different dimensions to the material being studied. some of these realities have become part of the instructor’s educational scheme and continue to be incorporated into course activities when teaching precepts f2f. in this instance, distance education has informed the practices on campus. educationally, the instructor has benefited as a facilitator and as a learner. her questioning, active listening, and facilitation skills were enhanced as a result of these experiences. she has learned that her methods for detecting and handling student discomfort and disequilibrium in a f2f setting are not as effective when used in a distance learning environment. however, with attention to students’ needs and realities, a few alterations, some flexibility, and considerable patience, it can be just as productive. a dominant area of intelligence for the instructor is interpersonal intelligence (gardner, 1983). although aware of its influence on her as a learner and an instructor, the distance education experience further validated that the feedback and energy from the class are fundamental to her functioning in the classroom. there is a symbiosis of energy and involvement between instructor and students. the distance caused the instructor to be livelier and more involved in an effort to invoke this symbiotic relationship with the students. this is an area warranting additional attention as it relates to distance education – how do different intelligences and learning styles impact receptiveness to this medium as a learner and as an instructor? although f2f interaction is the instructor’s preferred style, she understands that the attributes she dislikes most about the medium – lack of personal interaction, anonymity, and exclusive use of written responses, are the ones that some students find most suitable for their learning. to better meet student needs and “hear” the multiplicity of voices and tones, more familiarity and ease with this medium is warranted. a major benefit of the instructor’s involvement with the distance-education classes is her increased use and familiarity with technology for educational purposes. she has developed listservs and uses email much more frequently than before becoming involved with distance education. the areas of interpersonal intelligence and technology converge when the instructor emails students. humor is a major tactic used by the instructor to maintain focus and interest in material considered to be “dry”, i.e., theories and their tenets by most students. in f2f classes when students misunderstand the instructor’s humor, her response, or manner of responding; their body language usually signals this and she immediately addresses the confusion. email and listserv posts do not include these clues. the instructor became acutely aware of the importance of “tone” in email. based on students’ reactions to content of her emails and on students’ perceptions concerning the “tone” of her messages, she has discovered that her audience may receive a message other then the one she intended to convey. she has worked on this aspect of her communication with some of the technical advisors at the institution and implemented several of their recommendations to use emoticons as indications of humor (smile, ☺, grin, etc.) in email responses. improvement in this area has enhanced her communication skills for all classes. johnson, c. and brescia, jr., w. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 71 v. conclusion and implications the initial hesitancy, frustration, and bafflement concerning the constructivist approach and its expectations were as common at a distance as f2f. with each succeeding class, the instructor learned more about the importance of the sense of community within the class for the student interaction and involvement which are fundamental to the effectiveness of the pedagogical methods and student assignments. over the years she has learned how to be more effective at mediating interpersonal and systemic challenges, to ensure that students receive equitable assignments and exercises (anderson, 2001; hardwick, 2000). for each class the degree to which a learning community is established needs to be assessed, and in the case of there not being one, it needs to be facilitated. for distance education this challenge is especially cogent because the instructor is less knowledgeable about all the realities and/or experiences of the students who are taking the course. the experience of distance education students with the library sources is characteristic of the logistic problems and the “hidden curriculum” embedded within distance education that has received limited attention in the literature (apple, 2004). to ensure that the experiences of distance education students especially those in rural, less-developed areas are equitable with students on campus, access arrangements must be made for technological linkages (anderson, 2001). calling on students and asking several questions related to or associated with the local realities helps to establish community and build interpersonal relationships. listening and using information from previous responses regarding the area and its realities when referring to a theoretical idea also invoke insightful responses. students are often surprised that an instructor listens, values their responses, and builds on them. active listening in any class is important when sincere, continued responses from students are a goal. however, at a distance, this is even more critical to the success of the collaboration and sharing of ideas. instructors need to receive focused training and be given opportunities to practice using the relevant technologies before they face students who are relying on them for instruction. in addition to practicing with the technology, practicing appropriate distance-learning techniques is necessary for all instructors who want to foster fruitful collaborations and interactions. the establishment of community is necessary for collaboration and for constructivism to be effective in classrooms whether they are f2f or distant (bernard et al. 2000; hardwick, 2000; keiny, 1994; mcalpine, 2000; raymond, 2000). the necessity and benefits of designing instruction for distance education with these philosophical beliefs in mind have been documented (bernard et al. 2000; hardwick, 2000; mcalpine, 2000; raymond, 2000). creating appropriate environments involves more labor and time, and has its own distinctive set of challenges. however, doing so is beneficial for higher-order learning. ensuring that learning experiences develop analytical and problem-solving skills equitable to those nurtured in students in f2f classes is essential for those who do not have direct access to university classrooms. otherwise, distance education becomes another means by which the societal hegemony is continued by providing less for those most in need. given that the focus or purpose of distance education is to extend educational equity and access, assurances must be made that students who receive their education via this medium are sufficiently skilled to compete and achieve. it is expected that in the future there will be an even greater need for higher-order thinking skills, problem-solving skills, and effective interaction skills with a range of individuals. concurrently, the use of technology to meet the growing needs johnson, c. and brescia, jr., w. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 72 of our students is growing exponentially. therefore, further study on how to use and implement collaboration and constructivism more effectively via distance is needed. additionally, better understanding of the impact of culture on the distance education learning processes is integral to developing effective environments that are equitable for a wide range of students with differing realities. references abdal-haqq, i. (1998, december). “constructivism in teacher education: considerations for those who would link practice to theory.” eric digest, eric clearinghouse on teaching and teacher education. american council on education (2000). annual status report: minorities in higher education. washington, dc: author. anderson, t. (2001). “the hidden curriculum in distance education.” change, 33(6) 28 – 35. apple, m. w. (2004). ideology and curriculum, 25th anniversary 3rd edition. new york: routledge. baec, y. k. & westrom, m. (2001). “building a cyber training center for teachers in korea.” journal of technology and teacher education, 9(2), 171-179. bernard, r.m., derubalcava, b.r., & st. pierre, d. (2000). “collaboration online distance learning: issues for future practice and research.” distance education, 21(2), 260 – 277. bonk, c. j. (2002, november/december). “collaborative tools for e-learning.” chief learning officer, pp. 22-24, & 26-27. retrieved october 1, 2004: http://www.clomedia.com/content/templates/clo_feature.asp?articleid=41&zoneid=30 brescia, w. f. “a support taxonomy for improving online discussions.” journal of public affairs education, 9(4) 289-298, 2003. brooks, j.g. & brooks, m.g. (1999). in search of understanding: the case for constructivist classrooms (2nd ed). alexandria, va: association for supervision and curriculum development. brown, j. s., collins, a., & duguid, p. (1989). “situated cognition and the culture of learning,” educational researcher, 18(1) 32-42. crumpacker, n. (2001). “faculty pedagogical approach skill and motivation in today’s education milieu.” online journal of distance learning administration, 4(4) retrieved online october 19, 2004: http://www.westga.edu/~distance/ojdla/winter44/crumpacker44.html. delpit, l. (1995). other people’s children: cultural conflict in the classroom. new york: the new press. duffy, t. & kirkley, j. (2004). “learning theory and pedagogy applied in distance learning: the case of cardean university.” in t. duffy & j.r. kirkley (eds.), learner-centered theory and johnson, c. and brescia, jr., w. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 73 practice in distance education: cases from higher education (pp. 107-141). mahwah, nj: lawrence erlbaum associates, inc. freire, p. (1970). the pedagogy of the oppressed. new york: continuum publishing gardner, h. (1983) frames of mind: the theory of multiple intelligences. new york: basic. hardwick, s.w. (2000). “humanizing the technology landscape through a collaborative pedagogy.” journal of geography in higher education, 24(1) 123 – 129. johnson, d.w., johnson, r., holubec, e.j., & roy, p. (1984). circles of learning: cooperation in the classroom. alexandria, va: association for supervision and curriculum development. jonassen, d. h. (1991). “objectivism versus constructivism: do we need a new philosophical paradigm?” educational technology research and development, 39(3), 5-14. keiny, s. (1994). “constructivism and teacher’s professional development.” teaching and teacher education, 10(2), 157 – 167 kim, m. & hannafin, m (2004). “designing online learning environments to support scientific inquiry.” the quarterly review of distance education, 5(1), 1-10. knowles, m.s. (1990). the adult learner: a neglected species (4th ed.). houston, gulf publishing. mangan, p. (2001). “what is distance learning?” management quarterly, 42(3), 30 – 35. mcalpine, i. (2000). “collaborative learning online.” distance education, 21(1), 66 – 80. muirhead, b. (2001). “practical strategies for teaching computer-mediated classes.” ed at a distance. 15(5). retrieved october 14, 2004 from http://www.usdla.org/html/journal/may01_issue/article02.html oakes, j. (1985). keeping track: how schools structure equality. new haven, ct: yale university press. raymond, f.b. iii (2000). “delivering distance education through technology: a pioneer’s experience.” campus-wide information systems, 17(2), 49 – 55. van looy, b., callaert, j., debackere, k. & verbeek, arnold. (2004). “patent related indicators for assessing knowledge-generating institutions: towards a contextualized approach.” journal of technology transfer. 28(1) 53-61. retrieved october 14, 2004 from http://journals.kluweronline.com/issn/0892-9912/contents wadsworth, b.j. (1996). piaget’s theory of cognitive and affective development: foundations of constructivism (5th ed.). white plains, ny: longman publishers. watts, m. & pope, m. (1989). “thinking about thinking, learning about learning: constructivism in physics education.” physics education, 24, 326–331. johnson, c. and brescia, jr., w. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 74 wertsch, j.s. (ed.) (1985). culture, communication and cognition: vygotskian perspectives. new york: cambridge press. woolfolk, a. (2004). educational psychology (9th ed.). needham heights, ma: allyn and bacon. 503 service temporarily unavailable 503 service temporarily unavailable nginx/1.14.1 v11n4powell journal of the scholarship of teaching and learning, vol. 11, no. 4, december 2011, pp. 1 – 11. using clickers in large college psychology classes: academic achievement and perceptions selma powell1, carrie straub, jacqueline rodriguez, and barbara vanhorn2 abstract: this research study explored the effects of the use of clicker technology as a means of formative assessment in large, college introductory psychology courses. academic achievement, as measured by performance on tests of knowledge, was compared between students who used clickers and students that did not use clickers. there was a significant difference between the two groups, although a small effect was noted. students using the clickers responded to survey items created to solicit information regarding student perceptions of increased understanding, ability to self-identify content deficits, fun, and financial value of clickers. data from the survey indicated that the majority of students perceived the usage of clickers increased their understanding of and ability to self-identify concept areas. all of the students reported that using clickers was fun, although approximately only one third of those same students reported that purchasing a clicker was worthwhile. implications for administrators are discussed and alternative technologies are explored. keywords: clickers, psychology, formative assessment, education technology it is not uncommon for undergraduates in public colleges and universities to find themselves faced with their first college lecture in an auditorium-style classroom (kenwright, 2009; mayer et al., 2009). lecture-hall classrooms accommodate large numbers of learners, and lessons are delivered using a teacher-centered lecture format. within this context, researchers and faculty from a small public college in the southeast united states located in a diverse urban community sought to establish effective instructional supports for large, lecture-style classes, specifically through the use of class wide electronic response systems. the purpose of this research is to examine the use of clicker technology on student achievement and perception. in the spirit of boyer’s (1990) call to action, the research team examined classroom teaching practice as a means of “transforming and extending” the knowledge base for instructors with large class enrollments (p.24). i. background. instructors of mass classes rely on traditional, lecture-style lessons to deliver information to large numbers of students; however this lesson format provides little or no interaction or discussion (geske, 1992; gleason, 1986). although large lecture halls may accommodate more students by making efficient use of faculty and facility resources, student understanding of content can be compromised. this learning environment may seem passive and impersonal for undergraduates (hoekstra, 2008). hall, collier, thomas, and hilgers (2005) identified two problems that 1 department of child, family and community sciences, university of central florida, 4000 central florida blvd., orlando, fl 32816-1250, selmapowell@knights.ucf.edu 2 department of social sciences, indian river state college, 3209 virginia avenue, fort pierce, fl 34981, bvanhorn@irsc.edu powell, s., straub, c., rodriguez, j., and vanhorn, b. journal of the scholarship of teaching and learning, vol. 11, no. 4, december 2011. www.iupui.edu/~josotl 2 students in mass class settings face: (1) lack of engagement in the lesson and (2) lack of metacognitive awareness or ability to self-identify deficits in subject area awareness. the lack of engagement coupled with the students’ inability to self-identify deficits in subject area knowledge may result in an inability to master course content. furthermore, edmonds and edmonds (2008) stated that mass class learning environments do not provide opportunity for instructors to gauge students’ understanding of subject areas until the exam results are presented, a scenario leaving instructors and students in the dark regarding subject area understanding until it is too late to remediate misconceptions. according to generative theory of learning, learners actively engage in cognitive processing during learning by attending to relevant material, mentally organizing the selected material, and integrating the organized material with prior knowledge (mayer, 2001; mayer et al., 2009; wittrock, 1974). therefore, instructors optimize learning by engaging in instructional activities which “prime active cognitive processing in learners” (mayer et al., 2009, p.53). the research on generative learning theory indicates that the outcome of active cognitive processing is a meaningful learning outcome (mayer et al., 2009). anderson et al. (2001) theorize that learning outcomes, including the knowledge and skills learned in class, can be evaluated using a variety of measures, including academic test items of class content. to foster the generative learning process, instructors may use questioning methods (king, 1992; rosenshine, meister, & chapman, 1996). yet, faced with the challenge of assessing student knowledge and increasing engagement in mass classrooms, how do instructors gain knowledge about student understanding to provide responsive instruction? in classes with smaller groups of students, instructors may use questioning methods to assess student understanding, as incorrect student responses provide opportunity for instructors to provide corrective feedback to change student misconceptions (kenwright, 2009; mayer et al., 2009). however, in large, lecture-style class’s students may be intimidated by asking or responding to questions (poirier & feldman, 2007). one method for increasing student understanding and engagement is through the use of electronic clickers, a tool for generative learning. clickers, also referred to as electronic response systems, personal response systems, audience response systems, classroom communication systems, or student response systems, allow students to individually respond to closed-ended questions posed by the instructor. anonymous student responses are gathered via a remote handheld device, commonly referred to as a “clicker”, and responses are immediately provided to the instructor. the instructor may then broadcast the results to the class using a multi-media projection device. publicly examining student responses achieves a variety of objectives, which include but are not limited to, assessment of student understanding, increasing student engagement in the lesson, and providing students with a venue for objectively self-assessing their knowledge of the subject area (mayer et al., 2009). the clicker system is comprised of two parts: a classroom-wide response system and a set of individual, handheld, remote clicker devices. colleges generally incur several thousand dollars in initial cost to install the classroom-wide response system, passing along the cost of the reusable handheld clicker device (between $10 and $40) to students. periodic maintenance as well as technical support costs are incurred by the college over time. clicker-based pedagogy has the potential to counteract the passive, teacher-centered instruction often encountered in mass classes of today’s colleges and universities. teachers and students may benefit from the real-time assessment of subject area knowledge. additionally, increased student engagement in class may result in increased perceptions of fun, providing motivation to return to class. although there is evidence to suggest that clickers increase powell, s., straub, c., rodriguez, j., and vanhorn, b. journal of the scholarship of teaching and learning, vol. 11, no. 4, december 2011. www.iupui.edu/~josotl 3 academic achievement, as measured by performance on tests of knowledge, as well as provide ability to assess student knowledge, and increase student perceptions of fun, it is also critical to understand student perceptions of the value of clickers as a learning tool. a. utilization of clicker-based pedagogy increases academic achievement. a growing body of research has been used to provide evidence for increased academic achievement, as measured by performance on tests of knowledge, using clickers. active learning has been linked to increased student achievement, which can be measured by final grades (united states department of education, 1996). a study of 11 parallel courses taught at the university of wisconsin over two years showed a statistically significant impact of clicker use on student academic achievement (kaleta & joosten, 2007). similarly, salmonson, andrew, and everett (2009) reported that increasing the engagement of non-participatory students through the use of clickers resulted in an increase in grades. el-rady (2006) found statistically significant differences in exam scores between classes compared in two consecutive semesters and furthermore found evidence to suggest that clickers improved student retention. in a comparison of clicker technology versus paper pencil technology, mayer et al. (2009) found that students who experienced in-class questioning and responded with paper and pencil scored significantly lower on class exams than those who responded using clickers. therefore, the research team hypothesized that the use of clickers would increase students’ performance. thus, the first research question is: rq1: to what extent does the use of clickers increase student academic achievement in a large, college-level introductory psychology class as measured by final grade percentage score? b. increased understanding of content through formative assessment. clickers ostensibly serve as vehicles to increase student understanding of content by providing instructors with the opportunity to take on the role of agile teachers who can quickly assess student comprehension and modify instruction to student needs (beatty, garace, leonard & dufrense, 2006; mayer et al, 2009; mula & kavanaugh, 2009). researchers distinguish between pedagogy and technology, highlighting the clicker’s role as a piece of formative assessment technology (beatty et al., 2006), which provides “college instructors with a non-intrusive, effective pedagogy and students with a more engaging learning format” (mula & kavanagh, 2009, p. 2). researchers suggest that clickers alone do not increase academic achievement, but rather the utilization of clickers increases opportunities for instructors to engage students by using interactive questioning methods which, in turn, are related to increased academic achievement (mayer et al., 2009). kaleta and joosten (2007) reported survey results yielding that 100% of instructors appreciated the ability to assess student knowledge and understanding, while 74% agreed or strongly agreed that clickers improved student learning. the research team hypothesized that students using clicker technology would perceive an increase in understanding of the topic based on the use of clickers. thus, the second research question is: rq2: did students using clicker technology in a large, college-level introductory psychology class perceive that clickers increased “understanding of the topic”? powell, s., straub, c., rodriguez, j., and vanhorn, b. journal of the scholarship of teaching and learning, vol. 11, no. 4, december 2011. www.iupui.edu/~josotl 4 c. using clickers to self-identify content knowledge deficits. while instructors report value in assessing understanding of content, clicker responses also provide valuable information for students to identify personal deficits in understanding. in an examination of meta-cognition skills of individuals, flavell (1979) identified limitations in the ability to self-monitor comprehension, reporting that individuals were often unable to identify holes in understanding. the use of clickers may reveal misconceptions to students; when instructors broadcast anonymous responses to questions using clickers, students are able to compare individual responses to the correct answers in an unobtrusive setting. in this way, each student has the opportunity to practice formative assessment at the individual level (beatty et al., 2006). student-identified deficits in content knowledge have the potential to allow students to create a customized, focused plan of study. clickers may provide one method for students to gather objective information about their learning. the research team hypothesized that students using clicker technology would perceive an increase in their ability to identify areas of deficit in their subject matter knowledge. thus, the third research question is: rq3: did students using clicker technology in a large, college-level introductory psychology class perceive that the use of clickers helped them find out what they “still needed to study”? d. using clickers to increase student perception of fun. although there is minimal research related to student perception of fun in a clicker-based classroom (fies & marshall, 2006), investigating student perceptions of clicker use is worthwhile to establish social validity of the utilization of clickers (wolf, 1978). beatty et al. (2006) writes, “by fostering an active, interactive classroom environment, classroom communication systembased pedagogy helps keeps students interested and attentive” (p. 6). in a large introductory psychology class, poirier and feldman (2007) found that students using clickers earned higher final exam scores and reported positive attitudes toward utilizing clickers in class. researchers report that students find clicker-based classes to be more fun than non-clicker based classes (burnstein & lederman, 2001; dufresne, gerace, leonard, mestre, & wenk, 1996; fies, 2005). based on these results, the research team hypothesized that students using clicker technology would perceive using clickers as fun. thus, the fourth research question is: rq4: did students using clicker technology in a large, college-level introductory psychology class perceive that the use of clickers to respond to questions was “fun”? e. financial value of clickers. a primary concern of colleges and universities using the clickers is the issue of cost. students enrolled in classes using clicker-based pedagogy are required to purchase the handheld clicker device that may cost between $10 and $40. students may re-use the device each semester, but if lost, students must purchase and register another. although some researchers report that students “appreciate the system’s value” (beatty, 2004, p.6), researchers of two recent literature reviews of clicker technology did not discuss whether or not students perceived the clicker was worth purchasing (fies & marshall, 2006; roschelle, abrahamson, & penuel, 2004). because students must incur the cost of utilizing clickers after the decision to invest in the system has already been powell, s., straub, c., rodriguez, j., and vanhorn, b. journal of the scholarship of teaching and learning, vol. 11, no. 4, december 2011. www.iupui.edu/~josotl 5 made by administrators, it is important to investigate whether or not students perceive the clicker is worth purchasing to establish social validity of the utilization of clickers (wolf, 1978). the research team hypothesized that students using clicker technology would indicate that they would be willing to purchase a clicker for a class to help them learn. thus, the fifth research question is: rq5: did students using clicker technology indicate that they would be “willing to purchase a clicker that cost $30 $40” to help them learn? ii. methodology. a. participants. participants in this study were enrolled in an urban public college located in the southeast united states in introductory psychology classes and were grouped according the section of the course in which they were enrolled. the same instructor taught both of the sections sampled for this study. the sample (n = 145) was established when students met the criteria of enrollment in the introductory course in psychology and completion of the final exam. students self-selected their participation in the study and were not provided monetary compensation, nor were they promised any benefits not normally associated with using a learning tool. participant information was numerically coded to maintain anonymity and all identifiable information was maintained in a secure location by the principal investigator. participation for this study totaled 183 students across two separate introductory psychology classes and students were assigned into control and treatment groups. of the 183 participating students, 145 students received a final grade in the course and were subsequently involved in the analysis. the control group did not receive clickers and consisted of 78 students from one class. the treatment group received clickers at varied intervals and consisted of 67 participants from two other classes. participants responded to a demographic questionnaire, information on gender was broken down as follows: 56 of the participants were male while 127 of the participants were female, while responses to questions about ethnicity were as follows: caucasian, 91; african–american, 52; hispanic, 24; asian, 2; other, 14. only 7 of the participants were under the age of 18, while the preponderance of the participants were between the ages of 18-24 (152), and 24 students were over 25 years old. b. instruments. primary outcomes were academic and measured by final class grades. for the purposes of this study, a common grading rubric was used for both of the introductory classes as defined in pasw as: a = 100-90%, b = 89-80%, c = 79-70%, d = 69-60%, and f = 59% or less. data on student perception was collected using a researcher-created questionnaire comprised of 11 items with responses on a 5-point likert scale. questions selected for use in this research study were as follows: a) using a clicker increased my understanding of the topic, b) using a clicker helped me find out what i still needed to study, c) using a clicker to respond to questions was fun, and d) i would be willing to purchase a clicker that cost $30-$40 to help me learn. likert responses were as follows: 1) i strongly disagree, 2) i somewhat disagree, 3) neutral, 4) i somewhat agree, and 5) i strongly agree. demographic information for all participants was collected using a 21-question survey at the beginning of the course. at the powell, s., straub, c., rodriguez, j., and vanhorn, b. journal of the scholarship of teaching and learning, vol. 11, no. 4, december 2011. www.iupui.edu/~josotl 6 completion of the course, the group who used the clicker technology completed a questionnaire that consisted of 10 items related to attitude toward using the clickers. c. data collection and analysis procedures. in this quasi-experimental study, measures of academic performance were compared across two groups of students: one class utilizing clickers (consisting of embedded questions and procedures for responding using clickers) and one class that did not utilize clickers. the same doctoral level professor with 10 years of experience taught each class, and each of the classes participating in this study met over the course of a semester. in all the courses, content was delivered via powerpoint and lecture in a face-to-face setting. for the course with embedded questions, the average percentage of question to content slides was 21%, and the professor utilized clickers throughout the delivery of the lecture. upon response to question, results were immediately displayed on a bar graph. if the distribution showed 10% or more of incorrect student responses, the feedback would be used to re-teach the topic by providing additional explanations or responding to questions. if more than 90% of students chose the correct response, the instructor would confirm the correct response, giving a brief explanation and then moving on. in both courses, online review quizzes allowed students to review content prior to administering class exams. the clickers system employed for the study was the iclicker manufactured in partnership with the worth-freeman publishing company. iii. findings. participants in all the psychology classes totaled 183 students. of the total student participation, 145 received a final grade for the course, 78 of the participants were in the control group and 67 participants were in the experimental group. due to missing data, 38 cases were not included in the analysis. thus, data was analyzed using 78.2% of the 183 total students (n=145). using predictive analytics software (pasw), the researchers compared the 38 missing cases to determine if the ratios of missing data were similar between groups. results yielded similar rates of missing data when compared across groups. of the 38 cases, 20 cases of missing data occurred in the control group (accounting for 20.5% of the total control group), while 18 cases of missing data occurred in the experimental group (accounting for a loss of 21.2% of the experimental group). inferential statistical analysis was performed for the first research question comparing experimental and control groups. descriptive statistics on the perception of clicker technology usage were collected on the experimental group only for research questions two through five. rq1: to what extent does the use of clickers increase student performance in a large, college-level introductory psychology class as measured by final grade percentage score? student performance was measured by final grades, defined by a percentage score of cumulative total points earned divided by cumulative total points possible. an independent t test was conducted to determine if there was a significant difference (p < 0.05) in mean score of final percentage course grades of students in a college-level introductory psychology class using a clicker system versus students who did not using a clicker system. the test was conducted using an alpha of 0.05. levene’s test indicated that the assumption of homogeneity of variances was met (f = 2.029, p = 0.157). the test was statistically significant, t(143) = 8.367, p < 0.05. students using the clicker system scored higher on average (m = 83.43, sd = 11.88) than powell, s., straub, c., rodriguez, j., and vanhorn, b. journal of the scholarship of teaching and learning, vol. 11, no. 4, december 2011. www.iupui.edu/~josotl 7 students not using the clicker system (m = 76.82, sd = 15.13). the 95% confidence interval for the difference between means was -11.13 to -2.09. the effect size was calculated by eta squared and found to be 0.055 indicating that approximately 6% of the variance in scores was accounted for by whether or not the student used clickers. approximately 81.27% of the treatment group (n = 67) received a grade of c or better compared to 76.92% of the non-treatment group (n = 78). the results provide evidence to support the hypothesis that students using clickers score higher in measures of academic performance. rq2: did students using clicker technology in a large, college-level introductory psychology class perceive that clickers increased “understanding of the topic”? in order to address the second research question, the questionnaire results (n = 67) from the treatment group were dichotomized based on responses. the responses of 1 or 2 were classified as “disagree” and the responses of 4 or 5 were classified as “agree,” and the response of 3 was neutral and not included in this comparison. based on analysis, approximately 91% of students perceive that clickers increased their understanding of a topic, leaving only 9% of students in disagreement with the statement that using clickers increased their understanding. the survey results provide evidence to support the hypothesis that the majority of students perceive that using clickers increase their understanding of the topic. rq3: did students using clicker technology in a large, college-level introductory psychology class perceive that the use of clickers helped them find out what they “still needed to study”? in response to the third research question and based on the dichotomized responses, approximately 81% of the students (n = 67) who responded to the survey agreed that clickers improved their ability to identify their individual deficits in the subject matter during the lecture. only approximately 6% disagree that clickers had an impact on their ability to identify their individual deficits in the content. once again, the survey results support the hypothesis that the majority of students perceive that clickers improved their ability to self-identify subject matter they still need to study. rq4: did students using clicker technology in a large, college-level introductory psychology class perceive that the use of clickers to respond to questions was “fun”? based on the dichotomized results of the survey, the fourth research question had an overwhelming positive response, with 100% of the students (n = 67) reporting that the use of clickers was fun as related to learning the content in an introductory psychology course. the results of the survey support the hypothesis that using clickers in the course was fun. rq5: did students using clicker technology indicate that they would be “willing to purchase a clicker that cost $30 to $40” to help them learn? finally, for the last research question, the responses were dichotomized, however, this time the majority, approximately 35% of those who responded to the survey (n = 67) disagree that purchasing a clicker was worthwhile, with approximately 32% of students agreeing that purchasing a clicker was worthwhile. the results of this question on this survey were not in line with the original hypothesis. iv. conclusion. overall, there is evidence to suggest that the use of clickers in a large, college level introductory psychology class contributed positively to learning. students using clickers performed significantly better when compared with a group of students not using clickers, although the powell, s., straub, c., rodriguez, j., and vanhorn, b. journal of the scholarship of teaching and learning, vol. 11, no. 4, december 2011. www.iupui.edu/~josotl 8 effect size was minimal. students using clicker technology reported that they felt utilization of clickers increased their understanding of the topic and allowed them to self-identify areas of deficit in their learning. all of the students in the experimental group agreed that using clickers to respond to questions was fun; however, only approximately one-third of those same students thought that purchasing a clicker was worthwhile. while these students may have benefited from and enjoyed using the clickers, they do not seem to be willing to bear the expense. this study contained limitations that may bring in to question the validity of the findings. the results may not be generalizable to other instructional settings because this research was conducted with large, college level introductory psychology classes. researcher bias may also exist, as this study was conceptualized and conducted by researchers who were also involved in delivering instruction to the experimental and control groups. another limitation in this study may be a consequence of the fact that the sample was not randomly selected and the conditions were not randomly assigned, leading to unequal groups. error in measurement may exist for research questions two through five, as the measure was researcher-generated and not fieldtested for reliability and validity. future research into the use of clickers should engage methods to compare procedures for embedding questions in lecture (e.g., pre-post lecture, throughout the lecture, etc.). to further explore the academic achievement of students using clickers, it would be worthwhile to link questions presented using the clicker system to questions on course tests, to determine if utilization of the clickers contributes to specific knowledge acquisition. examining the reliability and validity of the researcher created measure would decrease measurement error and lend credibility to the findings of students’ perceptions. v. implications for higher education administrators and faculty. since this research was conducted by professors engaging in scholarship of teaching and learning (boyer, 1990), the researchers’ aim was to improve instructional practice within the college, while at the same time informing purchasing decisions of clickers. currently, clickers are often used in large, college-level classes to increase student engagement, allowing teachers to poll classes by posing a question and providing an opportunity for students to respond in real-time via individual clickers. it appears that the utilization of clicker technology supports the generative theory of learning, by priming the cognitive processes used when learning, as demonstrated by improved academic performance. faculty of large, lecture-style classes seeking to engage students may be able to use clicker technology as one means of increasing generative learning, and therefore increasing student performance. yet, teachers, administrators, and students may question the efficacy of clickers because of the costs associated with adoption of the technology. students are often required to purchase clickers, which range in cost from $10 to $40, while administrators must adopt the clickers, which require a substantial financial investment. before assuming these costs and requiring students to make financial commitments, administrators are wise to ask whether or not the use of clickers increases student academic achievement, as well as what other options are available for formative assessment. other options, such as polleverywhere.com, provide instant audience feedback using mobile devices of audience members at significantly less expense (e.g., the cost of a text message). polleverywhere.com provides free templates for powerpoint slides which can be customized with questions about course content and embedded into existing powerpoint presentations. audience members are able to respond using sms texting for the cost of a text powell, s., straub, c., rodriguez, j., and vanhorn, b. journal of the scholarship of teaching and learning, vol. 11, no. 4, december 2011. www.iupui.edu/~josotl 9 message, with no cost to the higher education institution for groups of up to 30 participants at a time, and $65 per month for 250 participants at a time. as new technologies develop, higher education administrators, instructors, and students have increasingly diverse options to improve academic achievement. new developments in technology should be evaluated to determine if perceived benefits outweigh the costs. references anderson, l. w., krathwohl, d. r., airasian, p. w., cruikshank, k. a., mayer, r. e., pintrich, p. r., et al. (2001). a taxonomy of learning for teaching: a revision of bloom’s taxonomy of educational objectives. new york, ny: longman. beatty, i. (2004, february). transforming student learning with classroom communication systems. educause research bulletin, 2004(3), 1-13. beatty, i., garace, w., leonard, w., & dufrense, r. (2006). designing effective questions for classroom response system teaching. american journal of physics, 74(1), 31-39. boyer, e.l. (1990). scholarship reconsidered: priorities of the professoriate. princeton, nj: the carnegie foundation for the advancement of teaching. burnstein, r., & lederman, l. (2001). using wireless keypads in lecture classes. the physics teacher, 39, 8-11. dufresne, r.j., gerace, w.j., leonard, w. j., mestre, j. p., & wenk, l. (1996). classtalk: a classroom communication system for active learning journal of computing in higher education, 7, 3-47. edmonds, c.t., & edmonds, t.p. (2008). an empirical investigation of the effects of srs technology on introductory managerial accounting students. issues in accounting education, 23(3), 421-434. el-rady, j. (2006). to click or not to click: that's the question. innovate 2(4), 1-5. fies, c. (2005). classroom response systems: what do they add to an active learning environment? unpublished doctoral dissertation, the university of texas, austin, tx. fies, c., & marshall, j. (2006). classroom response systems: a review of the literature. journal of science education and technology, 15(1), 101–109. flavell, j.h. (1979). metacognition and cognitive monitoring: a new area of cognitivedevelopmental inquiry, american psychologist, 34, 906 911. geske, j. (1992). overcoming the drawbacks of the large lecture class. college teaching, 40(4), 151–154. powell, s., straub, c., rodriguez, j., and vanhorn, b. journal of the scholarship of teaching and learning, vol. 11, no. 4, december 2011. www.iupui.edu/~josotl 10 gleason, m. (1986). better communication in large courses. college teaching, 34(1), 20–24. hall, r., thomas, m., collier, h., & hilgers, m. (2005). a student response system for increasing engagement, motivation, and learning in high enrollment lectures. paper presented at eleventh americas conference on information systems, omaha, ne. hoekstra, a. (2008). vibrant student voices: exploring the effects of the use of clickers in large college courses. learning, media, and technology, 33(4), 329-341. kaleta, r., & joosten, t. (2007). student response systems: a university of wisconsin system study of clickers. boulder, co: educause center for applied research. kenwright, k. (2009). clickers in the classroom. techtrends, 53(1), 74-77. king, a. (1992). comparison of self-questioning, summarizing, and note taking review as strategies for learning from lectures. american educational research journal, 29, 303–323. mayer, r. e. (2001). multimedia learning. new york, ny: cambridge university press. mayer, r. e., stull, a., deleeuw, k., almeroth, k., bimber, b., chun, d., bulger, m., campbell, j., knight, a., & zhang, h. (2009). clickers in college classrooms: fostering learning with questioning methods in large lecture classes. contemporary educational psychology, 34(1), 51-57. mula, j.m., & kavanagh, m. (2009). click go the students, click-click-click: the efficacy of a student response system for engaging students to improve feedback and performance. e-journal of business education & scholarship of teaching, 3(1), 1-17. poirier, c. r., & feldman, r. s. (2007). promoting active learning using individual response technology in large introductory psychology classes. teaching of psychology, 34, 194-196. roschelle, j., abrahamson, l., & penuel, w. (2004). integrating classroom network technology and learning theory to improve classroom science learning: a literature synthesis. paper presented at the meeting of the american educational research association, san diego, ca. rosenshine, b., meister, c., & chapman, s. (1996). teaching students to generate questions: a review of intervention studies. review of educational research, 66, 181–221. salmonson, y., andrew, s., & everett, b. (2009). academic engagement and disengagement as predictors of performance in pathophysiology among nursing students. contemporary nurse, 32(1-2), 123-132. united states department of education. (1996). getting america’s students ready for the 21st century: meeting the technology literacy challenge, a report to the nation on technology and education. retrieved from http://www.ed.gov/about/offices/list/os/technology/plan/ national/ title.html powell, s., straub, c., rodriguez, j., and vanhorn, b. journal of the scholarship of teaching and learning, vol. 11, no. 4, december 2011. www.iupui.edu/~josotl 11 wittrock, m.c. (1974). learning as a generative process. educational psychologist, 19(2), 87– 95. wolf, m.m. (1978). social validity: the case for subjective measurement or how applied behavior analysis is finding its heart. journal of applied behavior analysis, 11(2), 203-214. journal of the scholarship of teaching and learning, vol. 23, no. 2, june 2023, pp.15-28. doi: 10.14434/josotl.v23i2.33756 why faculty underestimate low-income students’ family responsibilities elizabeth pierce wake forest university piercee@wfu.edu abstract: low-income college students face costly moral choices between pursuing their personal academic success and fulfilling their family responsibilities. they almost certainly face these choices more frequently and at greater personal cost than their faculty recognize. this article explores the sources and nature of that professorial lacuna; the article argues that this moral oversight results from the fact that middle-class people and low-income people often practice family in subtly but significantly different ways. they tend to emphasize different moral norms (independence vs. mutual aid) which shape the qualitative nature of college students’ obligations within their families. they also tend to utilize different family structures (nuclear vs. complex and extended) which create quantitative differences in the number of people to whom family responsibilities can attach. the paper ends with a practical implications section that discusses ways to address this lacuna so instructors can gain insight about their students’ familial obligations. keywords: class, family responsibilities, family structure, low-income college students “sorry for the background noise. i’m taking care of my nieces and nephews right now.” “may i turn in my draft tonight? i’ve been trying to stay up on my schoolwork and my job at the same time because my dad isn’t working right now, and there’s no telling when he’s going back, given all the covid stuff.” “sorry i stepped out of class earlier. that phone call was from my son’s school.”1 introduction low-income college students face costly moral choices between pursuing their own academic success and fulfilling their family responsibilities (morton, 2021). they almost certainly face these choices more frequently and at a greater personal cost than their faculty recognize. this article explores the sources and nature of that professorial lacuna, in the hopes that probing it will help us begin to ask better questions about our students’ moral situation. this article is also a reflection on my own teaching experience. twenty-something-year-old students have often impressed and sometimes astounded me by their commitment to fulfilling family obligations, like heading home for a few weeks when a mother’s partner got violent or missing a day of class to save family keepsakes from a storage unit foreclosure. of course, i noticed when these acute situations removed my students from the classroom. but what about all the other ongoing, subtler draws on their time, attention, and money: siblings who need childcare, parents who need emotional support, cousins who could use a little extra cash? i suspect these family responsibilities fly under most instructors' radars, including mine, still.2 faculty cannot begin crafting caring and just pedagogical responses to students’ often tragic trade-offs until we understand them better. more research is needed on that front. however, this 1 quotes from college students, aged 19-25 years old, at a first generation and low-income serving institution. 2 recent scholarship indicates these subtler family responsibilities are both common and important, materially impacting low-income graduation rates. see goldrick-rab (2016) and kinsley (2014). mailto:piercee@wfu.edu pierce journal of the scholarship of teaching and learning, vol. 23, no. 2, june 2023. josotl.indiana.edu article starts one step further back: what is it that prevents faculty from seeing and understanding their students’ family obligations in the first place? why are our low-income students’ family responsibilities so difficult to fully appreciate? this paper offers the beginning of an explanation of why we don’t know what we don’t know. in short: because of class.3 people from middle-class and low-income backgrounds often practice family in subtly but significantly different ways. these groups tend to emphasize different moral norms (independence vs. mutual aid) which shape the qualitative nature of college students’ obligations within their families. they also tend to utilize different family structures (nuclear vs. extended) which may create quantitative differences in the number of people to whom students’ family responsibilities can attach. because faculty usually come from middle-class (and often upper-class) backgrounds, it would be unsurprising if they brought middle-class expectations of family norms and structures with them into the classroom (nietzel, 2021; nunlee, 2016).4 this article attempts to show how these expectations could lead faculty to radically underestimate the depth and breadth of family responsibilities low-income students perceive for themselves. this article ends by briefly discussing three strategies for bringing our expectations into alignment with our students’ own moral commitments to their families. moral norms: individual autonomy vs. mutual aid young people growing up in middle-class and low-income families learn somewhat different lessons about who in their families is responsible to care for whom and how. for middle-class students, responsibility flows mostly downhill, from older to younger generations. middle-class parents shield children from weighty family responsibilities throughout their early to mid-twenties, enabling a period of personal exploration and professional development known as “emerging adulthood” (arnett, 2000). by contrast and often of necessity, low-income families tend to initiate their children into the work of caring for other family members’ material and emotional wellbeing at younger ages. the sections below explore the moral expectations reflected in these class-influenced parenting strategies. individual autonomy middle-class parents believe that one of their principal responsibilities is to help their children become independent. raising children who can take care of themselves and make decisions to secure their own well-being secures a family’s middle-class identity as much as it preserves the parents’ standard of living (zaloom, 2019, p. 7). 3 throughout this paper i use the term “middle-class” and “low-income” to describe different socio-economic groups. the terms are not ideal because they are asymmetrical. it would be preferable to use either “class” or “income” to describe both groups. still, for two reasons i have chosen to use this mismatched terminology. first, it strikes me as disrespectful to refer to any group of people as low-class; that phrasing has a moral bite to it, a demeaning judgment about people’s ontological worth. that language threatens to undermine my ability to describe this group’s life situation respectfully. also, i am not at all sure that most people with low incomes think of themselves as belonging to a shared cultural group that could be described accurately by the term “low-class.” by contrast, middle-class americans often embrace that cultural identity as such (nunlee, 2016). my use of the term “middle-class” captures this self-understanding, as well as the markers of income and educational attainment that characterize this group (reeves, guyot, and krause, 2018). in focusing on how class impacts faculty’s misperceptions of students’ family responsibilities, i do not mean to imply that class is the only or even the most important cause of misunderstanding. gender, race, nationality of origin, and/or ethnicity likely also play important roles. 4 faculty also tend to be white. seventy five percent of them were in 2017 (“race and ethnicity of college faculty”). 16 pierce journal of the scholarship of teaching and learning, vol. 23, no. 2, june 2023. josotl.indiana.edu so argues caitlin zaloom in indebted: how families make college work at any cost (2019). zaloom documents the lengths to which middle-class parents will go to provide their children with a college education: taking on student loan debt; deferring retirement investments; purchasing homes in neighborhoods they can barely afford in order to send their children to public k-12 schools where their children will receive adequate preparation for college. the reason middle-class families pay so dearly for their children’s college educations is that a 4-year degree is the sine qua non of financial security in the modern economy (at least, middle-class families believe that to be the case).5 a driving goal of middle-class parenting is securing their children’s long-term economic autonomy. educational decisions are not the only indicator of the middle-class’s commitment to intergenerational independence. one can also detect this norm by paying attention to the sources of worry and shame in middle-class family life. many middle-class parents fret—even as they write tuition checks—that their inadequate retirement savings will render them “burdens” to their children in old age (zaloom, 2019). other middle-class individuals feel shame when adult children move back home for some period after college. middle-class people often pathologize such housing decisions, treating them as “failure(s) to launch” rather than simply a matter of bad luck or a difficult housing market (hamlett, 2018; zaloom, 2019). middle-class families’ commitment to their children’s independence begins long before college application season. almost from birth, middle-class families nurture their children’s ability to steer their own life courses. parents teach children to make choices, practicing on minor decisions (would you like apple slices or fries with lunch?) so that children feel empowered to exercise choice in more important areas (which summer internship should i pursue?). middle-class parents also train their children to negotiate with authority figures (ask the doctor any questions you want, tell the teacher how you learn best) so they know how to advocate for themselves within important institutions (lareau, 2011). annette lareau’s classic text unequal childhoods: class, race, and family life describes the results of this parenting style: the white and black middle-class children in this study…exhibited an emergent sense of entitlement characteristic of the middle class. they acted as though they had the right to pursue their individual preferences and to actively manage interactions in institutional settings. they appeared comfortable in these settings; they were open to sharing information and asking for attention. although some children were more outgoing than others, it was common practice among middle-class children to shift interactions to suit their preferences. (2011, p. 6, italics original) middle-class families spend years “concerted[ly] cultivat[ing]” (lareau, 2011, p. 2) their children’s ability to look out for their own interests. then parents invest tens of thousands of dollars (money they may or may not have) in college educations to give their children a shot at economic independence. no doubt most of these middle-class parents expect their young adult children not to squander these efforts through hard partying and general slackness. for the very same reasons, it would be odd if these middle-class parents expected their children to shoulder significant family responsibilities, if doing so threatened their dearly bought academic opportunities. the moral lesson middle-class children might reasonably take from this parenting style is that their first and highest responsibility is to chart a successful professional path and take care of themselves. 5 this situation is not irony free. in promoting their children’s future financial independence, parents often make financial sacrifices that undermine their ability to remain financially independent of children in old age (zaloom, 2019). 17 pierce journal of the scholarship of teaching and learning, vol. 23, no. 2, june 2023. josotl.indiana.edu mutual aid low-income families often teach their children something quite different, that their responsibility is to take care of other family members. this is not to say that low-income parents teach their children to ignore their personal interests, but low-income families generally cannot afford to encourage the (extreme?) degree of individualism embraced by middle-class families. low-income families need everybody to pitch in with childcare, cooking, housecleaning, emotional support, and/or bill-paying. linda burton (2007) offers the concept of “adultification” (p. 336) to describe this phenomenon in low-income families. adultification happens when “youth are...exposed to adult knowledge and assume extensive adult roles and responsibilities within their family networks” (burton, 2007, p. 338).6 she outlines four degrees of adultification. 1) “precocious knowledge,” (burton, 2007, p. 336) experienced by the vast majority of burton’s sample, happens when children share in adult worries and frustrations, especially concerns about covering basic needs like food, housing, and healthcare. children become aware of these concerns by hearing adults discuss them, sometimes with each other, sometimes directly with their children. 2) in “mentored-adultification,” (burton, 2007, p. 338) children take on not only adult-like perspectives (worries about money and so forth) but also some limited adult-like tasks, conducting those tasks with little supervision from adults. for instance, a child in this stage of adultification might cook dinner for the family several nights a week and make sure the house is clean when their parents come home from work. 3) “peerification” (burton, 2007, p. 338) involves children assuming so many adult tasks and responsibilities that the hierarchy between parent and child starts to dissolve. parents may confide in their children as peers, and children may see themselves as holding equal status and authority as their parents. one quote from burton’s data summarizes this attitude: “my father and i bring the same amount of money into the house every month to pay the bills. so, i have the same rights he does. i can do what i want” (burton, 2007, p. 339). 4) the rarest and most intense level of adultification is “parentification” (burton, 2007, p. 339), in which children assume not just adult responsibilities but full time quasiparental responsibilities for their siblings and/or their own parents. while parentification almost always harms young people, other forms of adultification can be quite healthy. for instance, understanding the dangers of payday loan agencies (an example of precocious knowledge) may help a young person be appropriately weary when handling her own paychecks. likewise, youths whose parents have taught them to handle some important household task like making dinner (an example of mentored adultification), may express higher levels self-esteem and personal responsibility (burton, 2007). for our purposes, the most important point to draw from burton’s study is not whether adultification is good or bad for youth. rather, the concept of adultification merits attention because it highlights how adolescents in economically stressed households are likely to understand their obligations to care for family members. it is exceedingly common for low-income adolescents to be intimately aware of their families’ material and emotional needs, and they frequently see themselves as 6 this definition raises the question of what counts as “adult knowledge…roles and responsibilities” in the first place. burton recognizes that societies construct definitions of adulthood and childhood, that these are not natural categories. she utilizes common western institutional (and, importantly quite middle-class) understandings of children’s and adults’ roles and responsibilities because her article addresses social workers, teachers, and other representatives of these institutions. she wants to help these professionals see how low-income children are stuck between a “ ‘rock and a hard place’ as they navigate multiple worlds while simultaneously being considered a child by some ‘adult’ by others…for example, the assertive behaviors children exhibit in managing their homes can be viewed as assets by their parents and as liabilities when children exercise those same behaviors in the classroom” (2007, p. 331). 18 pierce journal of the scholarship of teaching and learning, vol. 23, no. 2, june 2023. josotl.indiana.edu at least somewhat able and responsible to provide for those needs. that sense of responsibility is instilled before they graduate high school, long before they start their freshman years of college.7 do low-income college students take this awareness of family needs and obligations with them to college? sarah goldrick-rab’s book, paying the price: college costs, financial aid, and the betrayal of the american dream (2016), touches on that question. her in depth interviews with low-income college students suggest that they do indeed carry a sense of family responsibility on to campus. as an example, goldrick-rab (2016) describes the experiences of student “ian william” (a pseudonym): growing up poor, the idea that family members would help each other out as needed was a given in williams’s home. “that’s how my mother raised us,” he explained in an interview, “if a piece of us falls, we all fall.” ...williams’s father made his expectations clear. “you’re always supposed to look out for your family, no matter how much money you’ve got”...when williams became a college student, his family’s longtime practice of sharing continued. he shared his limited funds—from grants, loans, and work—with his mother and brothers. the financial-aid system assumed that williams’s family was helping him by providing his [expected family contribution] of $425 annually, but his mom couldn’t make that payment. instead, williams paid the efc with student loans, and he used his financial aid to help her. he explained, “my mama was my motivation. she kept me out of trouble and that type of environment, even though it was hard for her because she was going to work most of the time … that’s why i’ve got to do something; i’ve got to help my mother out.” (pp. 149-50) the williams’ perspective was not particularly unusual in goldrick-rab’s study. of the students she surveyed who lived on campus, 11% reported providing $50 to their families per month and 14% reported contributing 10 hours per week to their family’s childcare and eldercare needs (p. 156). these numbers may be underreported. goldrick-rab observed that her interviewees tended to downplay their family contributions, treating them as simply what families are supposed to do for each other. more importantly, low-income students who live off campus likely contribute significantly more to their families’ financial and labor needs. we have here two contrasting moral visions for young adults’ responsibilities within their families. middle-class parents teach their children to take responsibility for their professional success (no small task in the modern labor market) even while parents (and sometimes grandparents) work hard to launch their young people into economic independence (zaloom, 2019). we can debate whether such an individualistic telos is a morally good idea, but this goal clearly holds great motivational power for middle-class families. low-income parents cannot afford to encourage their children to adopt such an individualistic focus. they need their children to understand and invest in other family members’ welfare. at the ages of 18-24 years old, low-income college students face the 7 burton’s article depicts low-income’s youth’s self-perception, but her study—a 20-year ethnographic project—only looked at the experiences of a relatively small sample of people. quantitative research, drawing on the panel study of income dynamics, child development supplement of 2002, and the transitions to adulthood study of 2007, paints a similar picture (kendig, mattingly, and bianchi, 2014). kendig and co-authors found that adolescents from poor families were more likely to assume adult responsibilities early and less likely to receive less financial support as they transitioned into young adulthood. eighteen to 24-years-olds from poor or low-income families were far more likely to be completely financially independent and more likely to be parents themselves than their peers who had never experienced poverty or low-income. these same young people were more likely to have contributed financially to their households as teenagers, taken care of their siblings emotionally and materially, and supported their parents in everyday tasks. not surprisingly, they were also more likely to worry about money and their futures. 19 pierce journal of the scholarship of teaching and learning, vol. 23, no. 2, june 2023. josotl.indiana.edu daunting task of balancing student loans, coursework, internships, work study jobs, other side jobs, and significant family responsibilities to nuclear and possibly extended family members. family structure: nuclear vs. complex and extended not only may many low-income students feel a (qualitatively) more developed sense of responsibility for the wellbeing of their family members than middle-class students, low-income students may also feel responsible for a (quantitatively) greater number of people. this quantitative difference results from a variance between low-income and middle-class families so basic it is likely to go unnoticed: family structure.8 middle-class people tend to build simple nuclear families. low-income families tend to form more complex families who, furthermore, place a greater emphasis on extended family connections. nuclear families one defining feature of middle-class identity—in addition to income and educational attainment—is a common pattern of family membership. that common pattern consists of two adults who marry (often cohabiting first), then bear/adopt children, and remain married for many years if not the rest of their lives (lundberg, pollak, stearns, 2016; wang, 2018). while this “first comes love, then comes marriage” pattern has declined since the 1970s, it was still prevalent enough in 2018 that 75% of children growing up in families in the three middle income quintiles were being raised by both of their parents (reeves and pulliam, 2020). marriage rates rise along with household income (reeves and pulliam, 2020), so professors’ children may be more likely than other middle-class children to grow up in two parent households, since professors tend to marry other professionals. a middle-class understanding of family members’ roles and responsibilities often attends this nuclear family structure. that understanding places heavy emphasis on parents’ responsibilities to care for their children, almost to the exclusion of any familial responsibilities beyond the nuclear family.9 consider below the narrative of a “normal” middle class family life offered by ara francis in family trouble (2015). the book tracks the experiences of middle-class parents caring for special needs children, whose “trouble” makes living out their life narrative impossible. this disjuncture reveals (usually implicit) expectations about what constitutes a “normal” family life: [p]arents imagined their own and children’s lives as unfolding along a particular trajectory. children were, parents assumed, headed toward a higher education, marriage, and children of their own. parents expected to take pride in their performances as mothers and fathers, to someday grow into adult relationships with their children, to become grandparents, and enjoy a retirement free of intensive caregiving. (francis, 2015, p. 11) 8 this quantitative difference in family size also results from the fact that, in the us, parents with the less education and income have more children per capita than parents with more education and more income (martinez, daniels, and febovazquez, 2018). so, even when considering only nuclear family size, low-income students are likely to come from slightly larger families. it is still important, however, to consider family membership size above and beyond the nuclear family. i suspect extra-nuclear relationships play a more important role in determining the number of family obligations students feel themselves to have. 9margaret nelson describes the boundaries drawn around white middle-class family membership as a “force field” (2020, p. ix-x) which invisibly but nonetheless powerfully hinders outsiders from getting too close emotionally or physically. 20 pierce journal of the scholarship of teaching and learning, vol. 23, no. 2, june 2023. josotl.indiana.edu this narrative (probably inaccurately) neglects the common enough middle-class expectation that adult children will need to care for aging parents.10 but note what else this narrative leaves out (probably accurately): it anticipates no familial obligations to great aunts and elderly neighbors, nieces and nephews, cousins and step siblings, and other extended family and family friends. the imagined family “network of care” is drawn tightly for middle-class people.11 this tight focus on the nuclear family may be a geographic necessity. young academics are especially notorious for taking whatever post-docs or assistant professorships they can land, wherever they can land them (“it’s hard to build…”, 2018). while panicked or gleeful trans-continental moves might be particularly common in academia, similar patterns show up among other educated professionals. people with college degrees move across state lines twice as often as people with only high school diplomas, and they move further away (bui and miller, 2015; molloy, smith, and wozniak, 2011). over 80% of couples without college degrees live within 30 miles of one of their mothers, compared with roughly 50% of couples with two college degrees (compton and pollak, 2009). those couples often live much, much farther away. (consider all the far-flung places where your graduate school peers landed.) there are several possible explanations for why college graduates move further from home, but the most interesting for our purposes is that college graduates tend to marry other college graduates. if both spouses want to be employed, they often need to move to a larger city and larger labor market where they can both find jobs (costa and kahn, 2000). in addition to the two-career pressure, college graduates are more likely to marry persons who grew up in different states (compton and pollack, 2009). if one spouse is from iowa and the other is from florida, they cannot possibly live near both sets of family. unless, of course, they embark on a long-distance marriage, a relationship form to which academics seem remarkably prone (walters, 2010). even in those cases, job locations rather than family proximity generally dictate spousal geography. regardless of the cause of their mobility, geographic distance makes it difficult for many welleducated couples to contribute or receive many hours of family labor within their extended kin networks. they cannot, at a distance, participate in the daily routines of family caregiving (taking aunt grace to her doctors’ appointment or helping nephew bill fix his junker first car), although they may make financial contributions to extended family members’ care (i.e. chipping in to refit mom’s house for wheelchair accessibility). the reverse is also true. extended family members cannot offer many hours of support to far away well-educated professionals. dual career parents know the burden of care for the little ones falls squarely and almost exclusively on their shoulders (and the shoulders of the people they hire to help).12 zoom calls, cards, shared family vacations do allow extended kin to remain emotionally connected (roschelle, 1997). but those avenues to emotional connection do not add up inevitably to material forms of interdependence. 10 thirty percent of americans express the belief that family members, rather than the government or the aging parents themselves, are responsible for aging parents’ care. seventy six percent of americans say that providing financial assistance to aging parents is an adult child’s responsibility. those statistics refer to the entire american populace, not just the middle or upper classes. still, the percentage of americans who express a sense of responsibility for aging parents is so high, it would be nearly impossible for a large proportion of middleor upper-class people not to share those sensibilities (pew research center’s social & demographic trends project, 2015). 11 middle-class families’ actual “network[s] of care” (hansen, 2004, p. 12) may be more expansive than their family ideologies suggest, but the ideology of the independent nuclear family remains strong. 12 and hire help they understandably do. the higher a family’s income, the more likely they are to pay for licensed childcare, rather relying on family members or possibly less expensive non-licensed providers. the more education a child’s mother has, the more money that her family tends to spend on childcare (malik, 2019). 21 pierce journal of the scholarship of teaching and learning, vol. 23, no. 2, june 2023. josotl.indiana.edu so, it makes a great deal of sense for well-educated, middle-class people, like professors, to imaginatively locate most family responsibilities within the nuclear family. furthermore, we should expect faculty persons’ deepest relationships to reinforce similar expectations, since most faculty were raised by college educated parents, and tend to marry college educated spouses (cohen, 2013).13 they also tend to stay married to these spouses (wang, 2015). in doing so, well-educated people either teach their children to value stable marriages, or show them how stable marriages work, or possibly both (amato and deboer, 2001). and, as discussed above, college educated people spend extraordinary sums of money to send their children to college, launching them into marriage and job markets that require geographic mobility. in sum, what we have here is a self-reinforcing cycle of college-educated, geographically-mobile nuclear families generating college-educated, geographically-mobile nuclear families. this is not to say that educated middle class families (faculty or otherwise) always live out their visions of family life (hansen, 2004). it is simply to say that the middle-class moral imagination readily perceives the responsibilities accrued in the nuclear family—especially childcare responsibilities. the nature of family responsibilities above and beyond the nuclear family may be murky to them. that murkiness makes it harder to anticipate, understand, and constructively respond to the conflicts our students face between academic responsibilities and family responsibilities arising outside the nuclear family. extended and complex families arise they likely will. membership structures of low-income families tend to differ in two ways from middle-class families: low-income families exhibit more complexity, and they place greater importance on extended family relationships.14 the cumulative effect of these differences is that low-income students’ family responsibilities will often be more numerous than their middle-class peers. there are simply more people to whom low-income may feel potentially responsible to provide care.15 family complexity to explain the meaning of “family complexity” it is perhaps easiest to start with a working definition of a “simple” nuclear family, which consists of two parents and their biological children living in the same household. a family grows more complex with every parental or sibling relationship that diverges from that form (manning, brown, and stykes, 2014). for example, the simpsons are a simple family (two parents who produce biological offspring) while the family julia roberts joins in the stepmom is slightly more complex (both children are biologically related to each other and the father but not the 13 in claiming that most faculty persons were raised by college educated people, i am extrapolating from the family backgrounds of all people who earned doctorates (national science foundation, 2018). that extrapolation is imperfect since not all professors have doctorates and not all people with doctorates become professors. 14 these two phenomena, family complexity and close ties with extended family, feel very similar, almost like two different ways of saying the same thing. they both depict a complex web of family relationships; there is a certain symmetry here. nonetheless, these phenomena are distinct. complexity refers to the composition of the nuclear family, while extended family ties refer to relationships beyond the nuclear family. it is possible for a simple, stable nuclear family to enjoy a rich network extended family and friends. likewise, members of a complex family—who may move frequently between households, parents, etc.—may have a harder time keeping up with extended relatives. 15 average household size varies only a little bit by class, as measured by educational attainment. the average number of people in the most educated households is 2.51; the average number of people in the least educated households is 2.78 (“america’s families and living arrangements: 2016,” 2016). however, average household size is not the same thing as average family size, since important members of one’s family—parents, siblings, half-siblings, extended family members— may reside outside one’s household. 22 pierce journal of the scholarship of teaching and learning, vol. 23, no. 2, june 2023. josotl.indiana.edu stepmother). the brady bunch is more complex (three children are biologically related to each other but not to the other three children, and all children are biologically related to one parent but none are related to both parents). the kardashian/jenner clan is more complex still, involving full, step, and half-siblings growing up among parents, step-parents, and former step-parents. generally, harder-todraw family trees mean more complexity. as evinced by the examples above, family complexity usually results from “union instability” (musick and michelmore, 2018, p. 1390). union in/stability is a measure of the duration of intimate cohabiting or married partnerships. union stability correlates strongly positively with education; college educated people tend to get married and stay married; less educated people marry and later divorce or cohabitate and later separate with higher frequency (musick and michelmore, 2018). this is not to say more educated or higher earning people value commitment to their romantic partners more. survey results from more than 6000 floridian respondents indicate that low-income, middle-income, and high-income people are quite similar in their expressed approval for marriage, disapproval of divorce, and personal trouble in navigating conflict in intimate relationships (trail and karney, 2012). moral commitments regarding marriage do not differ much by class. other stressors— chronic financial precarity, for instance—provide a better explanation for higher divorce rates among low-income couples than attitudinal differences (trail and karney, 2012). family complexity, almost by definition, tends to expand the size of a family, as step-parents add themselves and often step-children to the household (at least part time). these family expansions may generate both more resources and more responsibilities for college students. perhaps a stepmother will help pay for her stepson’s text books. likewise, if a stepsister needs a place to stay for a couple of weeks, a student might feel obliged to offer her couch. complex families produce more potential—and sometimes more actual—relationships of responsibility than nuclear families produce. extended family “family complexity” may not, however, be the most relevant concept for understanding low-income students’ family structures. by focusing on nuclear families (simple or complex), the family complexity literature neatly sidesteps another important set of kin relationships: extended family and close family friends. these relationships are harder to study through census records and other traditional survey instruments. still, the fact these relationships elude easy measurement does not ipso facto render them less important to low-income students. writing in the wake of the moynihan report, carol stack’s oft-cited 1974 ethnography all our kin: strategies for survival in a black community certainly argues otherwise. stack highlighted the critical role extended family and friend networks could play in sustaining low-income african american families, materially and emotionally. her book kicked off waves of research along similar lines, including dan clawson’s and naomi gerstel’s unequal time: gender, class, and family in employment schedule (2014). unequal time probes the relationship between a family’s labor schedule and a family’s membership structure. it offers the concept of the “web of time” (clawson and gerstel, 2014, p. 3) to show how the most powerful ties that bind are not necessarily ties of blood or marriage. rather, they are the ties tethering one person’s schedule to multiple other persons’ schedules. for low-income workers, family is the girlfriend of your cousin who helps take care of your children while you’re at work. or, family is the sister-in-law you take to medical appointments because you own the only car in the extended family. family might even be the co-worker who picks up your shift when your toddler comes down with a fever. being family is less about sharing a roof or a last name than it is about sharing a calendar. unequal time implies (though does not explicitly argue) that the same rule applies for wealthier workers: kin are the people you rely on for no pay in a time crunch. it’s just that wealthier families 23 pierce journal of the scholarship of teaching and learning, vol. 23, no. 2, june 2023. josotl.indiana.edu tend to deal with time crunches within the nuclear family. a parent—often a mom—takes off work to stay home with a sick kid. she may be the person responsible to make sure the family vehicles are newish and well enough maintained to avoid broken-down car trouble. extra income and, to some extent, traditional gender norms enable middleand upper-class families to meet family labor needs in house or through market-based relationships rather than affective ones. a particularly clear example of this class-based definition of family is clawson and gerstel’s observations of certified nursing assistants’ responses to their employer’s family bereavement policies. cnas objected that their employers treated the loss of a partner or child as more important than the loss of a grandparent or sibling, awarding employees 3 and 2 days of leave respectively. of course, cnas objected to the short time allowed them for grieving. (consider: would you be able to return to work three days after your child died?) but that was not their only or even their primary objection; they also protested the assumption that nuclear family relationships were more important than extended family relationships: “i need a week for my grandmother!” “and only two days for my sister?” “at least, say not a difference in days for these [extended family relationships compared to nuclear family relationships]” (2015, p. 172). one can understand employers’ desire to draw neat lines around who counts as “family.” if we take a step back and look at the bereavement policies from a mathematical standpoint (a morally dubious exercise), the cnas’ proposal could multiply the number of bereavement days they take. more frequent bereavement leave creates more paperwork for hr, more hassle for the scheduling manager, and reductions in profit margins. faculty and university administrators face similar pressures. should we grant paper extensions when a student loses a cousin? if a student wants to take a midsemester leave-of-absence to grieve a great-aunt, should the financial aid department allow it? such decisions could cost us time, hassle, and money. how many of these relationships can our students have? possibly quite a few. we should expect that many low-income students will have grown up depending on extended kin and close family friends for care. it makes sense that they would see these family members as people to whom they owe concern and investment, that they would feel both affective and moral connections to this social web. the relational threads of this web may very well be more numerous than nuclear families’ tightly bound knots of care. like complex family structures, extended family structures may expand the number of persons to whom students feel a sense of caregiving obligation. in sum, compared to middle-class students, we should anticipate that many low-income students will possess (quantitatively) more family relationships and (qualitatively) deeper kinds of responsibility within those relationships. their family obligations could include time for child or elder care, emotional attention, and/or financial support. even when family members do not call upon their college students to fulfill family obligations, low-income students may very well carry an extra burden of stress. knowing their families could use their help at home—even if they do not ask for it—could make it harder to invest a summer in an unpaid internship or pay a little extra attention to their statistics homework. these demands on students’ “bandwidth” (verschelden, 2017) are often subtle. it would be unsurprising if middle-class professors missed them entirely. while understandable, this lacuna can still undermine our ability to support our students’ learning. implications for practice what are we to do with this mismatch of moral visions? middleand upper-class faculty’s first task is to identify strategies that will help us better understand our students’ moral experiences. reducing the trade-offs students face between fulfilling academic and family obligations should be our ultimate but 24 pierce journal of the scholarship of teaching and learning, vol. 23, no. 2, june 2023. josotl.indiana.edu also secondary goal, to be pursued after faculty and institutions acquire an understanding of the particular kinds of trade-offs their specific student populations face. the most immediate action faculty can take is to begin talking with their students about their family responsibilities. broaching this subject is not easy or ethically straightforward. prying into students’ personal lives is morally bad, and students should not feel that their grades depend on building rapport with faculty by sharing private information. on the other hand, students often need to know that their instructors care about the challenges they face outside the classroom; they need to know that family obligation is an acceptable topic to broach with their instructors.16 one way of broaching this topic is to lead students through a general “values affirmation” activity (verschelden, 2017, p. 79) at the beginning of a course. while such activities can take many forms, one common practice involves inviting students to select from a list of key values their top ten, then narrowing that list to their top three values, and then writing a letter describing “why these values are important to you and what difference they have made in your life. give some examples of things you have done or choices you have in your life based on these three values” (verschelden, 2017, p. 82). students, especially low-income students, can gain multiple benefits from this simple activity, including better grades, higher retention rates, and improved subjective well-being (verschelden, 2017, p. 79). just as importantly for our purposes, this activity could improve student-faculty communication about family responsibilities (and students’ other motivating concerns) in two ways. first, it provides faculty with critical information about students’ overall value commitments, including but not limited to family responsibilities. this understanding could help us better anticipate generally when students’ academic goals and other values might be in tension. second, spending class time on a valuesaffirmation activity communicates to students that faculty genuinely care about their moral concerns. this care may may make it easier for students to engage faculty in problem-solving when they face trade-offs between fulfilling family and academic responsibilities. for instance, if a student has already told her accounting instructor that the desire to improve her family’s economic situation motivated her to study finance and that student felt understood and affirmed by her instructor in that commitment, then that student might feel more comfortable requesting an extension on a paper when she needs to spend the weekend caring for a sibling rather than writing a paper due monday. valuesaffirmation activities are not magic bullets, but they do demonstrate the kind of conversation-opening practices faculty can utilize to better grasp students’ moral situations. in addition to speaking with students directly about their values and family obligations, faculty can gain insight by talking with student affairs staff, particularly those who work closely with firstgeneration college students. in order to support these students’ academic personal and success, staff need to grasp first-generation students’ “lived experience” of navigating a social environment that is more-or-less foreign to their families of origin (troy et al., 2022, p. 1). these staff take note of which students spend most weekends off-campus, taking care of family needs, how students spend financial aid checks, not just on books but on meals for family members, why students worry about representing their families well, etc. (ibid, 2, 7). these professionals can interpret students’ lived experiences to their faculty, but not all institutions facilitate those kinds of conversations. faculty forums, lunch-andlearns, communities of practice, and other gathering types could be used to regularize discussion among faculty and student affairs staff about students’ family obligations. last but not least, faculty can contribute to research that systematically compares faculty’s perceptions of their students’ family obligations with low-income students’ own perceptions. some 16 in this respect, faculty are in a similar position to workplace managers. we have an opportunity to invite negotiation and constructive problem solving so that school/workplace tasks can be fulfilled without compromising the student/worker’s family’s wellbeing. but maintaining proper boundaries, protecting family from surveillance by the teacher or employer, is an ongoing challenge. 25 pierce journal of the scholarship of teaching and learning, vol. 23, no. 2, june 2023. josotl.indiana.edu research exists regarding first generation and low-income students’ understanding of their roles in their families (berg, 2010, p. 70; mitchall & jaeger, 2018) though there is certainly much work still to be done on this front (bettencourt et al., 2022). faculty perceptions of low-income students’ family responsibilities seem to have received little or no attention. likewise, i have not found any research comparing these two vantage points (though i would be happy to be corrected). given this dearth of research, faculty who underestimate low-income students’ family responsibilities are likely in very good, and certainly very numerous, company. to improve our understanding of students’ family obligations, we need not wait for the emergency of academic research (welcome though it is). we can begin improving our praxis through conversations about students’ family obligations on our campuses, among experienced faculty, student affairs professionals, and students themselves. references amato, p. r., & deboer, d. d. (2001). the transmission of marital instability across generations: relationship skills or commitment to marriage? journal of marriage and family, 63(4), 1038– 1051. https://doi.org/10.1111/j.1741-3737.2001.01038.x arnett, j. j. (2000). emerging adulthood. a theory of development from the late teens through the twenties. the american psychologist, 55(5), 469–480. beatty, b. j. (2019). hybrid-flexible course design. edtech books. https://edtechbooks.org/hyflex berg, g. a. (2010). low-income students and the perpetuation of inequality: higher education in america. taylor & francis group. http://ebookcentral.proquest.com/lib/wfu/detail.action?docid=483652 bettencourt, g. m., mansour, k. e., hedayet, m., feraud-king, p. t., stephens, k. j., tejada, m. m., & kimball, e. (2022). is first-gen an identity? how first-generation college students make meaning of institutional and familial constructions of self. journal of college student retention: research, theory & practice, 24(2), 271–289. https://doi.org/10.1177/1521025120913302 brown, s., & manning, w. (2009). family boundary ambiguity and the measurement of family structure: the significance of cohabitation. demography, 46, 85–101. https://doi.org/10.1353/dem.0.0043 bui, q., & miller, c. c. (2015, december 23). the typical american lives only 18 miles from mom. the new york times. https://www.nytimes.com/interactive/2015/12/24/upshot/24up-family.html, https://www.nytimes.com/interactive/2015/12/24/upshot/24up-family.html burton, l. (2007). childhood adultification in economically disadvantaged families: a conceptual model. family relations, 56 (4), 329–345. https://doi.org/10.1111/j.1741-3729.2007.00463.x clawson, d., gerstel, n., & russell sage foundation staff. (2014). unequal time: gender, class, and family in employment schedules. russell sage foundation. http://ebookcentral.proquest.com/lib/wfu/detail.action?docid=4003853 cohen, p. (2013, april 4). college graduates marry other college graduates most of the time. the atlantic. https://www.theatlantic.com/sexes/archive/2013/04/college-graduates-marryother-college-graduates-most-of-the-time/274654/ compton, j., & pollak, r. a. (2009). proximity and coresidence of adult children and their parents: description and correlates. ssrn electronic journal. https://doi.org/10.2139/ssrn.1513453 costa, d. l., & kahn, m. e. (2000). power couples: changes in the locational choice of the college educated, 1940–1990. the quarterly journal of economics, 115 (4), 1287–1315. https://doi.org/10.1162/003355300555079 26 https://doi.org/10.1111/j.1741-3737.2001.01038.x https://edtechbooks.org/hyflex http://ebookcentral.proquest.com/lib/wfu/detail.action?docid=483652 https://doi.org/10.1177/1521025120913302 https://doi.org/10.1353/dem.0.0043 https://www.nytimes.com/interactive/2015/12/24/upshot/24up-family.html,%20https:/www.nytimes.com/interactive/2015/12/24/upshot/24up-family.html https://www.nytimes.com/interactive/2015/12/24/upshot/24up-family.html,%20https:/www.nytimes.com/interactive/2015/12/24/upshot/24up-family.html https://doi.org/10.1111/j.1741-3729.2007.00463.x http://ebookcentral.proquest.com/lib/wfu/detail.action?docid=4003853 https://www.theatlantic.com/sexes/archive/2013/04/college-graduates-marry-other-college-graduates-most-of-the-time/274654/ https://www.theatlantic.com/sexes/archive/2013/04/college-graduates-marry-other-college-graduates-most-of-the-time/274654/ https://doi.org/10.2139/ssrn.1513453 https://doi.org/10.1162/003355300555079 pierce journal of the scholarship of teaching and learning, vol. 23, no. 2, june 2023. josotl.indiana.edu fomby, p., & kravitz-wirtz, n. (2019). family systems and parents’ financial support for education in early adulthood. demography, 56(5), 1875–1897. https://doi.org/10.1007/s13524-019-00807-0 francis, a. (2015). family trouble: middle-class parents, children’s problems, and the disruption of everyday life. rutgers university press. http://ebookcentral.proquest.com/lib/wfu/detail.action?docid=4414543 goldrick-rab, s. (2016). paying the price: college costs, financial aid, and the betrayal of the american dream (illustrated edition). university of chicago press. hamlett, m. (2018, february 28). why do we still shame adults who live with their parents? the guardian. http://www.theguardian.com/commentisfree/2018/feb/28/michael-b-jordanlive-with-parents-millennials-shame hansen, k. v. (2004). not-so-nuclear families: class, gender, and networks of care. rutgers university press. it’s hard to build a life when you need to move cities for an academic career (2018, october 26). the guardian. http://www.theguardian.com/education/2018/oct/26/its-hard-to-build-a-lifewhen-you-need-to-move-cities-for-an-academic-career kendig, s. m., mattingly, m. j., & bianchi, s. m. (2014). childhood poverty and the transition to adulthood. family relations, 63 (2), 271–286. https://doi.org/10.1111/fare.12061 kinsley, p. m., (2014). the pull of home: family dynamics and the initial college experiences of low-income undergraduates (doctoral dissertation). university of wisconsin, madison. lareau, a. (2011). unequal childhoods: class, race, and family life, 2nd edition with an update a decade later. university of california press. lundberg, s., pollak, r. a., & stearns, j. (2016). family inequality: diverging patterns in marriage, cohabitation, and childbearing. journal of economic perspectives, 30(2), 79–102. https://doi.org/10.1257/jep.30.2.79 malik, r. (2019). working families are spending big money on child care. center for american progress. retrieved july 30, 2020, from https://www.americanprogress.org/issues/earlychildhood/reports/2019/06/20/471141/working-families-spending-big-money-child-care/ manning, w. d., brown, s. l., & stykes, j. b. (2014). family complexity among children in the united states. the annals of the american academy of political and social science, 654(1), 48–65. https://doi.org/10.1177/0002716214524515 martinez, g. m., daniels, k., & febo-vazquez, i. (2018). fertility of men and women aged 15–44 in the united states: national survey of family growth, 2011–2015 (no. 113; national health statistics report, p. 17). centers for disease control. https://www.cdc.gov/nchs/data/nhsr/nhsr113.pdf molloy, r., smith, c. l., & wozniak, a. (2011). internal migration in the united states. the journal of economic perspectives, 25(3), 173–196. jstor. morton, j. (2021). moving up without losing your way: the ethical costs of upward mobility. princeton university press. musick, k., & michelmore, k. (2018). cross-national comparisons of union stability in cohabiting and married families with children. demography, 55 (4), 1389–1421. https://doi.org/10.1007/s13524-018-0683-6 national science foundation, (2018) survey of earned doctorates. national science foundation. retrieved february 18, 2021, from https://ncses.nsf.gov/pubs/nsf20301/report/path-to-thedoctorate#parental-education nelson, m. k. (2020). like family: narratives of fictive kinship. rutgers university press. nietzel, m. t. (2021). the well-heeled professoriate: socioeconomic backgrounds of university faculty. forbes. retrieved june 4, 2021, from 27 https://doi.org/10.1007/s13524-019-00807-0 http://ebookcentral.proquest.com/lib/wfu/detail.action?docid=4414543 http://www.theguardian.com/commentisfree/2018/feb/28/michael-b-jordan-live-with-parents-millennials-shame http://www.theguardian.com/commentisfree/2018/feb/28/michael-b-jordan-live-with-parents-millennials-shame http://www.theguardian.com/education/2018/oct/26/its-hard-to-build-a-life-when-you-need-to-move-cities-for-an-academic-career http://www.theguardian.com/education/2018/oct/26/its-hard-to-build-a-life-when-you-need-to-move-cities-for-an-academic-career https://doi.org/10.1111/fare.12061 https://doi.org/10.1257/jep.30.2.79 https://www.americanprogress.org/issues/early-childhood/reports/2019/06/20/471141/working-families-spending-big-money-child-care/ https://www.americanprogress.org/issues/early-childhood/reports/2019/06/20/471141/working-families-spending-big-money-child-care/ https://doi.org/10.1177/0002716214524515 https://www.cdc.gov/nchs/data/nhsr/nhsr113.pdf https://doi.org/10.1007/s13524-018-0683-6 https://ncses.nsf.gov/pubs/nsf20301/report/path-to-the-doctorate#parental-education https://ncses.nsf.gov/pubs/nsf20301/report/path-to-the-doctorate#parental-education pierce journal of the scholarship of teaching and learning, vol. 23, no. 2, june 2023. josotl.indiana.edu https://www.forbes.com/sites/michaeltnietzel/2021/03/28/the-well-heeled-professoriatesocioeconomic-backgrounds-of-university-faculty/ nunlee, m. (2016). when did we all become middle class? taylor & francis group. http://ebookcentral.proquest.com/lib/wfu/detail.action?docid=4626175 pew research center’s social & demographic trends project (2015, may 21). supporting family members. pew research center. https://www.pewresearch.org/social-trends/2015/05/21/2supporting-family-members/ reeves, r.v., guyot, k., & krause, e. (2018, may 7). defining the middle class: cash, credentials, or culture? brookings. https://www.brookings.edu/research/defining-the-middle-class-cashcredentials-or-culture/ reeves, r. v. & pulliam, c. (2020, march 11). middle class marriage is declining, and likely deepening inequality. brookings. https://www.brookings.edu/research/middle-classmarriage-is-declining-and-likely-deepening-inequality/ roschelle, a. r. (1997). no more kin: exploring race, class, and gender in family networks. sage publications. http://ebookcentral.proquest.com/lib/wfu/detail.action?docid=1598450 stack, c. b. (1974). all our kin: strategies for survival in a black community ([first edition].). harper & row. trail, t. e., & karney, b. r. (2012). what’s (not) wrong with low-income marriages. journal of marriage and family, 74(3), 413–427. https://doi.org/10.1111/j.1741-3737.2012.00977.x troy, c., jackson, k., pearce, b., & rowe, d. (2022). laying the foundation for creating firstgeneration student programming. in developing and implementing promising practices and programs for first-generation college students. taylor & francis group. http://ebookcentral.proquest.com/lib/wfu/detail.action?docid=7070746 united states census bureau, (2016). america’s families and living arrangements: 2016. the united states census bureau. retrieved august 8, 2021, from https://www.census.gov/data/tables/2016/demo/families/cps-2016.html verschelden, c. (2017). bandwidth recovery: helping students reclaim cognitive resources lost to poverty, racism, and social marginalization (illustrated edition). stylus publishing. walters, j. (2010, march 2). degrees of separation. the guardian. http://www.theguardian.com/education/2010/mar/02/academic-career-live-apart-couple wang, w. (2015, december 4). the link between a college education and a lasting marriage. pew research center. https://www.pewresearch.org/fact-tank/2015/12/04/education-andmarriage/ zaloom, c. (2019). indebted: how families make college work at any cost. princeton university press. http://ebookcentral.proquest.com/lib/wfu/detail.action?docid=5802071 28 https://www.forbes.com/sites/michaeltnietzel/2021/03/28/the-well-heeled-professoriate-socioeconomic-backgrounds-of-university-faculty/ https://www.forbes.com/sites/michaeltnietzel/2021/03/28/the-well-heeled-professoriate-socioeconomic-backgrounds-of-university-faculty/ http://ebookcentral.proquest.com/lib/wfu/detail.action?docid=4626175 https://www.pewresearch.org/social-trends/2015/05/21/2-supporting-family-members/ https://www.pewresearch.org/social-trends/2015/05/21/2-supporting-family-members/ https://www.brookings.edu/research/defining-the-middle-class-cash-credentials-or-culture/ https://www.brookings.edu/research/defining-the-middle-class-cash-credentials-or-culture/ https://www.brookings.edu/research/middle-class-marriage-is-declining-and-likely-deepening-inequality/ https://www.brookings.edu/research/middle-class-marriage-is-declining-and-likely-deepening-inequality/ http://ebookcentral.proquest.com/lib/wfu/detail.action?docid=1598450 https://doi.org/10.1111/j.1741-3737.2012.00977.x http://ebookcentral.proquest.com/lib/wfu/detail.action?docid=7070746 https://www.census.gov/data/tables/2016/demo/families/cps-2016.html http://www.theguardian.com/education/2010/mar/02/academic-career-live-apart-couple https://www.pewresearch.org/fact-tank/2015/12/04/education-and-marriage/ https://www.pewresearch.org/fact-tank/2015/12/04/education-and-marriage/ http://ebookcentral.proquest.com/lib/wfu/detail.action?docid=5802071 microsoft word hoyertfinal.doc journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006, pp. 1 13. a brief intervention to aid struggling students: a case of too much motivation? mark sudlow hoyert and cynthia d. o’dell1 abstract: this study examines two interventions for altering achievement goals in an effort to enhance academic success in struggling introductory psychology students. the procedures involved an in-class lecture and an interactive computerized tutorial. both procedures were successful in altering motivation. this led to changes in study activities and increased examination and course grades. in fact, the rate of failure was cut in half in the students who completed the interventions. however, the procedures were not as successful at attracting students who were at risk for academic failure. only 12% of the target students completed the first intervention while 55% of the target students completed the computerized tutorial. participation in the computerized intervention was enough to markedly reduce the number of failing grades earned by the class as a whole. additions and alterations to the interventions are proposed to extend their efficacy. keywords: achievement goals, goal orientation, academic success, mastery, performance i. introduction many teachers firmly believe that motivation is important to the learning process. most teachers believe that a minimal amount of motivation is necessary and that higher amounts of motivation are better. teachers often attribute problems in the learning process to a lack of motivation. in this paper, we explore some of the links between motivation and academic success. our work is informed by a theory, alternatively known as achievement motivation theory or goal orientation theory, which posits that multiple varieties of motivation exist and that moderate levels of some varieties are preferable to high levels of other varieties. in fact, some of the problems in the learning process may result from high levels of particular kinds of motivation. goal orientation theory suggests that when students engage in a class, they strive to reach one or more goals (ames & archer, 1988; dweck & leggett, 1988; harackiewicz, barron, & elliot, 1998; hidi & harackiewicz, 2000). two goals are of primary importance: mastery goals and performance goals. students who adopt mastery goals are interested in learning the material in the class and strive to master that material (ames, 1992; blumenfeld, 1992; dweck & leggett, 1988). students who pursue performance goals are interested in demonstrating their competence, especially relative to other students (dweck & leggett, 1988; urdan, 1997). mastery and performance goals are multidimensional motivational constructs. that is, these goals provide a framework through which a variety of behavioral, cognitive and affective responses are energized and directed (ames, 1992; dweck & leggett, 1988; blumenfeld, 1992). for instance, research under a variety of laboratory and classroom settings has found that students who pursue mastery goals display a wide variety of largely adaptive behaviors and 1 respectively, professor of psychology, indiana university northwest, mhoyert@iun.edu, and associate professor of women’s studies and psychology, indiana university northwest, codell@iun.edu. address correspondence to mark hoyert, psychology department, indiana university northwest, gary, in 46408, 219-980-6687. hoyert, m.s. and o’dell, c.d. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 2 attitudes not seen in all students. mastery oriented students seek to improve their competence through acquiring new skills and knowledge and by surmounting novel and difficult problems. mastery goals have been found to be associated with increased interest, the enjoyment of challenge and challenging tasks, and the belief that competence is obtained incrementally through effort. students who pursue mastery goals commonly use effective learning strategies such as elaboration and organization, have developed multiple strategies, are interested in developing new skills, become involved in the learning process, display greater persistence, and are likely to respond to challenges through the use of greater effort and the exploration of alternative learning strategies (albaili, 1998; ames, 1992; ames & archer, 1988; blumenfeld, 1992; elliot & harackiewicz, 1994; graham & golan, 1991, harackiewicz, barron, carter, lehto, & elliot, 1997; harackiewicz, barron, tauer, carter, & elliot, 2000; harackiewicz & elliot, 1998; pintrich, zusko, schiefele, & pekrum, 2001). the pursuit of performance goals is more complex, but sometimes has been associated with a less adaptive set of academic outcomes such as self-aggrandizing, task aversion, the pursuit of effort minimizing strategies, a reluctance to seek help, impaired problem solving, greater feelings of self-consciousness, self-handicapping, and helplessness. students who adopt a performance goal orientation seek to elicit favorable judgments of their competence and avoid negative evaluations. these individuals tend to prefer and to seek out easier tasks where success and validation can be obtained and view competence as static and unaffected by effort. the maladaptive behaviors are more likely to appear when validation is not available (albaili, 1998; dweck, 1999; dweck & leggett, 1988; graham & golan, 1991; harackiewicz, barron, & elliot, 1998; harackiewicz, barron, tauer, carter, & elliot, 2000, harackiewicz & elliot, 1993; kong & hau, 1996; midgley, 1993; pintrich & schunk, 1996; pintrich, zusko, schiefele, & pekrum, 2001; ryan, gheen, & midgley, 2000; somuncuoglu & yildirim, 1999; urdan, kneisel, & mason, 1999; urdan & maehr, 1995; urdan, midgely, & anderman, 1998). the adoption of mastery goals should be associated with academic accomplishment. in the college classroom, this would translate into better examination and course grades and a higher grade point average. one would expect that a mastery oriented student who monitors comprehension, connects new information with old, can discriminate more important information from less important information, who sets goals, uses elaboration and other adaptive learning strategies should attain higher levels of academic success than a performance oriented student who procrastinates and self-handicaps. however, most studies employing an objective measure of academic performance in the classroom have not found a consistent or robust advantage. only a third of these studies have reported higher grades for mastery oriented students (bouffard, boisvert, vezeau, & larouche, 1995; eppler & harju, 1997; pintrich, zusko, schiefele, & pekrum, 2001; schraw, horn, thorndike-christ, & bruning, 1995; wolter, yu, & pintrich,1996) and about two thirds have obtained null results (beck, rorer-woody, & pierce, 1991; elliot & church, 1997; harackiewicz & elliot, 1998; harackiewicz, barron, carter, lehto, & elliot, 1997; harackiewicz, barron, tauer, carter, & elliot, 2000; harju & eppler, 1997; greene & miller, 1996; pintrich & garcia, 1991; pintrich & garcia, 1993; roedel & schraw, 1995). no studies have found that pursuing mastery goals is associated with declining academic success. we have identified a set of conditions under which the relative pursuit of mastery and performance goals plays a potent role in academic success. one goal orientation theory (dweck, 1999; dweck & leggett, 1988) suggests that the effects of these goals can be altered or amplified during stress or challenge. we can identify an obvious and frequent source of stress in hoyert, m.s. and o’dell, c.d. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 3 our students, failure on an examination. to see how relevant achievement goals are in this situation, we tracked students to see what happened on the subsequent examination after an objective failure. students who endorsed mastery orientation enjoyed a 15-point increase on the next examination. students who pursued performance goals suffered a 10-point decrease. not only did different patterns of goal orientation lead to an effect on academic performance, but it was very robust and consistent. a 25-point difference in examination scores can lead to a difference of two or three letter grades. further, ninety-five percent of the mastery-oriented students attained a higher examination grade on the subsequent test. about half of the performance-oriented students had a decrease on their test grade on the subsequent examination (hoyert & o’dell, 1999; 2000; 2001a; 2001b; 2004a, 2004b; o’dell & hoyert, 2000, 2002). as a result, we developed an intervention to aid struggling introductory psychology students. we measured goal orientation at the onset of the semester and invited students who primarily pursued performance goals to attend brief tutorial sessions after they had failed an examination. student peers led the sessions. the goal of the intervention was to alter goal orientation in order to prevent the decrement sometimes seen in highly performance-oriented students. the tutors’ coached students on a variety of techniques including orientation modeling from several different perspectives, discussion of multiple study techniques, goal setting, and value referencing. the intervention provided students with knowledge about and practice expressing mastery goals. it must be pointed out that the tutorials only address motivational issues. we did not cover classroom material. the intervention produced clear benefits: students in the intervention began to endorse mastery goals to a greater extent and earned higher examination and course grades. the average student achieved a 15-point increase on their next examination. further, the effects persisted over the duration of the semester even though the intervention ended. the d/f failure rate was nearly cut in half (to 47%). these improvements are even more impressive when their performance is compared to that of the control participants. one could view the control group as a predictor of what could have happened to the intervention group participants without the intervention. these two groups of participants had the same goal orientation profile and the same grades on the first examination. both sets of students endorsed performance goals more than mastery goals. after the first examination, the control students= grades declined precipitously. the failure rate (d, f) was 78%. the successes of this intervention are heartening. however, the procedure reached a relatively small number of students and was labor intensive. over the course of three semesters we were able to involve a little over 60 students. we believe that there are many more students who could benefit from the intervention. in any particular class, we estimate that about half of the students are predominantly performance-oriented and that about half of the students who are predominately performance-oriented will fail at least one examination. this amounts to several hundred students each year at our school. in this research, we explored two techniques for extending the intervention to reach all struggling performance-oriented students enrolled in sections of introductory psychology in an effort to alter the success profile in the class as a whole. ii. method a. instruments goal orientation was measured using a locally developed inventory modeled after roedel, schraw, and plake’s (1994) goals inventory. this instrument consists of 24 statements hoyert, m.s. and o’dell, c.d. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 4 rated on a 5-point likert-type scale for strength of agreement. students were asked to consider how much each of the statements applies to themselves within the introductory psychology class. the statements assess attitudes and behaviors towards learning and performance goals as described by dweck and leggett (1988). b. procedure intervention 1: guest lecture. during the first day of the semester and after the final examination, the goal inventory was administered to all students. one hundred eighteen students were enrolled in the class. many of these students experience an academic challenge over the course of the semester. eleven students did not take the first examination. twenty-eight students received a failing grade on the first examination. sixty students failed at least one examination over the course of the semester. eighty-three students received a grade that was less than a c on at least one examination over the course of the semester. the intervention occurred during the class immediately after the first examination and took the form of a lecture and discussion. in the intervention, we provided information about goals and the meaning of failure, and practiced setting mastery goals. all of the exercises were aimed at increasing the adoption of mastery orientation. the techniques included orientation modeling from several different perspectives, discussion of multiple study techniques, goal setting, and value referencing. following the class, students were instructed to write a paper defining the two goals, describing individuals who exemplify the traits of each goal, and considering their own experience with goal orientation. seventy-two students attended the class and 19 wrote and submitted the paper. at the end of the academic term, the students’ introductory psychology examination and course grades were obtained from the instructor. intervention 2: interactive computerized tutorial. this intervention occurred with a different class during a different semester. following the first examination all students received a cd rom containing the goals inventory and the intervention. the intervention provided students with information about goals and the meaning of failure and practiced setting mastery goals. it took the students 35 to 55 minutes to complete the tutorial. two hundred-twelve students were enrolled in the class. many of these students experienced an academic challenge during the semester. eighty-two failed the first examination. one hundred fifty-six failed at least one examination over the course of the semester. one hundred eighty-three received a grade that was less than a c on at least one examination over the course of the semester. one hundred thirty-seven (65% of the class) students completed the intervention; 75 did not. there were 147 students who were performance-oriented and failed at least one examination. eighty-two out of the 147 (56% of the group) target students completed the intervention. iii. results a. intervention 1 nineteen students completed the motivational intervention. nine of these were not struggling or were not high-performance oriented students. ten were students that we predict could struggle in the class. in this study, we are primarily interested in struggling students. we predict that a student will be at risk for academic failure when they pursue performance goals more than learning goals and when they fail an examination or are otherwise challenged. we will refer to these students as the “target” students. because the 10 struggling students in the hoyert, m.s. and o’dell, c.d. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 5 intervention group and the 9 other students responded to the intervention approximately the same we will display the group data because it is based on more observations. ninety-nine students did not participate in the intervention. of these 72 were at risk for failure because of grades and goal orientation. thus 12% of the target students completed the intervention. the students who completed the intervention did not differ from the students who did not complete the intervention in goal orientation or grades on examination 1. the students who wrote the paper had a mean mastery score of 3.69 (sd=0.79) (both goal orientation scores can range from 1 to 5). the students who did not write the paper had a mean mastery goals score of 3.86 (sd=0.88). these did not differ (t(94)=.612, p=.542). the same pattern holds for performance goals (intervention: m=3.59, sd=0.61; comparison: m=3.75, sd=1.06; t(94)=.425, p=.672). the scores on the first examination also did not differ (intervention group: m=74.63, sd=14.00; comparison: m=71.18, sd=14.19, t(105)=0.963, p=.338). because the two groups were similar, the students who did not complete the intervention can be treated as a convenience control group or a comparison group. the intervention produced an increase in the pursuit of mastery goals. before the intervention the mean was 3.69 (sd=0.79) and after it had increased to 3.78 (sd=0.54) however, the increase was not statistically significant (t(18)=0.527, p=.604). the intervention also produced a decrease in performance goals (first day: m=3.86, sd=0.89; last day m=3.77, sd=0.92; t(18)=.749, p=.463). in contrast, students who had not completed the intervention displayed dramatic decreases in both mastery (first day: m=3.58, sd=0.55; last day m=3.18, sd=1.00; t(37)=2.40, p=.021) and performance (first day: m=3.76, sd=1.07; last day m=3.37, sd=0.70; t(37)=2.31, p=.026) goals over the course of the semester. students who completed the intervention displayed steadily but slightly increasing examination grades (exam 1 m=74.63, sd=14.00, exam 2 m=75.89, d=14.60, exam 3 m= 76.44, sd=12.53, exam 4 m=80.94, sd=12.79, exam 5 m=78.09, sd=10.11). the gradual increase is not statistically significant (f(4,64)=1.11, p=.358). the scores on examination 1 for students who did not complete the intervention were very similar to the students who completed the intervention. however, after the first examination, the grades for the students who did not complete the intervention declined precipitously (exam 1 m=71.18, sd=14.25, exam 2 m=68.33, sd=15.98, exam 3 m= 65.42, sd=13.77, exam 4 m=64.75, sd=19.96, exam 5 m=57.96, sd=13.57). this decline is significant (f(4,144)=6.01, p<.001). the overall pattern of increasing grades for the students who completed the intervention is different from the pattern of decreasing grades for the comparison students (f(1,204)=3.42, p=.010). figure 1 presents the difference in mean examination grades between students who completed the intervention and those who did not. on examination 1 the difference was less than 3 points. with each subsequent exam, the disparity widened. by examination 5, there was a 20-point difference between the test scores of the two groups. the intervention also led to improvements in their final course grades. table 1 displays the percentage of grades in the two groups. we compared the semester average for the two groups. students who completed the intervention earned higher average grades than the other students (intervention: m=78.9, sd=19.47, comparison: m=46.89, sd=27.99; t(116)=4.75, p<.001). because we are primarily interested in the effects of the intervention on the target students who are highly performance-oriented and who fail an examination, we report their scores separately. the pattern of results was similar for these target students as for the overall sample. the intervention led to slight increases in mastery goals (first class: m=3.62, sd=0.83; hoyert, m.s. and o’dell, c.d. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 6 figure 1: difference in examination scores between intervention and comparison groups 0 5 10 15 20 25 1 2 3 4 5 examination m ea n di ff er en ce be tw ee n sc or es table 1: grade distributions for students completing intervention 1 and for the comparison group grade a b c d f/w completed intervention 26.3 31.6 31.6 5.3 5.5 comparison group 5.2 15.6 17.7 15.6 45.8 last class: m=3.74, sd=0.53; t(17)=0.594, p=0.560) and slight decreases in performance goals (first class: m=3.89, sd=0.90; last class: m=3.78, sd=0.84; t(17)=0.868, p=0.398). target students who failed an examination and did not complete the intervention demonstrated considerable decreases in mastery goals (first class: m=3.58, sd=0.55; last class: m=3.18, sd=1.00; t(17)=2.40, p=0.021), and performance goals (first class: m=3.76, sd=1.07; last class: m=3.37, sd=0.70; t(17)=2.31, p=0.026). as can be seen in table 2, the target students who completed the intervention also maintained higher grades. table 2: grade distributions for performance-oriented students completing intervention 1 and for the comparison group grade a b c d f/w completed intervention 0.0% 30.0% 60.0% 0.0% 10.0% comparison group 0.0% 6.8% 23.3% 19.2% 50.7% hoyert, m.s. and o’dell, c.d. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 7 b. intervention 2 one hundred thirty-seven students completed the motivational intervention. fifty-five of these were not struggling or were not high-performance oriented students. we predict that 82 of these students would struggle in the class because of goal orientation. seventy-five students did not participate in the intervention. of these, 66 were “at risk” because of grades and goal orientation. thus about 55% of the target students completed the intervention. the scores of students who completed the intervention did not differ from the students who did not complete the intervention on examination 1 (intervention, m=54.5, sd=14.12; comparison, m=55.0, sd=14.56; t(197)=0.23, p=0.749). because the two groups were similar, the students who did not complete the intervention can be treated as a comparison group. the intervention produced an increase in the pursuit of mastery goals in both the target students as well as the general body of students who completed the intervention. since our primary interest is in the target students we will present these data first. before the intervention the mean was 4.10 (sd=0.09) and after it had increased to 4.29 (sd=0.10) (t(80)=3.34, p=0.002). the intervention also produced a decrease in performance goals. before the intervention, the mean was 4.07 (sd=0.42) and after it had decreased to 3.06 (sd=0.91) (t(80)=8.69, p<0.001). target students who completed the intervention displayed steadily increasing examination grades (exam 1 m=54.4, exam 2 m=55.9, exam 3, m= 56.0, exam 4 m=57.9, exam 5 m=55.6). in contrast, the students who did not complete the intervention earned increasingly lower grades (exam 1 m=55.0, exam 2 m=47.3, exam 3, m= 43.8, exam 4 m=37.7, exam 5 m=31.0). the grade on examination 1 for the two groups is not statistically significant. however, the overall pattern of increasing grades for the students who completed the intervention and decreasing grades for the comparison group is significant (f(1,109)=43.06, p<0.001). figure 2 presents the difference in mean examination grades between students who completed the intervention and those who did not. on examination 1, the students who did not complete the intervention maintained a higher mean exam grade than the students who would later complete the intervention. after completing the intervention, the intervention students outscored the comparison group by nine points. with each subsequent exam, the disparity widened. by examination 5, there was a 25-point difference between the two groups. figure 2: difference in examination scores between intervention and comparison groups -5 0 5 10 15 20 25 1 2 3 4 5 examination m ea n di ff er en ce be tw ee n sc or es hoyert, m.s. and o’dell, c.d. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 8 the intervention also led to improvements in their final course grades. the dfw rate in introductory psychology is typically similar to the percentage of students who received an f on any one test. in this class the percentage of students who received an f was 74%. thus, we could expect a very high dfw rate. the comparison group resembled this historical pattern. table 3 reports that 76% of these students failed the class. in contrast, in the intervention group, only 37% of the students received an f for the course. the pattern of grades in the intervention group is statistically different from the comparison group (x2(4)=12.38, p<0.001). table 3: grade distributions for students completing intervention 2 and for the comparison group grade a b c d f/w completed intervention 1.2% 9.9% 21.0% 30.9% 37.0% comparison group 0.0% 1.5% 9.1% 13.6% 75.8% we are also interested in the overall effect on grades in the class. table 4 displays the effects on course grades. given the historical pattern and the pattern of goals and examination scores we would expect that 156 students would have earned less than a c. after the intervention, we found that 123 students received a d, an f, or withdrew. table 4: overall effect on the grade distribution students completing intervention 2 and the predicted grade distribution a, b, or c d, f, or w intervention 42.0% 58.0% predicted 26.4% 73.6% iv. discussion teachers are frequently puzzled by the disparate reactions of students to challenge. after failing an examination some students will react with despair and will give up. other students will react to the same challenge by buckling-down, increasing their effort, and developing better study habits. the study of goal orientation seems to provide some insight into patterns of reactions such as these (dweck, 1999; dweck & leggett, 1988; hoyert & o’dell, 1999; 2000; 2001a; 2001b; 2004a, 2004b; o’dell & hoyert, 2000, 2002). the interventions assessed here attempt to encourage students who may have been prone to despair to adopt a more positive approach to challenge. we have now attempted three variants of the intervention. the first involved peer-tutors. it was not successful in reaching large numbers of students. however, when students participated, they enjoyed changed goals and increased academic success (hoyert & o’dell, 2004b). to an extent, goal orientation theory suggests a reason for the lack of participation. highly performance-oriented students tend to avoid evidence of a lack of competence. to a hoyert, m.s. and o’dell, c.d. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 9 performance-oriented student, seeking help from a tutor is an admission that they are not capable of completing the task without assistance, something that a truly competent student could have done. as a result, we developed two variants of the intervention that can be embedded in the course structure so that participation will not carry any perceived stigma. providing the intervention through a guest lecture was similar in effect to the peer-tutors. not very many students were willing to participate. however, when they completed the intervention, they enjoyed a changed pattern of goals, greater persistence, and higher grades. one unique finding occurred during this particular class. mastery and performance goals tend to remain rather stable over time. typically, when we measure these goals at the beginning and the end of a class without an intervention, they are approximately the same for each student and for the class as a whole. in this class, goals changed dramatically. both mastery and performance goals declined significantly over the semester. we have not observed this before and suspect that it may have been related to messages provided by the instructor or to the pattern of academic success experienced by the students (hoyert & o’dell, 1999; 2000; 2001a; 2001b; 2004a, 2004b; o’dell & hoyert, 2000, 2002). given the backdrop of declining goals, the increase in mastery goals observed in the students who completed the intervention provides strong evidence for the efficacy of the intervention. the intervention occurred once, early in the semester, and lasted about 45 minutes. however, the effects of the intervention persisted for the duration of the semester. it is noteworthy that the average examination grades of these students increased over the course of the semester in much the same way as the mastery goals. the students who completed the intervention passed the class and earned high grades at far higher rates than the students who did not. the major drawback of this intervention is that it did not attract many students. only 12% of the target students completed the intervention. one of our goals in this research is to determine whether we can alter the pattern of academic success in a whole class. because we did not attract many students to complete the intervention, we were not able to decrease the proportion of students who did not pass the class (dfw rate). the computerized tutorial has been the best procedure for involving students in the intervention to date. over half of the students who we predicted could be at risk for academic failure participated in the intervention. after completing the brief intervention, they pursued mastery goals to a greater extent and were less interested in pursuing performance goals. the change in goals led to changes in studying and increased academic success over the duration of the semester. one of our interests is in trying to decrease the rate of failure in individual students. the students who completed the intervention earned failing course grades at less than half the rate of the historical pattern and at less than half the rate of the comparison group. another interest is in altering the rate of failing grades in an entire class. it appears that the intervention was successful in decreasing the rate of the f grade by about 25%. the present interventions produce positive results. however, they still can be improved. both procedures could be more fully integrated into the class structure to encourage more students to participate. for instance, making the programs mandatory components of the classes could be explored. in addition, the intervention has only been attempted within introductory psychology classes. an interesting extension could be to try to introduce the intervention in other introductory level courses in other disciplines. the success profiles in many mathematics and science classes are lower than introductory psychology. this would suggest that they are prime candidates for this type of intervention. hoyert, m.s. and o’dell, c.d. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 10 finally, this study contained a troubling methodological flaw. both interventions may have a selection problem. the students determined who would participate in the intervention. we have identified goal orientation and challenge as important variables in course success. we have observed that these variables are similar in the two groups. thus, we believe that it is reasonable to use these students as a comparison group. to a certain extent, this was unexpected in the first intervention. however, we would predict that in the event of a low rate of participation, more learning-oriented students would participate in the intervention than performance-oriented students. regardless, because students could determine participation, it could be that the differences in grades, persistence, and changes in grades are related to the selfselection. another important future study could involve a procedure in which students are randomly assigned to the intervention or to one of several control conditions. we are optimistic that these changes will further strengthen our findings that the intervention positively alters goal orientation, examination scores and course grades in a college setting. references albaili, m. (1998). “goal orientations, cognitive strategies and academic achievement among united arab emirates college students.” educational psychology, 18, 195-203. ames, c. (1992). “classrooms: goals, structures, and student motivation.” journal of educational psychology, 84, 261-271. ames, c., & archer, j. (1988). “achievement goals in the classroom: students’ learning strategies and motivation processes.” journal of educational psychology, 80, 260-267. aronson, j., & fried, c. (1998). “reducing stereotype threat and boosting academic achievement of african americans: the role of conceptions of intelligence.” unpublished manuscript. beck, h. p., rorer-woody, s., & pierce, l. g. (1991). “the relations of learning and grade orientations to academic performance.” teaching of psychology, 15, 35-37. blumenfeld, p.c. (1992). “classroom learning and motivation: clarifying and expanding goal theory,” journal of educational psychology, 84, 272-281. bouffard, t., boisvert, j., vezeau, c., & larouche, c. (1995). “the impact of goal orientation on self-regulation and performance among college students.” british journal of educational psychology, 65, 317-329. dweck, c. s. (1999). self-theories: their role in motivation, personality, and development. philadelphia: psychology press. dweck, c. s. & leggett, e. l. (1988). “a social-cognitive approach to motivation and personality.” psychological review, 95, 256-273. elliot, a. j., & church, m. a. (1997). “a hierarchical model of approach and avoidance achievement motivation.” journal of personality and social psychology, 72, 218-232. hoyert, m.s. and o’dell, c.d. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 11 eppler, m. a. & harju, b. l. (1997). “achievement motivation goals in relation to academic performance in traditional and non-traditional college students.” research in higher education, 38, 557-573. graham, s., & golan, s. (1991). “motivational influences on cognition: task involvement, ego involvement, and depth of information processing.” journal of educational psychology, 83, 187-194. greene, b. a., & miller, r. b. (1996) “influence on achievement: goals, perceived ability, and cognitive engagement.” contemporary educational psychology, 21, 181-192. harackiewicz, j. m., barron, k. e., carter, s. m., lehto, a. t., & elliot, a. j. (1997). “predictors and consequences of achievement goals in the college classroom: maintaining interest and making the grade.” journal of personality and social psychology, 73, 1284-1295. harackiewicz, j. m., barron, k. e., & elliot, a. j. (1998). “rethinking achievement goals: when are they adaptive for college students and why?” educational psychologist, 33, 1-21. harackiewicz, j.m., barron, k.e., tauer, j. b., carter, r. a., & elliot, a. j. (2000). “short-term and long-term consequences of achievement goals: predicting interest and performance over time.” journal of educational psychology, 75, 183-205. harackiewicz, j.m., & elliot, a.j. (1993). “achievement goals and intrinsic motivation.” journal of personality and social psychology, 65, 904-915. harackiewicz, j.m., & elliot, a.j. (1998). “the joint effects of target and purpose goals on intrinsic motivation: a mediational analysis.” personality and social psychology bulletin, 24, 675-689. harju, b. l. & eppler, m. a. (1997). “achievement motivation, flow and irrational beliefs in traditional and nontraditional college students.” journal of instructional psychology, 24, 147157. hidi, s. & harachiewicz, j.m. (2000). “motivating the academically unmotivated: a critical issue for the 21st century.” review of educational research, 70, 151-179. hoyert, m.s. & o’dell, c.d. (1999). “achievement motivation in traditional and non-traditional students.” poster presented at the annual meeting of the midwestern psychological association, april 29 may 1, 1999, chicago, il. hoyert, m.s. & o’dell, c.d. (2000). “achievement motivation and the impostor phenomenon.” poster presented at the annual meeting of the midwestern psychological association, may 5, 2000, chicago, il. hoyert, m.s. & o’dell, c.d. (2001a). “goal orientation and challenge in college students.” paper presented at a symposium entitled “toward a scholarship of teaching and learning in developmental psychology” at the biennial meeting of the society for research in child development, april 10-22, 2001, minneapolis, mn. hoyert, m.s. and o’dell, c.d. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 12 hoyert, m.s. & o’dell, c.d. (2001b). “achievement motivation and excuse making.” poster presented at the annual meeting of the midwestern psychological association, may 3-5, 2001, chicago, il. hoyert, m. s., & o’dell, c.d. (2004). “the impostor phenomenon and goal orientation.” journal of the indiana academy of social sciences, 7, 1-10. hoyert, m.s., & o’dell, c.d. (2004). “altering achievement motivation to enhance classroom learning.” selected papers from the 15th international conference on college teaching and learning, 15, 119-135. kong, c., & hau, k. (1996). “student achievement goals and approaches to learning: the relationship between emphasis on self-improvement and thorough understanding.” research in education, 55, 74-85. midgley, c. (1993). “motivation and middle level schools”, in advances in motivation and achievement (vol. 8, pp. 217-274). greenwich, ct: jai press. o’dell, c.d., & hoyert, m.s. (2000). “achievement motivation and academic success in traditional and non-traditional aged college students.” poster presented at the 16th biennial meeting of the conference on human development, april 14-16, 2000, memphis, tn. o’dell, c. d., & hoyert, m.s.(2002). “altering achievement motivation and the response to failure.” poster presented at the ctup creative classroom poster session held during the 73rd annual meeting of the midwestern psychological association. pintrich, p. r., & garcia, m. (1991). “student goal orientation and self-regulation in the college classroom.” in advances in motivation and achievement (vol. 11, pp. 123-158). greenwich, ct: jai press. pintrich, p. r., & garcia, m. (1993). “intraindividual differences in students= motivation and self-regulated learning.” german journal of educational psychology, 7, 99-107. pintrich, p. r., & schunk, d. h. (1996). motivation in education: theory, research, and applications. englewood cliffs, nj: prentice hall. pintrich, p. r., zusho, a., schiefele, u. & pekrun, r. (2001). “goal orientation and selfregulated learning in the college classroom: a cross-cultural comparison.” in f. salili, c. y. chiu & y. y. hong (eds.), student motivation: the culture and context of learning. new york: plenum. roedel, t. d. & schraw, g. (1995). “beliefs about intelligence and academic goals.” contemporary educational psychology, 20, 464-468. roedel, t. d., schraw, g., & plake, b. s. (1994). “validation of a measure of learning and performance goal orientations.” educational and psychological measurement, 54, 1013-1021. hoyert, m.s. and o’dell, c.d. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 13 ryan, a.m., gheen, h. h., & midgley, c. (2000). “why do some students avoid asking for help? an examination of the interplay among students’ academic efficacy, teachers= socialemotional role, and the classroom goal structure.” journal of educational psychology, 90, 528535. schraw, g., horn, c., thorndike-christ, t., & bruning, r. (1995). “academic goal orientations and student classroom achievement.” contemporary educational psychology, 20, 359-368. somuncuoglu, y. & yildirim, a. (1999). “relationship between achievement goal orientations and use of learning strategies.” journal of educational research, 92, 267-277. wolters, c. a., yu, s. l., & pintrich, p. r. (1996). “the relation between goal orientation and students= motivational beliefs and self-regulated learning.” learning and individual differences, 8, 211-238. urdan, t. c. (1997). “examining the relations among early adolescent students’ goals and friends’ orientation toward effort and achievement in school.” contemporary educational psychology, 22, 165-191. urdan, t. c., kneisel, l., & mason, v. (1999). “interpreting messages about motiation in the classroom: examining the effects of achievement goal structures.” in advances in motivation and achievement (vol. 11, pp. 123-158). greenwich, ct: jai press. urdan, t.c., & maehr, m. l. (1995). “beyond a two goal theory of motivation: a case for social goals.” review of educational research, 65, 213-243. urdan, t., midgley, c., & anderman, e. (1998). “the role of classroom goal structure in students= use of self-handicapping strategies.” american educational research journal, 35, 101-122. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006, pp. 14 – 38. students’ race and participation in sociology classroom discussion: a preliminary investigation jay howard1, aimee zoeller2 , and yale pratt3 abstract: this study utilizes observation, survey and interview methodologies to investigate the impact of student race on participation in discussion in introductory sociology courses at a large midwestern us university with a minority enrollment of approximately 15 percent. while results are mixed there is some evidence that white students participated at a higher rate than minority students. however, in certain circumstances (e.g., discussion of racism), minority students became the “experts” during particular class sessions and participated at a greater rate than did white students. key words: discussion, college students, race, learning, interaction i. introduction and literature review the 2003 supreme court decision on race sensitive college admission policies at the university of michigan once again focused attention on minority students in higher education. higher education researchers have long been interested in the impact of race on end-of-first-year degree plans (pascarella, wolniak, and pierson, 2003), development of problem-solving and group skills (terenzini, cabrera, colbeck, bjorklund, and parente, 2001), preferences towards collaborative learning (cabrera, crissman, bernal, nora, terenzini, and pascarella, 2002), and adjustment to college (cabrera, nora, terenzini, pascarella, and hagedorn, 1999; schwitzer, ancis, and griffin, 1999). researchers have also compared the experiences and perceptions of african-american students who attend historically black colleges and universities with africanamerican students attending predominately white institutions (terenzini, yaeger, bohr, pascarella, and amaury, 1997). feagin, cera and imani (1996) concluded that african-american college students continue to face many obstacles in higher education. feagin (2003) found that black college students face a continuum of discriminatory practices that included aggression, exclusion, dismissal of subculture and typecasting which may be responsible, in part, for declining college enrollment and graduation for black americans. one largely unaddressed issue is whether these obstacles and discriminatory practices impact minority students’ participation in college classroom discussions. antonio, et al (2004) demonstrated that white college students display higher levels of complex thought when they are placed in discussion groups with a black student. given that participation in classroom discussion has also been associated with learning (astin, 1985; johnson and johnson, 1991; kember and gow, 1994; mckeachie, 1990), critical thinking 1 professor of sociology and head of division of liberal arts, indiana university purdue university columbus, 4601 central ave., columbus, in 47203, jhoward@iupuc.edu. this research was supported by grants from the indiana university purdue university indianapolis office for professional development and the mack center at indiana university for inquiry on teaching and learning. 2 adjunct lecturer in sociology, indiana university purdue university columbus, 4601 central ave., columbus, in 47203, anzoelle@iupui.edu 3 yale pratt was graduate research assistant at indiana university purdue university indianapolis during the data gather portion of this project. he was tragically killed in an automobile accident before the project was completed. howard and zoeller wish to acknowledge his contributions to this study by including him as a co-author. howard, j., zoeller, a., and pratt, y. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 15 (garside, 1996; smith, 1977; weast, 1996), and degree completion (tinto, 1975; tinto, 1997), it makes sense to ask, does race matter in participation in classroom discussion? this study seeks to address this void in the literature on student participation by addressing the role of student race in classroom discussion in introductory sociology courses at a large urban university with a racially mixed student population. studies of student participation in discussion in the college classroom have addressed a number of variables thought to have significant impact including student gender, student age, instructor gender, class size, instructor traits, student traits, and classroom environment. the variable most often examined is student gender. this line of research springs from hall and sandlers’ (1982) “chilly climate” thesis which postulated that patterns of interaction and behavior in the college classroom create a climate that is less hospitable to female students than to male students. despite the ongoing concern with student gender in classroom participation, the research support has been mixed. a number of studies have found that males participate more frequently than females (auster and macrone, 1994; brooks, 1982; crawford and macleod, 1990 [in their small college sample, but not in their university sample]; crombie, pyke, silverthorn, jones, and piccinin, 2003; fassinger, 1995; karp and yoels, 1976; o’keefe and faupel, 1987; and statham, richardson, and cook, 1991). brooks (1982) concluded that males participate more only in courses taught by female instructors. other studies have suggested the opposite – males participate more frequently in male taught courses (pearson and west, 1991; sternglanz and lyberger-ficek, 1977) but not in courses taught by female instructors. fassinger (1995) and karp and yoels (1976) found that females participate more in courses taught by female instructors than in courses taught by male instructors. a large number of studies found no significant difference in participation based on student gender (boersma, 1981; constantinople, cornelius, and gray, 1988; corneilius, gray, and constantinople, 1990; crawford and macleod, 1990 [in their university sample]; heller, puff, and mills, 1985; howard, james, and taylor, 2002; jung, moore, and parker, 1999). one possible explanation for the lack of significance of student gender is the presence a high percentage of females in the classroom. however, in their study which utilized a survey methodology, crombie, pyke, silverthorn, jones, and piccinin (2003) failed to find a significant relationship between percentage of female students in a class and students’ participation. several studies have demonstrated that student age has a stronger impact than student gender on participation in classroom discussion. nontraditional students (25 years of age or older) have been consistently shown to participate more frequently than traditional students (less than 25 years of age) (howard, short, and clark, 1996; howard and henney, 1998; howard and baird, 2000; howard, james, and taylor 2002; jung, moore, and parker, 1999). however, crombie, pyke, silverthorn, jones and piccinin (2003) found no differences in participation by student age in a study that utilized self reports rather than observation. one study by faust and courtenay (2002), who observed only a single section of a single course, found the opposite – that traditional students contributed to class discussion more frequently than did nontraditional students. instructor gender is another variable that has been examined in relation to students’ participation in classroom discussion. the results have again been mixed. some studies have found that there is more discussion in courses with female instructors (canada and pringle 1995; constantinople, cornelius, and gray, 1988; crawford and macleod, 1990; fassinger, 1995; howard and baird, 2000; howard, james, and taylor, 2002; karp and yoels, 1976; pearson and howard, j., zoeller, a., and pratt, y. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 16 west, 1991; and statham, richardson, and cook, 1991). while none of the studies has suggested that students participate more frequently in courses with male instructors, numerous studies have failed to find a difference based on instructor gender (auster and macrone, 1994; cornelius, gray, and constantinople, 1990; crombie, pyke, silverthorn, jones, and piccinin, 2003; heller, puff, and mills, 1985). of these studies that failed to find an effect of instructor gender all but one (cornelius, gray, and constantinople 1990) relied on student self reports via survey rather than observation. this may account for the lack of significant findings. karp and yoels (1976) reported that while students reported no effect of instructor gender in their survey responses, based on observations of actual classroom behaviors female students participated significantly more in female taught courses than in male taught courses. howard and baird (2000) and howard, james and taylor (2002) had the same result with survey responses failing to find a relationship between instructor gender and participation, but observations of classroom behavior revealing that students participate more frequently in courses with female instructors. class size is another variable frequently found to have a significant impact on student participation in discussion. most studies have found that more interaction occurs in smaller classes (auster and macrone, 1994; constantinople, cornelius, and gray, 1988; cornelius, gray, and constantinople, 1990; crawford and macleod, 1990; fassinger, 1995; howard, short, and clark, 1996; howard and henney, 1998; neer and kircher, 1989). however, crombie, pyke, silverthorn, jones, and piccinin (2003) and karp and yoels (1976) failed to find a significant impact of class size. fassinger (1995) argued that instructor traits (e.g., gender) have little impact on student participation. instead, student traits (confidence, comprehension, interest, preparation) and class traits (size, emotional climate, interaction norms, frequent large group discussions) were more important influences on participation. likewise, aitken and neer (1993) concluded that it is a student trait (motivation or the lack thereof) that best explains students’ lack of participation. however, it is clear that instructor behaviors can influence student traits like comprehension and interest and can influence class traits such as emotional climate and interaction norms. nunn (1996) argued that it is instructor teaching techniques (such as praise, posing questions, asking for elaboration, and using students’ names) that significantly improve levels of discussion. thus nunn concludes that instructors do play an important role in student participation. fritschner (2000) found that students were more likely to participate in 300-400 level courses than in 100200 level courses. despite this extensive research examining numerous variables, to date research focusing on classroom discussion has not directly addressed the impact of race on college students’ participation. however, some have argued, based on personal experience rather than systematic research, that minority students at predominately white institutions manifest a fear of failure that may cause them to participate in class discussions less frequently than white students (see for example, saufley, cowan, and blake, 1983). likewise, asian students enrolled in australian universities have been presumed to bring learning experiences that favor passive rote reproduction and teacher centered learning in contrast to the active learning and critical thinking required in class discussions which are more typical in australia and the west (see for example, ballard and clanchy, 1991). adams (1992) argued that men and women of culture have alternative cultures which imply a need for more collaborative and less competitive instructional design. one possible interpretation of adams’ (1992) argument is that students of color may be better prepared for participation at least in collaborative classroom discussions. hardiman and jackson (1992) argue that instructors’ failure to understand and respect the racial identity of howard, j., zoeller, a., and pratt, y. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 17 students can lead to volatile situations in the classroom and on campus. weinstein and obear (1992) suggest that majority group students can “trigger” (p. 44) defensive and intense emotional reactions from minority group members that can easily silence classroom discussion altogether. each of these studies of the minority students rely primarily on personal experience and reflection rather than systematic research to determine whether minority students participate in classroom discussion at a different rate than majority students. this investigation seeks to fill this void. ii. methodology this study was conducted at both [identifying information removed] and [identifying information removed]’s satellite campus in [identifying information removed]. a triangulation of research methods was utilized to examine students’ participation in classroom discussions in introductory sociology courses. as noted above, multiple methods are important when examining student participation in the college classroom. while surveys allow researchers to access students’ attitudes and beliefs about their own participation, often students’ self reports are not supported by observations. thus this study utilizes observation, survey and interview methodologies to provide the fullest possible picture of student participation in the college classroom. both for convenience and in order to limit the effect of variation in the curriculum on participation in discussion we chose to limit our investigation to introductory sociology courses. all eleven instructors teaching in the fall 2003 semester at [identifying information removed] were invited to participate in the study. nine instructors (six males, three females) agreed to participate. two instructors (one male, one female) chose not to participate. one section of each of the nine instructors’ introductory sociology courses were observed for four class meeting sessions. the observations were spread over the course of the semester with one observation occurring approximately every four weeks. during our observations, we kept track of student participation in discussion by using a seating chart to note students’ gender, approximate age (traditional or nontraditional), and race (african-american, asian, hispanic, mixed, or white) as it appeared to the observer. any verbal response by students regardless of length or content was counted an instance of participation. thus a brief response (e.g., “can you repeat the definition?”) counted equally with a longer comment or question that demonstrated critical thinking. we also kept more general field notes regarding activity in the classroom. eight of the nine courses were “regular sections” with a maximum enrollment of 45 students. one of the observed courses was a “mass lecture” section with an enrollment of 182 students at the start of the semester. a total of 36 class meetings were observed with 1402 students in attendance (15.5 percent non-white). during the last three weeks of the semester all students in attendance in 15 sections of introduction to sociology (three instructors taught more than one section) were given a survey to assess students’ perceptions of their participation in classroom discussion and their reasons for participation and for non-participation. a total of 441 students completed the survey (13.2 percent non-white). finally, the researchers interviewed the nine instructors and ten students from the courses observed. we sought to interview an equal number of white and non-white students. this proved to be a since there were relatively few non-white students. also because [identifying information removed] is a commuter campus, it was difficult to schedule interviews with students who frequently left campus whenever they were not in class. numerous students agreed howard, j., zoeller, a., and pratt, y. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 18 to be interviewed, but then failed to appear at the agreed upon time. eventually, were are able to interview five white and five non-white students, a significantly fewer than the 20 students we had hoped to interview. in the interviews, students were asked about the effectiveness of discussions for facilitating learning and their perceptions of the students who participate most frequently in class discussion. iii. results and discussion table 1 presents the mean number of interactions per student per class session by student race, student gender, and student age. the typical 75 minute class session averaged almost 50 interactions from the 39 students in attendance. this resulted in a mean of 1.27 interactions per student. however, a caution is necessary. computing mean interactions can be somewhat misleading because as table 1 indicates over two thirds of all students present fail to contribute to discussion in a typical class session. on average, around 12 students, or 30 percent of those present, participate in discussion for a mean of 4.3 interactions per student participant. table 1 also presents results by student race. an anova comparison of means is used to test for significant differences in mean interactions per student and kendall’s tau is used to test for significant differences in the percentage of students participating. the vast majority of nonwhite students were african american (72.8%). because there were so few asians (15.6%), hispanics (6.0%), and mixed-race (4.6%) students in the sample, meaningful comparison of minority racial groupings were not possible. therefore, students were grouped by whites (84.5 percent of those present) and non-whites (15.5 percent of those present). the results reveal no significant differences between whites and non-whites. the percentage of whites and non-whites participating in classroom discussion is nearly identical (29.6 percent of whites compared to 29.5 percent of non-whites). white students had a higher mean interaction per student (1.31 to 1.05); however this difference was not statistically significant. the lack of significant findings may be due to the nature of the sample. if it were possible to separate the various minority groups, the results may have varied. for example, including asians in the same category with african americans and hispanics may be masking differences between racial groupings. interview evidence indicated that students themselves were uncertain whether minority students participated at the same rate as did white students. in their comments students would quickly note that because there were so many more white students in class, most contributions to class discussion came from white students. but they were unsure whether minority students participated at a rate that was proportional to their number. for example, a nontraditional black female student noted, “there were not too many minorities in the class. there were more whites to speak out. the minorities speak out just as much. but it doesn’t look like it because…there were not many minorities in the class.” instructors, on the other hand, tended to perceive minority students as less frequent contributors to discussion. one female instructor remarked, “i think it is always hard to have minorities be comfortable enough to speak up.” another female instructor stated, “i feel that there is less verbal discussion among minorities. i don’t know why. african american students speak and respond less.” in our observations, the participation of white and non-white students most often was very similar. however, there were occasions when non-white students became the class “experts” on a given topic, such as police profiling. on these occasions, the non-white students became the dominant talkers for the class session. one nontraditional white male student commented, “when we were discussing race, i noticed that a lot of people of color really seemed howard, j., zoeller, a., and pratt, y. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 19 table 1: interactions per class session (anova comparison of means) and percent of students participating (kendall’s tau) by student race, student gender, and student age. mean interaction per session mean attend mean no. students participate percent students participate mean interaction per student n all 49.4 38.9 11.5 29.6 1.27 1402 whites (84.5%) 43.1 32.9 9.8 29.6 1.31 1185 non-whites (15.5%) 6.3 6.0 1.8 29.5 1.05 217 males (28.9%) 12.3 11.3 2.9 25.4* 1.09 405 females (71.1%) 37.1 27.7 8.7 31.3* 1.34 996 traditional (89.7%) 33.2 34.9 9.1 26.1*** .95*** 1258 nontraditional (10.3%) 16.3 4.0 2.4 60.4*** 4.08*** 144 white males (24.3%) 11.3 9.5 2.5 26.7 1.19 341 non-white males (4.6%) 1.0 1.8 .3 18.7 .55 64 white females (60.2%) 31.8 23.4 7.2 30.8 1.36 843 non-white females (10.9%) 5.4 4.3 1.4 34.0 1.25 153 white traditional (77.3%) 30.1 30.1 7.9 26.1 1.00 1084 non-white traditional (12.4%) 3.1 4.8 1.3 25.9 .64 174 white nontraditional (7.2%) 13.1 2.8 1.9 67.3** 4.67*** 101 non-white nontraditional (3.1%) 3.2 1.2 .5 44.2** 2.70*** 43 n 1779 415 1779 1402 *** significant at the p < .001 level ** significant at the p < .01 level * significant at the p < at .05 level howard, j., zoeller, a., and pratt, y. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 20 to get involved on that topic. it seemed to hit more close to home.” instructors noted the same tendency, “when we were talking about racism and talked about hispanics, he [hispanic student] talked more than he usually does (female instructor).” table 1 also reveals that female students had a slightly higher, but not statistically significant, mean number of interactions per class session compared to 1.09 for male students (1.34 to 1.09). contrary to the chilly climate thesis, a significantly higher percentage of female students participated in class discussion than male students (31.3 to 25.4%). students differed significantly in their rates of participation by age (see table 1). nontraditional students, those the observers judged to be age 25 or older, had a mean number of interactions per class session which was over four times that of traditional students (4.08 to .95) and the percentage of nontraditional students who participated in discussion was more than double that of traditional students (60.4 to 26.1 percent). table 1 also presents a comparison of students by both race and gender. white male students had a higher mean number of interactions per class session (1.19 to .55) and a higher percentage of those present participating (26.7 to 18.7) than their non-white male counterparts. however, neither difference was statistically significant. again, the lack of significance may be due to sample size. only 64 non-white males (4.6% of all students) were in the sample, making demonstrations of statistical significance difficult. white and non-white female students were much closer in terms of mean interactions per class session (1.36 to 1.25). while the difference is small and is not statistically significant, it is interesting to note that the percentage of nonwhite females participating in discussion was slightly higher than that of white females (30.8 to 34.0). when we compared students by both race and age (see table 1), we again found white traditional students had a higher mean interaction per student (1.00 to .64) when compared with non-white traditional students. the percentage of students participating was nearly identical (26.1 to 25.9%). these differences were not statistically significant. for nontraditional students, however, whites had a significantly higher mean interaction per class sessions than did nonwhites (4.67 to 2.70). the nontraditional whites also had a significantly higher percentage of students participating compared to nontraditional non-whites (67.3 to 44.2%). table 2 presents a comparison of mean interactions per student per class session and the percentage of students participating in discussion by student race and instructor gender. nearly 80 percent of students observed, including those in the mass lecture class, were in courses taught by male instructors. a significantly higher percentage of students in female instructor courses participated in discussion as compared to students in male instructor courses (47.0 to 25.2%). students in female taught courses also had a significantly higher mean number of interactions per class sessions (2.69 to .91). these results support the previous findings that students participate more in female taught courses than in male taught courses. however, female instructors had the advantage of teaching smaller classes (23.8 to 39.9 students), a variable shown to significantly effect students’ interaction in previous studies (see, for example, auster and macrone, 1994). female instructors also had a higher percentage of nontraditional students in their courses (26.0 to 6.3%). as table 1 demonstrated, nontraditional students have a significantly higher mean number of interactions per class session. these differences in mean class size and percentage of nontraditional students may account for the differences in student participation in male and female instructors’ courses. howard, j., zoeller, a., and pratt, y. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 21 table 2: interactions per class session (anova comparison of means) and percent of students participating (kendall’s tau) by student race and instructor gender mean interaction per session mean attend mean no. students participat e percent students participate mean interaction per student n male instructor (79.7%) 36.3 39.9 10.0 25.2*** .91*** 1117 female instructor (20.3%) 64.0 23.8 11.2 47.0*** 2.69*** 285 white students male instructor (67.5%) 38.7 39.5 10.2 25.9 .98 947 non-white students male instructor (12.1%) 3.5 7.1 1.5 21.2 .49 170 white students female instructor (17.0%) 51.9 19.8 8.8 44.5 2.62 238 non-white students female instructor (3.4%) 11.9 3.9 2.3 59.64 3.04 47 n 1779 415 1779 1402 *** significant at the p < .001 level while white students in male instructors’ courses had a higher mean interaction per class session (.98 to .49), the difference was not statistically significant. the percentage of white and non-white students who participated in male instructors’ courses was very similar (25.9 to 21.2%). interestingly, non-white students in female instructors’ courses had both a higher mean interaction per student (3.04 to 2.62) and a higher percentage of non-white students in female instructors’ courses participated relative to white students (59.6 to 44.5%). however, neither difference was statistically significant. regardless of instructor gender, in our observations we could quickly identify which instructors’ classrooms included significant amounts of student discussion. when an instructor whose teaching style included significant use of discussion entered the classroom, informal interactions would begin immediately. sometimes it was the instructor greeting and engaging individual students, but most often it was students initiating interactions with the instructor. often the topic was related to course administration (e.g., “do you have our papers graded yet?”), but just as frequently the conversation was unrelated to the course. these instructors had clearly built relationships with their students so that the students felt very comfortable interacting with them. other instructors would enter the room and either be greeted with silence or would 4 significant at p. < 0.07 howard, j., zoeller, a., and pratt, y. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 22 face the challenge of gaining students’ attention when they were busy talking among themselves. this was particularly a problem in classes that consisted predominately of traditional college age students. for example, in a class that was 97 percent traditional students we observed the following. the instructor is having trouble getting students’ attention at start of class. he begins his lecture even though many students are still engaged in side conversations. the conversations at the rear of room continue very audibly even after class has clearly started. a student sitting in the front complains to the instructor that she cannot hear. the instructor chides those still chatting, “your talking is preventing classmates from being able to hear.” the talking softens but doesn’t completely stop. not only did the classes for these instructors not begin with productive student interaction, it was often 30 or more minutes into the session before the instructor first posed a question to the class. by this point, students appeared to have already gotten the, intended or unintended, message that their participation was neither needed nor desired. these late attempts to engage students in discussion were frequently met with silence from the students. instructors would pause awkwardly for a second or two, answer their own question, and return to their lecture. we observed both male and female instructors whose classes resembled the above, but most frequently larger (30 or more students) male taught courses with very high percentages of traditional students were the classes with the least interaction. table 3 presents a comparison of mean interactions per student and the percentage of students participating by student race and class type (regular session versus mass section). despite the fact that the mass class section had a very interactive instructor whose efforts resulted in more interactions per class meeting session and a higher mean number of students participating, the mean interaction per student was significantly lower than that of the regular sessions (1.83 to .40) as would be expected. likewise, the percentage of students participating in the regular sections was significantly higher than that of students in the mass section (37.3 to 17.8%). these results are simply a reality of mass sections. it would be difficult, if not impossible, for all 138 mass section students present to make even a single comment in a course meeting. when comparing white and non-white students by class type, we found no significant differences. white students in regular sections had a slightly higher mean interaction per student when compared with non-whites (1.89 to 1.50), but the percentage of students participating was nearly identical (37.6 to 36.2%). whites and non-whites in the mass class section had identical mean interactions per student (.40) and a very similar percentage of whites and non-whites participated (17.3 to 20.0%). in sum, while we found significant differences in the participation by student age, student gender, instructor gender and class type. significant differences by student race were relatively few. white non-traditional students had a higher mean number of interactions and a higher percentage of participation in discussion compared to non-white nontraditional students. however, reversing the trend, non-white female students participated at a slightly higher, but not significant rate than did white females. white students in courses taught by female instructors had a lower mean number of interactions and a lower percentage participated in discussion compared to non-white students in courses taught by female instructors. howard, j., zoeller, a., and pratt, y. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 23 table 3: interactions per class session (anova comparison of means) and percent of students participating (kendall’s tau) by student race and class type (regular versus mass section) mean interaction per session mean attend mean no. students participate percent students participate mean interaction per student n regular section (60.6) 48.7 26.6 9.9 37.3*** 1.83*** 850 mass section (39.4%) 55.8 138.0 24.5 17.8*** .40*** 552 regular section whites (85.1%) 42.7 22.6 8.5 37.6 1.89 723 regular section non-whites (14.9%) 6.0 4.0 1.4 36.2 1.50 127 mass section whites (83.7%) 46.2 115.5 20.0 17.3 .40 462 mass section non-whites (16.3%) 9.0 22.5 4.5 20.0 .40 90 n 1779 415 1779 1402 *** significant at the p < .001 level table 4 presents the results of another way to test for differences in participation linear regression of numerous variables on mean student interactions in order to more directly assess the impact of these variables. the first regression model (i) presents results of an analysis that includes the students enrolled in the mass section of introduction to sociology. because the dynamics of interaction in the mass class may be significantly different from that in the regular sections, table 4 also presents an analysis that excludes the students in the mass section (ii). the adjusted r square for the first analysis (including the mass section) was only .146. despite the lack of significance when we compared mean interactions by white and non-white students (see table 1), we found that when we controlled for other variables in the regression analysis, being white has a significant positive affect on the mean number of student interactions per class session. other variables with significant positive effects include student age (nontraditional = 1), percentage of non-traditional students in the class, and front third seating. the percent of nonwhite students in the class had a significant negative effect. this finding suggests that when more non-white students are in the class, students as a whole may be less willing to participate. howard, j., zoeller, a., and pratt, y. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 24 observations provided some further insight. as noted above, there were occasions when non-white students became the “experts” on a given topic in class and became the dominant talkers. interestingly, this participation was not prompted by “triggers” (weinstein and obear 1992) which provoked defensive and emotional reactions from non-white students. in each case, the non-white students readily volunteered to share their experience with and knowledge of the topic. during one class session when the topic was racial stratification, four non-white students became the dominant talkers for that class session. however, some white students, through their body language, shaking of heads, sighs, and crossing of arms, seemed to disagree with the perspective of their non-white classmates, but did not verbally challenge what was being said. thus it may be that when minority students speak up regarding controversial topics, white students, out of fear of appearing to be racist, stop participating. in an interview a white male instructor commented: when we get to the units having to do with race and ethnicity, i’ve noticed that most of the white students don’t want to hear it anymore. they really don’t. when i am talking about race, or homosexuality, they are mostly silent. if anyone is going to talk, it’ll be the african americans. ….the white students usually back down on their views. i think that part is unfortunate. they pick up that i am a liberal. therefore, my views on race are going to be closer to the africanamerican students’ views. so it would be silly to alienate your instructor. if that is true, it is a shame. the effect of student gender (female = 1), instructor gender (female = 1), and attendance in session were negative in direction, but not significant in model i. the negative direction of the effect of instructor gender suggests that the higher mean interactions and higher percentage of students participating in female taught courses (see table 2) are more likely due to the smaller class size and the higher percentage of nontraditional students present than to instructor gender. the adjusted r square for the second analysis (excluding the mass class section) was .130. being white had a significant positive impact on mean student interactions. student age (non-traditional = 1) and percent nontraditional also had significant positive effects. attendance in the session had a significant negative effect. as class size went up, the amount of student participation went down. the impact of percent non-white went from negative to positive and was not significant in the second model. front third seating, while significant in model i, was not significant in model ii. the effects of student gender and instructor gender remain negative but not significant. percent female remains positive, but not significant. as noted above, there is a need for caution in examining mean interactions per student. as previous research has demonstrated (see for example, howard and baird 1998 and karp and yoels 1976) there are no mean students. there are only talkers (students who participate frequently) and non-talkers (students who only rarely participate in discussion). therefore, it is important not only to examine mean interactions per student, but also to look at which students are most likely to become talkers. a. talkers and non-talkers the norm of the consolidation of responsibility was first identified by karp and yoels (1976). the consolidation of responsibility suggests that in the typical college classroom a small number of students will assume responsibility for speaking on behalf of the entire class. for the purposes of this study, we refer to these students as “talkers.” the rest of the students will howard, j., zoeller, a., and pratt, y. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 25 table 4: linear regression of mean student interactions by student gender, age, and race, percent female, percent non-white, percent non-traditional, instructor gender, attendance in session, and seating (i) including mass class and (ii) excluding mass class. i ii includes mass section excludes mass section variable b beta b beta student race (white =1) .63* (.26) .062 1.08* (.43) .084 student gender (female=1) -.12 (.21) -.015 -.16 (.29) -.018 student age (non-traditional=1) 1.71*** (.34) .141 1.91*** (.49) .145 percent female .0275 (.02) .070 .00 (.02) .008 percent non-white -.06* (.02) -.067 .00 (.4) .001 percent nontraditional .07*** (.01) .246 .03* (.02) .113 instructor gender (female=1) -.32 (.30) -.035 -.26 (.38) -.026 attendance in session -.00 (.00) -.041 -.13*** (.04) -.173 front third seating .99*** (.20) .125 1.24 (.31) .131 constant -1.15 (1.32) 3.24 (2.29) adjusted r square .146 .130 n 849 1402 *** significant at the p < .001 level * significant at the p < at .05 level remain either non-contributors to class discussions or only occasional contributors. these students we label “non-talkers.” table 5 presents a comparison of the percentage of students making two or more interactions per class session (talkers) and a comparison of the percentage of students making two or more interactions per class session by student race, gender and age. in the typical class meeting session, seven to eight students accepted the consolidation of responsibility and became talkers, accounting for 92 percent of all student interactions. we found no significant difference in the percentage of whites and non-whites who were talkers (19.7 to 18.4%). however, white talkers had a significantly higher mean number of interactions per class session than did nonwhite talkers (6.17 to 5.07). we also found that females were significantly more likely than males to be talkers (20.9 to 16.3%). however, male talkers had a slightly, but statistically significant, higher mean number of interactions per class session when compared to female 5 significant at p < .08 howard, j., zoeller, a., and pratt, y. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 26 talkers (6.14 to 5.93). when it comes to student age, non-traditional students were almost three times as likely as traditional students to be talkers (47.9 to 16.3%). non-traditional talkers also had a significantly higher mean number of interactions per class session compared to traditional talkers (8.26 to 5.21). as table 5 demonstrates, further comparisons by race and gender revealed few significant differences. a higher percentage of white males compared to non-white males were talkers (17.3 to 10.9%) and white male talkers had a higher mean number of interactions per class session (6.36 to 4.29), however, the differences were not statistically significant. interestingly, a slightly higher percentage of non-white females were talkers compared to white females (21.6 to 20.8%). however, white female talkers had a somewhat higher mean number of interactions per class session (6.06 to 5.24), but neither difference was statistically significant. there were also no significant differences when comparing white traditional students with non-white traditional students by either percentage of students who were talkers or mean interactions per class session by talkers. a significantly higher percentage of white nontraditional students were talkers compared to non-white non-traditional students (53.5 to 34.9%). however, the difference in mean interactions per class session by white and non-white nontraditional students was not significant. thus while statistically significant differences were few, the direction of the advantage most often favored white students over their non-white counterparts. table 6 presents a comparison of mean interactions by students making two or more interactions per class session and the percentage of students who are talkers by student race and instructor gender. a significantly higher percentage of students in courses with female instructors were talkers compared with students in courses with male instructors (36.5 to 15.2%). these talkers in female taught courses also had a higher mean number of interactions per class session (7.08 to 5.31). a significantly higher percentage of white students in male taught courses were talkers compared to non-white students in male taught courses (16.3 to 9.4%) and the white talkers spoke up more often during the class period (5.44 to 4.00). the pattern differed in female taught courses. a significantly higher percentage of non-white students were talkers (51.1 to 33.6%). however, the white talkers in female taught courses had a higher mean number of interactions per class session (7.46 to 5.79). the findings suggest that non-white students are more likely than whites to become talkers in courses taught by female instructors, but the pattern is reversed in courses taught by male instructors with white students more likely to become talkers. given that the courses taught by females were both smaller and had a higher percentage of nontraditional students, and given the small size of the sample of courses, it is not possible to determine whether these differences are due to instructor gender or the size and composition of the classes. b. students perceptions of classroom discussion table 7 presents students’ self reports via survey of characteristics of talkers and nontakers. as was the case in previous research (reported above), students’ self reports of frequency of participation in class discussion exceeded what was observed. in their self reports, 75% of students reported contributing to discussion twice or more in the typical class meeting. therefore, when reporting results from survey data, we chose to define talkers as those who reported making three or more contributions to discussion per session. the resulting percentage of students who were then defined as talkers (27.6%) was still higher than percentage of talkers observed (19.5%). using survey data we were also able to make comparisons of talkers and non howard, j., zoeller, a., and pratt, y. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 27 table five: mean interactions by students making two or more interactions per class session (anova comparison of means) and percent of students making two or more (twoplus) interactions per class session (kendall’s tau) by student race, student gender, and student age no. students making two plus interactions % students making two plus interactions mean interactions by two plus students no. interactions by two plus students % all interactions by two plus students n all 7.61 19.5 5.98 45.5 92.1 274 whites (84.5%) 6.50 19.7 6.17*** 39.9 92.5 234 non-whites (15.5%) 1.11 18.4 5.07*** 5.6 89.4 40 males (24.1%) 1.83 16.3* 6.14*** 11.3 91.6 66 females (75.9%) 5.78 20.9* 5.93*** 34.3 92.2 208 traditional (74.8%) 5.69 16.3*** 5.21*** 29.7 89.7 205 nontraditional (25.2%) 1.92 47.9*** 8.26*** 16.3 96.9 69 white males (24.3%) 1.64 17.3 6.36 10.4 92.1 59 non-white males (4.6%) .19 10.9 4.29 .8 85.7 7 white females (60.2%) 4.86 20.8 6.06 29.4 92.6 175 non-white females (10.9%) .92 21.6 5.24 4.8 90.1 33 white traditional (77.3%) 5.00 16.6 5.43 27.1 90.5 180 non-white traditional (12.4%) .69 14.4 3.64 2.5 82.0 25 white nontraditional (7.2%) 1.50 53.5* 8.48 12.7 97.0 54 non-white nontraditional (3.1%) .42 34.9* 7.47 3.1 96.6 15 n 1638 274 *** significant at the p < .001 level * significant at the p < at .05 level howard, j., zoeller, a., and pratt, y. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 28 table 6: mean interactions by students making two or more interactions per class session (anova comparison of means) and percent of students making two or more (twoplus) interactions per class session (kendall’s tau) by student race and instructor gender no. students making two plus interactions percent students making two plus interactions mean interactions by two plus students no. interactions by two plus students percent all interactions by two plus students n all 7.61 19.5 5.98 45.5 92.1 274 male instructor (79.7%) 7.08 15.2*** 5.31*** 37.6 89.0 170 female instructor (20.3%) 8.67 36.5*** 7.08*** 61.3 96.1 104 white students male instructor (67.5%) 6.42 16.3** 5.44*** 34.9 90.2 154 non-white students male instructor (12.1%) .67 9.4** 4.00*** 2.7 76.2 16 white students female instructor (17.0%) 6.67 33.6* 7.46*** 49.8 95.8 80 non-white students female instructor (3.4%) 2.00 51.1* 5.79*** 11.6 97.2 24 n 1638 274 *** significant at the p < .001 level ** significant at the p < .01 level * significant at the p < at .05 level talkers by their seating (front third versus back two thirds), class standing (freshmen and sophomores versus juniors and seniors), and by expected grades. table 8 presents a comparison of students’ perceived responsibilities by race (whites versus non-whites) and level of participation (talkers versus non-talkers). of the seven responsibilities at least 86 percent of all students agreed that six were part of their responsibility. there was a strong consensus that attending class, completing assigned tasks, studying for exams and quizzes, paying attention in class, learning the material and asking for help when needed were each a part of the students’ responsibilities. however, when it came to responsibility for participation in class discussion, less than 71 percent agreed this was part of the student’s responsibilities. howard, j., zoeller, a., and pratt, y. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 29 using an anova comparison of means, students who reported they were talkers had a significantly higher mean age than self reported non-talkers (24.7 to 21.0 years). this is consistent with our observations. non-traditional students were more likely to be talkers than traditional students (see table 5). using kentall’s tau, we found no significant self reported differences in the percentage of students who were talkers between whites and non-whites, females and males, juniors and seniors versus freshmen and sophomores, nor by student grades. however, significantly more non-traditional students reported being talkers compared to traditional students (49.2 to 23.1%) and significantly more students in female taught courses reported being talkers when compared to students in male taught courses (34.9 to 24.3%). finally, significantly more students seated in the front one third of the classroom reported being talkers than students in the back two thirds of the classroom (36.3 to 21.6%). these results are consistent with our observations, nontraditional students and students in female instructors’ courses were more likely to be talkers. differences by instructor gender, however once again, must be interpreted with caution because female instructors taught smaller courses with more nontraditional students compared to the courses taught by male instructors. when we compared white and non-white students, we found no significant differences in students’ perceived responsibilities except for responsibility to “learn the material.” a significantly higher percentage of white students agreed this was a student responsibility than did non-white students (97.6 to 88.3%). further examination revealed it was the non-talkers who differed in their responses on this responsibility. while white and non-white non-talkers had similar levels of agreement on five of the responsibilities, white non-talkers were significantly more likely than non-white non-talkers to agree students had a responsibility for learning the material (96.5 to 84.2%). another difference was asking for help from the instructor when needed (85.3 to 71.1%). this difference would be significant at p < .08. a greater percentage of non-white talkers reported their agreement with responsibility for participation in classroom discussion (64.3 to 52.6%), but the difference was not statistically significant. these findings are difficult to interpret. consistent with ballard and clanchy’s (1991) argument, we hypothesize that they may be due to prior experience in the educational system. non-white students’ primary and secondary experience may have emphasized more rote and teacher-centered learning than that of white students. as such non-white students may have been socialized to take a more passive approach to learning, seeing the teacher as more responsible than the student for learning and ensuring that each student understood what was being taught. non-whites may have also experienced a primary and secondary educational setting where teachers emphasized control and order in the classroom to a greater extent than they emphasized creativity and initiative in learning. further research will be necessary to test this hypothesis. white and non-white talkers agreed on each of the student responsibilities except participation in classroom discussion. interestingly, 100 percent of the small number of selfreported non-white talkers (n=16), indicated their agreement with responsibility for participation compared to almost 91 percent of white talkers. while this difference was statistically significant, both groups had a very high level of agreement. table 9 presents a comparison of students’ reasons for participation in discussion by level of participation (talker versus non-talker) and race (white versus non-white). we found no significant differences in responses of whites and non-whites. again, perhaps because of the small number of non-white non-talkers (n=38) and non-white talkers (n=16), statistically significant results were difficult to demonstrate. reasons for participation by white and nonwhite non-talkers were very similar. the top reasons cited for participation by non-talkers were howard, j., zoeller, a., and pratt, y. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 30 table 7: survey characteristics of talkers and non-talkers (kendall's tau except where indicated) characteristic non-talkers (72.4%) talkers (27.6%) n mean age (oneway anova) 21.0 24.7*** 347 percentage of white students (86.8%) 72.9 27.1 354 percentage of non-white students (13.2%) 70.4 29.6 54 percentage of female students (75.1%) 73.1 26.9 305 percentage of male students (24.9%) 71.4 28.6 98 percentage of traditional students (82.4%) 76.9 23.1*** 286 percentage of non-traditional students (17.6%) 50.8 49.2*** 61 percentage in female taught courses (31.2%) 65.1 34.9* 129 percentage in male taught courses (68.8%) 75.7 24.3* 284 percentage of front third seating (41.5%) 63.7 36.3*** 171 percentage of back two-thirds seating (58.5%) 72.5 21.6*** 241 percentage junior/senior (17.6%) 70.4 29.6 71 percentage frosh/soph (82.4%) 72.6 27.4 332 percentage self-defined a student (25/4%) 66.7 33.36 96 percentage self-defined b student (51.9%) 70.9 29.1 196 percentage self-defined c student (20.1%) 78.9 21.1 76 n 298 114 412 *** significant at the p < .001 level ** significant at .01 6 grade differences significant at .065 howard, j., zoeller, a., and pratt, y. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 31 table 8: students' perceived responsibilities by race and level of participation and race (kendall's tau) my responsibilities as a student include : (circle all that apply) all white nonwhite white nontalker non-white nontalker white talker nonwhite talker attend class 98.9 98.9 98.3 99.2 97.4 100 100 complete assigned tasks 98.6 98.8 96.7 98.8 94.7 100 100 study for exams/quizzes 97.3 97.6 95.0 96.9 92.1 100 100 pay attention in class 97.1 97.6 93.3 97.7 89.5 100 100 learn the material 96.4 97.6* 88.3* 96.5* 84.2* 100 93.8 ask for help from the instructor when i need it 86.5 87.8 80.0 85.3 71.17 95.8 93.8 participate in class discussion 70.7 72.1 66.7 64.3 52.6 90.6** 100** n 438 378 60 258 38 96 16 * significant at .05 ** significant at .01 “i have something to share,” “i need clarification,” “participation may help my grade,” and “my instructor creates a comfortable atmosphere by sharing about him/herself.” the least frequently cited reasons for participation in discussion by non-talkers were “it is required,” “if i don’t, no one else will,” and “i disagree with something the instructor said.” non-white non-talkers more frequently cited “it makes the class more interesting” as a reason for participation (31.6 to 24.8%), but the difference was not statistically significant among talkers, again there were no statistically significant differences in reasons cited for participation in classroom discussion. however, non-white talkers cited “i need clarification” more often than did white talkers (75.0 to 54.2%), a difference that would be significant at p < .1. non-white talkers also more frequently stated “i learn more when i participate” than did white talkers (81.3 to 61.5%), a difference that would be significant at p < .09. but the larger picture is one of agreement between white and non-white talkers on their reasons for participation in discussion. table 11 presents reasons why students choose not to participate by their level of participation (talker versus non-talker) and race (white versus non-white). we again found no significant differences by race. whites and non-whites were very similar in the degree to which they cited the four top reasons “i am shy,” “the feeling that i don’t know enough about the subject matter,” “i have nothing to contribute,” and “my ideas are not well enough formulated.” while the differences were not statistically significant, non-whites more frequently indicated they did not participate because “of the chance i would appear unintelligent to other students” (28.3 to 21.2%), “of the chance i would appear unintelligent to the instructor” (25.0 to 18.5%), and “i have not completed the assigned tasks” (26.7 to 17.2%). the only significant difference between white non-talkers and non-white non-talkers or white talkers and non-white talkers was that among talkers whites were more likely to avoid participation because of the perception that the instructor does not want participation or discussion (7.3 to 0.0%). 7 significant at p < .08 howard, j., zoeller, a., and pratt, y. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 32 table 10: reasons for participation in discussion by level of participation and race (kendall's tau) in this class, i participate in discussion because: (circle all that apply) all white non-white white nontalker non-white nontalker white talker non-white talker i have something to share 52.5 51.2 58.3 44.0 44.7 75.0 87.5 i need clarification 46.0 45.0 53.3 42.6 47.4 54.2 75.08 participation may help my grade 40.6 41.3 36.7 34.1 28.9 63.5 56.3 my instructor creates a comfortable atmosphere by sharing about him/herself 38.4 37.8 43.3 31.0 34.2 62.5 68.8 i learn more when i participate 37.2 35.7 45.0 25.6 28.9 61.5 81.39 it makes the class more interesting 34.8 34.4 35.0 24.8 31.6 62.5 43.8 i disagree with something another student said 25.1 25.1 23.3 19.4 18.4 42.7 37.5 i am familiar and comfortable with my classmates 23.3 23.8 20.0 19.8 13.2 36.5 37.5 the instructor calls one me 21.9 22.5 18.3 24.4 18.4 20.8 25.0 i don't participate in discussion 18.3 24.4 23.7 2.1 6.3 i disagree with something the instructor said 15.8 15.3 16.7 12.4 7.9 25.0 37.5 i am trying to help other students 12.6 12.7 11.7 9.3 7.9 25.0 25.0 if i don't, no one else will 12.0 12.7 8.5 12.0 8.1 17.7 12.5 it is required 10.2 10.6 8.3 10.9 10.5 11.5 6.3 n 408 378 60 258 38 96 16 8 significant at p <, .10 9 significant at p < .09 howard, j., zoeller, a., and pratt, y. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 33 table 11. reasons why students choose not to participate by level of participation and race (kendall's tau) in this class, when i choose not to participate in discussion i do so because: (circle all that apply) all whites nonwhites white non-talker non-white non-talker white talker non-white talker i am shy 42.0 43.4 35.0 50.8 47.4 24.0 12.5 of the feeling that i don't know enough about the subject matter 33.0 34.7 25.0 35.7 23.7 33.3 18.8 i have nothing to contribute 29.6 30.2 28.3 30.4 26.3 33.3 31.3 my ideas are not well enough formulated 22.9 22.8 25.0 22.9 21.1 22.1 25.0 someone else will participate therefore i don't need to. 22.8 23.3 20.0 28.3 18.4 13.5 31.3 of the chance i would appear unintelligent to other students 21.9 21.2 28.3 26.0 26.3 12.5 25.0 of the chance i would appear unintelligent to the instructor 19.2 18.5 25.0 22.9 26.3 7.3 12.5 i have not completed the assigned tasks (i am not prepared for class) 18.5 17.2 26.7 16.3 23.7 19.8 31.3 the class is too large 14.7 14.3 18.3 15.9 21.1 7.3 18.8 the course is not interesting to me 7.7 8.0 6.7 9.7 5.3 4.2 0.0 the instructor does not want participation or discussion 4.5 4.5 5.0 3.5 5.3 7.3* 0.0* of the possibility class may end early if no one participates 3.8 4.0 1.7 4.7 2.6 3.2 0.0 n 408 378 60 258 38 96 16 howard, j., zoeller, a., and pratt, y. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 34 iv. conclusion while antonio, et al’s (2004) work has demonstrated the benefits of multi-racial group discussion for white students, our research raises concerns about the participation of non-white students. the conclusions of this preliminary investigation, of course, are limited because it was conducted at a single university with non-white enrollment of approximately 15 percent and because the sample included only introductory sociology courses. clearly further research at a variety of institutions with a range of non-white enrollments and including a variety of disciplines is needed. while the results of this case study are mixed with regard to the impact of race on participation in classroom discussion, there is evidence that presents cause for concern. based on our regression analysis, white students are likely to participate at a significantly higher rate than non-white students. while there are occasions when the non-white students become the dominant participants in discussion (e.g., when discussing topics related to race), we also found evidence that white students may choose to disengage from these very discussions. likewise, we found some evidence that an increase in the percentage of non-white students may decrease overall participation. these preliminary findings require further investigation of the type described above. a major area for further examination is the impact of race on the participation of various minority groups. for example, we need to ask, do the interaction patterns of asians, for example, differ significantly from those of african americans or hispanics? do white students tend to withdraw from discussion only when large numbers of particular minority groups are enrolled in the course? or are they likely to be silent regardless of which minority group is represented? we also need to further investigate the topics which spark the participation of nonwhite students in class and determine how to encourage their interaction without closing the door to discussion and debate among students. does the percentage of minority students matter in this regard? are the classroom interaction dynamics different on a campus with 50 percent minority enrollment versus 15 percent minority enrollment? our study also failed to find a difference in the reasons why whites and non-whites chose to participate or not participate. survey studies will need to directly address the attitudes and behaviors of whites and non-whites with regard to one another as well as the usual reasons given for participation and non-participation. finally, we need to address the role of instructor gender. studies that include a larger number of courses with a larger number of instructors are necessary before any firm conclusions can be drawn with regard to the relationship between instructor gender and race. given the increasing attention that is being paid to race in higher education by academics and non-academics, continued study of the actual experiences of minority students within higher education is clearly warranted. as dedlacek (1983) suggested, different teaching methods may be necessary to facilitate the success of minority students. thus in order to be able to see the world from the viewpoint of minority students, as wu and morimoto (1983) argue is necessary, investigations at a wide range of campuses with a wide range of minority students will be necessary to capture the experiences of minority students in american higher education. references adams, maurianne (1992). “cultural inclusion in the american college classroom.” new directions for teaching and learning, vol. 49, pp. 5-18. howard, j., zoeller, a., and pratt, y. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 35 aitken, joan e., & neer, michael r. (1993). “college student question-asking: the relationship of classroom communication apprehension and motivation.” southern communication journal, vol. 59, pp. 73-81. antonio, antonio, chang, mitchell j., hakuta, kenji, kenny, david a., levin, shana, & and milem, jeffrey f. (2004). “effects of racial diversity on complex thinking in college students.” psychological science, vol. 15, no. 8, pp. 507-510. astin, alexander w. (1985). “achieving educational excellence.” san francisco: josseybass. auster, carol j. & macrone, mindy (1994). “the classroom as a negotiated social setting: an empirical study of the effects of faculty members’ behavior on students’ participation.” teaching sociology, vol. 22, pp. 289-300. ballard, brigid & clanchy, john (1991). “teaching students from overseas.” cheshire, melbourne: longman. boersma, p. d., gay, d., jones, r.a., morrison, l., & remick, h. (1981). “sex differences in college student-teacher interactions: fact or fantasy?” sex roles, vol. 7, pp. 775-84. brooks, v. (1982). “sex differences in student dominance behavior in female and male professors’ classrooms.” sex roles, vol. 8, pp. 683-90. cabrera, alberto f., crissman, jennifer l., bernal, elena m., nora, amaury, terenzini, patrick t., & pascarella, ernest t. (2002). “collaborative learning: its impact on college students’ development and diversity.” journal of college student development, vol. 43, no. 1, pp. 2034. cabrera, alberto f., nora, amaury, terenzini, patrick t., pascarella, ernest t., & hagedorn, l. (1999). “campus racial climate and the adjustment of students to college: a comparison between white students and african-american students.” journal of higher education, vol. 70, no. 2, pp. 134-60. canada, katherine & pringle, richard (1995). “the role of gender in college classroom interactions: a social context approach.” sociology of education, vol. 68, pp.161-186. constantinople, ann, cornelius, randolph, & gray, janet m. (1988). “the chilly climate: fact or artifact?” the journal of higher education, vol. 59, pp. 527-50. cornelius, randolph, gray, janet m. & constantinople, ann (1990). “student faculty interaction in the college classroom.” journal of research and development in education, vol. 23, pp. 189-97. crawford, mary, & macleod, margo (1990). “gender in the college classroom: an assessment of the “chilly climate” for women.” sex roles, vo. 23, pp. 101-22. howard, j., zoeller, a., and pratt, y. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 36 crombie, gail, pyke, sandra w., silverthorn, naida, jones, alison, & piccinin, sergio (2003). “students’ perceptions of their classroom participation and instructor as function of gender and context.” the journal of higher education, vol. 74, pp. 51-76. dedlacek, william e. (1983). “teaching minority students.” new directions for teaching and learning, vol. 16, pp.39-50. fassinger, polly a. (1995). “understanding classroom interaction: students’ and professors’ contributions to students’ silence.” the journal of higher education, vol. 66, pp. 82-96. faust, debra f., & courtenay, bradley c. (2002). “interaction in the intergenerational freshman class: what matters?” educational gerontology, vol. 28, pp. 401-422. feagin, joe r., hernan, vera, and imani, nikitah. (1996). “the agony of education: black students at white colleges and universities.” new york: routledge. feagin, joe r. (2003). “race and its continuing significance on our campuses: an interview with dr. joe r. feagin.” black issues in higher education, vol. 19, no. 24, pp. 24-27. fritschner, linda m. (2000). “inside the undergraduate college classroom: faculty and students differ on the meaning of student participation.” the journal of higher education, vol. 71, pp. 342-62. garside, colleen (1996). “look who's talking: a comparison of lecture and group discussion teaching strategies in developing critical thinking skills.” communication education, vol. 45, pp. 212-27. hall, roberta m. & sandler, bernice r. (1982). “the classroom climate: a chilly one for women?” project on the status and education of women. washington, d.c.: association of american colleges. hardiman, rita & jackson, bailey w. (1992). “racial identity development: understanding racial dynamics in college classrooms and on campus.” new directions for teaching and learning, vol. 52, pp.21-38. heller, jack f., puff, c. richard, & mills, carol j. (1985). “assessment of the chilly college climate for women.” journal of higher education, vol. 56, pp. 446-461. howard, jay r., & baird, roberta (2000). “the consolidation of responsibility and students’ definitions of the college classroom.” the journal of higher education, vol. 71, pp. 700-721. howard, jay r., & henney, amanda l. (1998). “student participation and instructor gender in the mixed age college classroom.” the journal of higher education, vol. 69, pp. 384-405. howard, jay r., short, lillard b. & clark, susan m. (1996). “student participation in the mixed-age college classroom.” teaching sociology, vol. 24, pp. 8-24. howard, j., zoeller, a., and pratt, y. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 37 howard, jay r., james, george, & taylor, david r. (2002). “the consolidation of responsibility in the mixed-age college classroom.” teaching sociology, vol. 30, pp. 214234. johnson, david w., johnson, roger t., & smith, karl a. (1991). “cooperative learning: increasing college faculty instructional productivity.” ashe_eric higher education report no. 4. washington, d.c.: george washington university. jung, jan, moore, rosemary l., & parker, jim (1999). “a study of older students’ participation in the college classroom.” unpublished. karp, david a., & yoels, william c. (1976). “the college classroom: some observation on the meaning of student participation.” sociology and social research, vol. 60, pp. 421-39. kember, david, & gow, lyn (1994). “orientations to teaching and their effect on the quality of student learning.” journal of higher education, vol. 65, pp. 58-74. mckeachie, wilbert j. (1990). “research on college teaching: the historical background.” journal of educational psychology, vol. 82, pp. 189-200. neer, michael r., & kircher, w. faye (1989). “apprehensives’ perceptions of classroom factors influencing their class participation.” communication research reports, vol. 6, pp. 70-77. nunn, claudia e. (1996). “discussion in the college classroom: triangulating observational and survey results.” the journal of higher education, vol. 67, pp. 243-66. o’keefe, tim f., & faupel, charles e. (1987). “the other face of the classroom: student ethnography.” sociological spectrum, vol. 7, pp. 141-155. pascarella, ernest t., wolniak, gregory c., & pierson, christopher t. (2003). “influences on community college students’ educational plans.” research in higher education, vol. 44, no. 3, pp. 301-314. pearson, judy, & west, richard (1991). “an initial investigation of the effects of gender on student questions in the classroom: developing a descriptive base.” communication education, vol. 40, pp. 22-32. schwitzer, alan m., ancis, julie r., & griffin, oris t. (1999). “validating a proposed model of african-american students’ social adjustment.” journal of the first-year experience and students in transition, vol. 11, no. 1, pp. 77-102. smith, daryl g. (1977). “college classroom interactions and critical thinking.” journal of educational psychology, vol. 69, pp.180-90. statham, a., richardson, l., & cook, j.a. (1991). “gender and university teaching: a negotiated difference.” albany: suny. howard, j., zoeller, a., and pratt, y. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 38 saufley, ronald w., cowan, katherine o. & blake, j. herman (1983). “the struggles of minority students at predominately white institutions,” new directions for teaching and learning, vol. 16, pp. 3-15. sternglanz, sarah hall, & lyberger-ficek, shirley. (1977). “sex differences in student-teacher interactions in the college classroom.” sex roles, vol. 3, pp. 345-52. terenzini, patrick t., cabrera, alberto f., colbeck, carol l., bjorklund, stefani a., & parente, john m. (2001). “racial and ethnic diversity in the classroom: does it promote student learning?” journal of higher education, vol. 72, no. 5, pp. 509-531. terenzini, patrick t., yaeger, patricia m., bohr, louise, pascarella, ernest t., & nora, amaury (1997). “african american college students’ experiences in hbcus and pwis and learning outcomes.” paper presented at the annual meeting of the association for institutional research (orlando, fl, may, 1997). tinto, vincent (1975). “dropout from higher education: a theoretical synthesis of recent research.” review of educational research, vol. 45, pp. 89-125. tinto, vincent (1997). “classrooms as communities: exploring the educational character of student persistence.” the journal of higher education, vol. 68, pp. 599-623. weast, don (1996). “alternative strategies: the case for critical thinking.” teaching sociology, vol. 24, pp. 189-194. weinstein, gerald & obear, kathy (1992). “bias issues in the classroom: encounters with the teaching self.” new directions in teaching and learning, vol. 52, pp. 39-50. wu, jean & morimoto, kiyo (1983). “on careful listening.” new directions for teaching and learning, vol. 16, pp. 51-57. journal of the scholarship of teaching and learning, vol. 6, no. 1, august 2006, pp. 39 55. metacognitive knowledge monitoring and self-regulated learning: academic success and reflections on learning randy m. isaacson and frank fujita1 abstract: during the past decade the relationship of self-regulated learning (srl) to academic success has been extensively explored but the impact of metacognition in this process has not been thoroughly examined. this study examined the relationship of metacognitive knowledge monitoring (mkm) to classroom performance. eighty-four undergraduate students in an introductory educational psychology class completed ten weekly in-class tests in which they were allowed to choose test questions. students were asked to identify the number of hours they studied, their level of confidence, and to predict their test results after completing the test but before it was graded. high achieving students were: more accurate at predicting their test results; more realistic in their goals; more likely to adjust their confidence in-line with their test results; and more effective in choosing test questions to which they knew the answers. the study supports the relationship of metacognitive knowledge monitoring to self-regulated learning and academic success. keywords: self-regulated learning, metacognitive knowledge monitoring i. introduction. the application of self-regulation to learning is a complicated process involving not only the awareness and application of learning strategies but also extensive reflection and selfawareness. pintrich (1995) describes self-regulation as the “active, goal-directed self-control of behavior, motivation, and cognition for academic tasks by an individual student.” (p. 5) students who are skillful at academic self-regulation understand their strengths and weaknesses as learners as well as the demands of the specific tasks. they approach learning with an assortment of strategies they might apply to achieve their goals and an understanding of when and how to implement their plan. but students who are expert learners have more than an arsenal of study strategies and the ability to regulate academic resources, they also know when they have mastered, or not mastered, the required academic tasks. that is, expert learners are also skillful at metacognitive knowledge monitoring (mkm). self-regulated learners are skillful at monitoring their learning and comprehension which has a direct effect on each step in the self-regulation process. pintrich et al. (2000) compares monitoring to the thermostat of a furnace. when the temperature falls below a specified level the thermostat tells the furnace to turn on the heat; when a learner is confused or does not comprehend what they are studying the monitor tells the learner to regulate their behavior, cognitive strategies, or motivation and affect to increase learning. to be effective learners, students must adjust their efforts based on their awareness of their own understanding and the level of difficulty of the upcoming task. one of the critical barriers to success for many students may be their inability to objectively assess their mastery of the academic tasks they are facing. accurate monitoring of learning can impact self-regulation throughout the learning 1 respectively, professor of education, indiana university south bend, risaacso@iusb.edu and associate professor of psychology, indiana university south bend, ffujita@iusb.edu. isaacson, r.m., and fujita, f. journal of the scholarship of teaching and learning, vol. 6, no. 1, august 2006. 40 process. zimmerman (1998) proposes three phases to self-regulation which incorporate metacognition. the first phase is forethought which includes goal setting, strategic planning, and self-efficacy; students identify their goals, their plans for achieving them, and consider how likely it is they will achieve their goals. the second phase is performance or volitional control which includes attention focusing, self-instruction, and self-monitoring; students attempt the learning tasks and monitor what they are learning. the third phase is self-reflection which focuses on comparing self-monitored information with a standard or goal and reactions to the results. during the reflection stage students assess their success or failure, modify their selfefficacy, make causal attribution, and adapt for future learning. in all three of these phases students are using academic goals as the yardstick against which they assess their learning; using the thermostat metaphor, goals are the set-temperature the thermostat uses to judge whether to turn-on the furnace. in each phase, self-reflection and selfmonitoring are critical to master the skills of self-regulation as well as the content being learned. in each phase students who are skillful at self-regulation are cognizant of their understanding and adjust their goals and self-efficacy based on internal as well as external feedback on their mastery of the tasks. students’ ability to monitor their learning is one of the key building blocks in selfregulated learning; students who are aware of the level of their mastery of material can adjust their study time and strategies. over the past decade a number of concepts have been used to describe students’ awareness of their learning. these concepts were originally investigated at a micro-level focusing on metamemory using an experimental format. ease-of-learning (eol) judgements refer to a student’s inferences about how easy or difficult a task will be to learn (nelson & narens, 1990; nelson & leonesio, 1988). eol judgments occur in advance of actual learning, but clearly are a stumbling block for students who underestimate the difficulty of the material or the level at which it must be mastered. judgments of learning (jol) occur during or after learning and are predictions of future test performance (kelemen, 2000; koriat, 1997; nelson & narens, 1990). students whose jol overestimate their actual learning are also likely to terminate their studying prior to mastering the material and fall short of their goals, especially when the demands of the tasks exceed the levels at which they have learned the required material. these assessments of learning have important implications for self-regulated learning as each influence the thermostat that adjusts how much time and effort a student devotes to studying. but the primary focus of the research on eol and jol has been on metamemory with very little attention to the higher level of concepts and problem solving which occurs in classrooms, particularly in postsecondary classrooms. in these studies metamemory is not a measure of understanding or comprehension, but rather the ability to retrieve information from long term memory. while obviously not irrelevant to classroom learning, these measures are primarily applicable to the most basic learning which occurs in school. the connection to the self-regulation of cognitive strategies is probably most suitable to rehearsal or the most rudimentary forms of elaboration. skillful self-regulators should be able to go beyond the assessment of their ability to recall facts; effective self-regulators should be able to estimate how well they have mastered a body of knowledge and how well they will be able to demonstrate their mastery. skillful self-regulators should be able to predict how well or poorly they will do, and have done, on a test while naive self-regulators should be less able to estimate their academic success. a series of studies by maki and colleagues has shown the significance of metacognition isaacson, r.m., and fujita, f. journal of the scholarship of teaching and learning, vol. 6, no. 1, august 2006. 41 on the comprehension of college students in experimental situations. in a study by maki and berry (1984) students who scored above the median on multiple choice items of text showed more accuracy in their test prediction than students who scored below the median. students who had done well on the test were also better at predicting their test scores: greater metacomprehension could be inferred to influence self-regulation. students who were asked to process information at high levels were also found to be more accurate in their test predictions (maki, et.al., 1990) as were students who had increasingly more information about the text (maki and serra, 1992). maki invited the connection between metamemory, metacomprehension, and classroom learning when she found an experimental relationship between higher order thinking questions and prediction accuracy. studies of metamemory had used simple recall questioning, but maki (1995) found that there was greater accuracy of predictions when students were asked higher order questions that did not require verbatim recall of information. this was in support of the finding of weaver (1990) who had proposed that multiple questions had increased the reliability and calibration of comprehension assessment. these finding have important implications for post-secondary education which places greater value on higher level thinking skills. do successful college students have better metacognitive awareness on tasks that require higher level thinking skills? can they adjust their study skills in response to the increasing demand of these tasks? the transition from high school to college puts many demands on young adults. in the classroom the greatest challenge may be the move from the declarative knowledge emphasized in high school to the higher level thinking skills typically required in college. when students do not recognize that these new demands require new approaches to learning and studying they may be unwilling and/or unable to make the necessary changes. at the core of this problem may be that students do not realize that their learning does not match the demands of the task. they assume they have learned the material if they can recall the important terms. they do not recognize that different academic tasks (e.g., different test formats) demand different levels of learning. they overestimate their understanding because they do not recognize the implication of different levels of learning and varying levels of task difficulty. to be successful in college, students may need to have a variety of srl and metacognitive skills that were not necessarily essential in high school. first, successful college students recognize that professors expect more than the memorization of declarative knowledge. second, successful college students use accurate mkm while studying to assess their mastery of the required material, particularly in relation to what will be require on the performance task (e.g., test). third, successful college students have an arsenal of srl strategies they can choose from to match their level of learning to the demands of the performance task. and finally, successful students are able to self-monitor their understanding and the demands of the performance task during evaluation to adjust their demonstration of the learning (e.g., to choose the right questions to answer during a test.) while it may be very difficult to assess metacognition before or during learning, it is possible to assess mkm during the assessment of their learning. while taking a test are expert students better at identifying what they know and what they do not know? for the past decade a program of research by tobias and everson (2000, 2002) has examined learners’ ability to differentiate between what they know and do not know. their findings indicate learners of all levels of ability and developmental stages are affected by their ability to monitor their learning. in dozens of studies with students of all ages and abilities, tobias and everson have found that students who are able to differentiate between when they know and when they do not know are more likely to excel than students who are not able to isaacson, r.m., and fujita, f. journal of the scholarship of teaching and learning, vol. 6, no. 1, august 2006. 42 distinguish their level of comprehension. the studies by tobias and everson have focused on the correlation between knowledge monitoring and student’s academic performance. but very little evidence exists which explores the relationship between knowledge monitoring and academic choices within classes: how do students who possess effective knowledge monitoring skills use these skills to make decisions which impact their academic success? these experimental studies of metamemory and metacomprehension are an open invitation to naturalistic studies performed in actual classrooms. studies on eol (nelson & narens, 1990) suggest that students who underestimate the difficulty of classroom content may abandon their learning efforts before they have mastered the material. experimental studies on jol (kelemen, 2000; koriat, 1997) indicate that students overestimate their understanding which can lead to discontinuing learning efforts prior to mastery. schommer & surber (1986) demonstrated an illusion of knowing (ik) when students believe that only shallow processing is necessary when the material is actually difficult, which could lead some students to underestimate the necessity of learning classroom content to the degree which will be required for success. each of these issues have clear implications for self-regulation in the classroom context. when students in post-secondary education are presented with learning tasks that require higher level thinking they must accurately judge the difficulty of material to be learned, accurately judge the level of their own understanding, and accurately judge the requirement of the performance task (e.g., test) they will be given in class. while laboratory predictions of test performance are revealing of student’s selfmonitoring there is little classroom research on this topic. in the first study of classroom confidence, shaughnessy (1979) reported high achieving students as being better able to distinguish between known and unknown information. in a study of self-efficacy, sinkavich (1995) reports a significant difference between high and low achieving students on their confidence on individual test items. hacker et al. (2000) report similar findings; high performing students were accurate in predicting their test results with their accuracy improving over multiple exams, while low performing students were poor at predicting test results. hacker et al. (2000) found significant differences between low performing students and high performing students in relation to their ability to predict (before taking a test) and postdict (after taking a test but before receiving their grade) their test results. the implication being that students who were doing poorly in a college course were unable to monitor their knowledge of the course material (i.e., they overestimate their test results in both prediction and postdiction) and therefore were unable to regulate their studying to assure mastery of the course material. the student sample in the hacker et al. study were college students in an undergraduate educational psychology class in which the students took three tests. on each test the highest achieving students on that test were more accurate in their predictions and postdictions, but the analysis did not focus on the overall achievement of students across the semester. would the highest achieving students for the semester be more accurate in their postdictions and would they be more likely to accurately adjust their test postdictions on a test-by-test basis? the present study examines students across ten tests during a one semester undergraduate educational psychology course comparing intra-individual differences for low and high preforming students. all three levels of zimmerman’s (1998) academic self-regulation learning cycle emphasize the importance of goals setting. in the forethought stage students set goals, in the performance and volitional control stage students monitor their learning in relation to their goals, and in the reflection stage students assess their success or failure in relation to their goals. extensive research has focused on goal setting (locke & latham, 1990) but no classroom isaacson, r.m., and fujita, f. journal of the scholarship of teaching and learning, vol. 6, no. 1, august 2006. 43 research to date has examined the impact of goal setting and goal achievement on self-regulation. the present study will explore the relationship among the variables of goal setting, selfmonitoring, performance, and self-regulation. in the performance and volitional control stage of zimmerman’s (1998) learning cycle students are attempting the learning task and monitoring their mastery of the task. in this cycle students who possess good mkm should be able to reflect on the application of their learning to the task and choose appropriate tasks. when given choices of tasks of varying difficulty, such as a selection of diverse test questions, students with good metacognitive awareness should be capable of choosing tasks in which they will succeed and avoid tasks in which they are less likely to succeed. the present study will examine the ability of students to use mkm to choose between tasks of varying difficulty. the self-reflective stage in zimmerman’s (1998) learning cycle examines the outcomes when students compare self-monitored information to their goals, especially in relation to the impact on self-evaluation, self-efficacy, and adaptation of strategies. the combination of goals with knowledge of performance impacts self-efficacy which heightens motivation (bandura, 1997). skillful self-regulators effectively monitor their progress in relation to their goals and then adjust their self-efficacy and future strategies. the present study will begin to explore how the feedback students receive from completing tasks of varying difficulty (e.g., tests) impacts their self-efficacy as they progress through a college course. this study will explore the metacognitive differences between high achieving students and low achieving students. high achieving students, who have good mkm, should be more reflective and thoughtful about decisions they make in their studying, test taking, and selfefficacy. are students who excel across the semester more likely to be more accurate in their estimation of their individual test grades and do they accurately adjust their estimations based on their mastery of the material? do students who excel have goals that are more consistent with their academic performance? do students who excel have realistic self-efficacy and how does that self-efficacy change over the course of a semester? are students who excel more likely to make accurate task choices based on their understanding of the required material. ii. method a. participants the participants were 84 undergraduate college students (59 females, 25 males) enrolled in an introductory educational psychology course on a commuter campus of a mid-western university. all students were university students enrolled as education majors. b. procedures participants took weekly objective tests (true-false and multiple choice) and completed a questionnaire for each test. part of the questionnaire was completed before the participants took the test and part immediately after taking the test but before scoring the test. prior to taking the test each student was asked to report the number of hours they had studied, how many points they would have to achieve to be satisfied with their performance (satisfaction goal), how many points they would have to achieve to be proud of their performance (pride goal), and how confident they were about achieving their satisfaction goal (pre-test self-efficacy). after completing the test, but before it was graded, each student was asked to identify isaacson, r.m., and fujita, f. journal of the scholarship of teaching and learning, vol. 6, no. 1, august 2006. 44 how many points they believed they would achieve on the test2 and how confident they now were about their achieving their satisfaction goal (post-test self-efficacy). then, tests were graded and were returned to the student for review before the student was dismissed from class. each of the weekly tests included 40 objective test questions: 18 lower level test questions which emphasized knowledge and comprehension and were worth 1 point each; 18 moderately difficult questions which emphasized application and were worth 2 points each; and 4 difficult test questions which emphasized analysis and synthesis and were worth 3 points each. for each weekly test, students were allowed to answer only 30 of the 40 test questions; their grade being dependant on both the accuracy of their answers (number of questions correct) and the type of test questions they chose and answered correctly. to earn an a in the class, students had to choose more difficult test questions (worth 2 or 3 points) and get them correct. to earn a lower grade, students could either take more difficult questions and get a lower percentage correct, or take less difficult test questions (worth 1 or 2 points) and get a higher percentage correct. therefore, the key to success in the course was not only correctly answering test questions, but also choosing the test questions you could answer correctly. the weekly tests were designed to reveal and substantiate student metacognitive awareness during testing. ten tests were administered during the semester (approximately one per week). iii. results this study examines the differences between high and low achieving students on a number of metacognitive variables: what are the long-term changes between their estimations of test grades early to late in the semester?; how are their satisfaction and pride goals different from their actual performance?; how does their self-efficacy change and how does that compare to their actual test score?; and, are students able to make academic choices based on their mkm? a. test scores postdiction following the format used by hacker (2000) we examined the hypothesis that high performing students would be more accurate in predicting their test scores than low achieving students. using hacker’s terminology, our participants made “postdictions” because they took the test before estimating their score on the test. for each of the ten tests, a correlation was computed between the test score and the squared error of the student’s postdiction [(test score – expected test score) 2]: let us call this type of correlation the matched-score format. students who have good mkm should be better at postdicting their test scores which would result in smaller squared error scores and a negative correlation between test scores and squared error scores in the matched-score format. the average (median) of these ten correlations was -.26 (all listed correlations are significant at p < .05 unless noted otherwise). for the three most difficult tests, the correlations were -.27, -.49, and -.65. thus, the students who were most accurate in their postdictions (having a low squared error of postdiction) tended to have higher test scores and this was particularly true on the most difficult tests (see table 1). from the previous analysis (matched-score format) it is possible that all students are 2 hacker (2000) makes the distinction between predictions and postdictions. predictions are a student’s estimates of their test scores before they take a test. postdictions are a student’s estimates of their test scores after they have taken the test but before their tests are graded. in the present study the students estimated their results after having taken the test but before the test is graded. the students were not asked to make predictions before taking the test. isaacson, r.m., and fujita, f. journal of the scholarship of teaching and learning, vol. 6, no. 1, august 2006. 45 equally good (or poor) at postdicting their test scores, they all postdict that they will do well, and that on any particular test a few students do poorly at random. to insure that it was the students who consistently performed well on their tests who were making the best postdictions, we also computed correlations between the total of the ten test scores (total points) and each of the ten squared error of postdictions. for example, for the first correlation, we took the squared error of postdiction for test 1, and correlated it with the student’s total points for the semester. let us call this type of correlation the total score format. this calculation was done for each of the ten tests. the median of these ten correlations was -0.18. for the three most difficult tests, the correlations were -0.27, -0.24, and -0.53. thus, the students who have the highest achievement across the semester are better at postdicting test results, and their postdiction accuracy is most pronounced on the most difficult tests. this confirms the findings by hacker (2000) and others (maki and berry, 1984; maki, et.al., 1990; maki and serra, 1992) that demonstrates that high performing students are better at metacognitive awareness, knowledge monitoring, and calibrating how they will do on tests in college. are there also differences in the goals students set and the changes that occur over time during a semester? table 1: correlations between test score and the squared error of predicted test score matched-score format total-score format chapter mean stddev r n p r n p 2 3 4 6 7 8 20 11 14 15 43.0 41.2 43.7 43.5 38.4 44.3 41.7 43.5 37.1 37.1 5.2 6.7 5.0 5.7 6.8 6.8 6.2 7.0 6.3 7.4 -0.270 -0246 +0.054 -0.392 -0.274 -01.54 -0.163 -0.055 -0.492 -0.651 81 78 81 81 77 81 80 77 80 75 0.02 0.04 0.64 0.00 0.02 0.17 0.15 0.64 0.00 0.00 -0.228 -0.190 -0.014 -0.133 -0.273 -0.180 -0.161 -0.238 -0.527 -0.160 76 77 78 79 75 78 77 76 77 74 0.05 0.10 0.91 0.24 0.02 0.11 0.16 0.17 0.04 0.00 mean median -0.264 -0.258 -0.210 -0.185 b. group differences by goals across time four roughly equally sized groups were created based on the total performance across the ten tests. to examine the differences between high achieving groups and low achieving groups in relation to goal setting, expected performance, and actual performance across the ten tests during the semester, a repeated measures manova was performed. the three factors were performance group (four levels), measure (satisfaction goal points, pride goal points, expected points, actual points), and time (the ten different tests). the important significant difference for this analysis was the group by measure interaction (f(9,107.2) = 5.76, p<.05, partial 02 = .274). the highest performing group has small differences between goal points, expected points, and actual points. the lowest performing group had very large differences between satisfaction and pride goal points (high), expected points (high), and actual points (low). the two intermediate groups had appropriately intermediate differences between the two extremes. the three-way isaacson, r.m., and fujita, f. journal of the scholarship of teaching and learning, vol. 6, no. 1, august 2006. 46 interaction was not significant. thus, the lowest performing group of students did not adjust their satisfaction or pride goal points, nor their expected points, to the reality of their actual points. this further supports the hypothesis that low achieving college students are less likely to use metacognitive awareness to make adjustments as they are learning in a college course. the actual test scores of the highest achieving students are very similar to their satisfaction goals and their expected points, with their pride goals figure 1: examination points as a function of type of points and performance level of student 35.00 38.00 41.00 44.00 47.00 50.00 53.00 actual points expected points satisfaction goal pride goal measure e x a m in a ti o n p o in ts a students b students c students d & f students approximately 4 points above their achievement. when the highest achieving students are incorrect in their postdiction they are more likely to under-postdict their score which may have led to more extensive studying before the test. as shown in figure 1, he actual test score of the lowest achieving students are significantly less than their expected points and their goals across the entire semester which may have led to less studying before their test. expected points, goals, and self-efficacy can vary dramatically both across individuals and within individuals across the semester. when a high achieving student is able to make accurate postdictions, is that just because the high achieving student lives up to the universal expectation of good performance, or does the high achieving student appropriately lower expectations when they are going to perform at a lower level in a way that the low achieving student does not? that is, in a course which has a weekly test even high achieving students occasionally have a bad day and if the hypothesis about metacognitive awareness is correct these students should make adjustments to their postdictions, self-efficacy, and goals as the semester progresses. c. intra-individual differences across time to focus on differences within individual students across the semester, we calculated isaacson, r.m., and fujita, f. journal of the scholarship of teaching and learning, vol. 6, no. 1, august 2006. 47 within-subject correlations of the test score, postdicted test score, hours studied, pre-test selfefficacy, and post-test self-efficacy within each individual student across the ten tests. each student will have, for example, a correlation between his or her ten test scores and his or her ten postdicted test scores to help explore how changes in test scores compare to changes in postdictions which indicate whether changes in a student's postdictions across the semester reflect changes in their test scores across the semester. the postdictions of students with effective mkm should rise and fall in concert with their test results indicating that they knew when they did well or poorly on test. these individual correlations are characteristics of each student, and as such can be considered a type of individual difference variable. these individual difference variables were themselves correlated across students with each student’s final grade. these correlations have a number of important implications for self-regulation. intra-individual post-diction accuracy across time. each student has a within-subject correlation between his or her postdicted points and actual points across the ten tests, which reflects the extent to which each student adjusts their postdictions to match their actual score across the test tests during the semester. let’s call this within-subject correlation “relative postdictive accuracy.” the mean relative postdictive accuracy across all students is 0.24. thus, using the weekly test scores and the student’s postdictions for each test, the average student is able to make a somewhat accurate relative postdiction of whether he or she will do better (or worse) on this exam than on the other nine exams. the between-subjects correlation of the relative postdictive accuracy with total points is 0.26 (p=.056) meaning that students whose relative postdictive accuracy is greater than the mean tend to score higher across the ten tests during the semester. students who are more accurate in adjusting their estimation of how well they have done on tests from week to week are more likely to achieve more total points during the semester. reliance on effort for pre-test self-efficacy. each student has a within-subject correlation between his or her number of hours studied and pre-test self-efficacy. let’s call this within-subject correlation “reliance on effort.” the mean reliance on effort is 0.30. the average student is more confident the more hours he or she studied for that test. but the between-subject correlation of reliance on effort with total points is -0.24 (p=0.12) meaning that students whose reliance on effort is greater than the mean are less likely to do well in the class. this could be interpreted to mean that the more a student depends on the number of hours they have studied to decide on their confidence for success, rather than on mkm to decide on how confident they are about the test, the less likely they are to do well on a test. it is good to be able to weigh metacognitive feedback during test preparation rather than being forced to depend primarily on the amount of effort expended when making a prediction on how well you’ll be able to do on an exam. metacognitive changes in post-test self-efficacy. each student has a within-subject correlation between his or her pre-test self-efficacy for achieving their satisfaction goal and their post-test self-efficacy for achieving their satisfaction goal across the ten tests. essentially this measures how much each student is likely to change their self-efficacy for achieving their satisfaction goal from before they take the test to after having taken the test. let’s call this within-subject correlation “self-efficacy constancy.” the mean self-efficacy constancy is 0.30. but the between-subjects correlation of self-efficacy constancy with total points is -0.39 (p<.05). thus, students whose self-efficacy constancy is greater than the mean tend to score lower than students whose self-efficacy constancy is less than the mean. students who use the feedback they receive from taking a test to adjust their self-efficacy are more likely to do better on tests isaacson, r.m., and fujita, f. journal of the scholarship of teaching and learning, vol. 6, no. 1, august 2006. 48 across the semester. this is consistent with what we would expect from students with good metacognitive awareness in that they are aware of how they have done after taking a test but before it is graded. these three individual difference variable all support the theory that high achieving students are consistently monitoring their understanding of their learning and adjusting their postdictions (relative postdictive accuracy), adjusting the time they spend studying (reliance on effort), and are better judges of how well they have done after completing a test (self-efficacy constancy) than low achieving students. this begs the question: can metacognitive knowledge monitoring be taught to students and will improvements in mkm lead to improvements in learning? d. metacognitive impact on choosing test questions each of the tests in the course used a variability difficulty variable weight test format where students were given choices about which questions they selected to be graded. this test format allowed students to eliminate questions to which they did not know the answer but at a cost: choosing more difficult questions earns more points but only if you get the correct answer. for each of the 10 exams, students received a number of questions correct out of 30. each student also had a number of points earned for the exam. since students are allowed to choose 30 questions from 40 available questions, and since their total points are dependant upon choosing 30 questions that they are likely to get correct, it is critical that students choose the appropriate questions from week to week depending upon their mastery of the material. that is, some weeks students have clearly mastered the material and can choose questions which are more difficult and worth more points. other weeks when students have not mastered the material as well, the students with good mkm can choose less difficult questions to which they know the answers while students with poor mkm are likely to guess at the answers and guess at which questions to choose to answer. to assess the accuracy of students' choices we correlated the number of points earned with the number of questions answered correctly within student so that each student had a within-subject correlation coefficient that measured the relationship between the number of questions answered correctly and the student’s score on the exam. we will call this within-subject correlation question dependency (qd). if all of the questions on the exam were worth the same number of points, qd would be 1.0 for all students. because students chose which questions to answer, and different questions were worth differing numbers of points, the average qd was 0.94, the minimum qd was 0.74, and the maximum qd was 0.99. while it is clear that the number of questions answered correctly is the key variable in any student’s score, some students are able to assess their understanding of the material and the test question and choose the appropriate test questions to positively influence their grade. students with high metacognitive skill should have a lower qd, because when they know that they do not understand the material well they will choose easier questions to answer, when they understand the material well they will choose more difficult questions which are worth more points, lowering their qd. students with high metacognitive skills know when they understand the material and are more likely to be able to adjust their choice of test questions based on accurately reflecting upon their learning. since the test format allows them to not answer some questions they can make their choices based on their assessment of their understanding. their test points will depend relatively less on the number of questions answered correctly and relatively more on the difficulty of the questions chosen. students with high metacognitive skill should also, on average, score higher on the exams. isaacson, r.m., and fujita, f. journal of the scholarship of teaching and learning, vol. 6, no. 1, august 2006. 49 a correlation across students was calculated between each student’s average test score and that student’s qd. the between-subjects correlation was negative ( r = -0.33, p < .01) as expected. students with low qd had higher average test scores which is evidence of their metacognitive awareness; they used metacognitive strategies to adjust their test item selection based on their knowledge monitoring. the student whose test scores are more question dependant are most likely to do well based on whether they get questions correct, independent of which questions they choose. expert college students in the present study adjusted their choice of test questions to enhance their test scores based on knowing-when-they-knew. it can be inferred that low achieving students were less accurate in selecting the questions they knew based on their metacognitive awareness of their understanding of the material and their comprehension of the test questions. iv. discussion the literature on metamemory and metacomprehension has demonstrated that better learners are able to make more accurate judgments about their learning. the classroom application of these findings for self-regulated learning has not been thoroughly explored although the work of hacker et.al. (2000) begins to shed light on these possibilities. our findings begin to make a connection between the accuracy of students’ metacognitive judgments of their learning and a number of variables related to self-regulation. the initial step, in zimmerman’s three step academic learning cycle, is forethought which includes goal setting and self-efficacy, the second step is performance and volitional control which is guided by self-monitoring, and the third stage is reflection including assessing success or failure and modifying self-efficacy. to be an effective self-regulated learner a student must use mkm to guide this process and make adjustments in goals, judgments of learning, selfefficacy, and task choice. this study begins to explore these relationships within individual learners. the first study of metacomprehension within a classroom context (hacker, et.al., 2000) confirmed the finding that high achieving students are better at predicting and postdicting their learning. our study supports this finding in relation to student postdictions. students’ postdictions correlated significantly to their test scores. of special interest was the finding that this correlation was greater for tests that were more difficult which supports the finding of maki (1995). this relationship has potential implications for metacognition and self-regulation. if difficult tests are better at differentiating high achievers from low achievers the reason may be that difficult tasks require a higher level of metacognition and self-regulation. given the demands for higher level thinking skills that occur when students transition from high school to college, it would be revealing to explore if effective mkm becomes increasing important in academic tasks that require these thought processes. the groups in the hacker et.al. (2000) study were assigned based on their results on each individual test which leaves open the possible interpretation that the reason for the discrepancy between groups could have been a regression to the mean. if the test goals of most students were approximately 80%, and each student were to predict achieving their goal, the group discrepancy between predictions and actual score would be attributable to their initial goals. since the group assignment changed across each of the three tests it is possible that group membership changed while goals, predictions, and postdictions stayed the same. in our study the group assignments were made based on the students’ accumulated test points across all ten exams. group assignment based on accumulated points across the entire isaacson, r.m., and fujita, f. journal of the scholarship of teaching and learning, vol. 6, no. 1, august 2006. 50 semester rules out the possible regression to the mean explanation. the twenty students in the top group were the students with the highest scores across all ten tests. the twenty students in the bottom group were the students with the lowest scores. the repeated measures manova demonstrates that the top group of students had the smallest differences between their satisfaction and pride goals, expected points, and actual points across the course of the semester while the lowest group of students had the widest discrepancy between these variables. future research should examine whether students’ mkm changes over time, what factors might impact these metacognitive skills, and whether these skills can be taught to lower achieving students. the results from hacker et.al (2000) also suggest that high achieving students are not only more accurate in their judgements but more likely to under-estimate their results (leading to under-confidence) while low achieving students were more likely to grossly over-estimate their results (resulting in overconfidence.) in these conditions self-regulation theory would predict that high achievers who are under-confident might be defensive-pessimists (garcia and pintrich, 1994) which would lead to an increase in their efforts and their success. students who overestimate their result may be self-handicapping which may result in a decrease in their efforts which would lead to failure. the relationship between calibration, confidence, and self-efficacy will need to be more fully explored since self-protective perceptions may influence the accuracy of calibration (dembo and jakubowski, 2003). what is the impact when students underestimate or overestimate their mastery of the course material? how does this impact studying before a test? what impact does this underestimation and overestimation have on motivation to study? according to zimmerman (1998), adjusting goals, expectations, and self-efficacy over time is a critical skill in self-regulated learning. the weekly class test format in our study allowed students to set and readjust goals and expectations before each test during the course of the semester. the pattern of pride and satisfaction goals, expected points, and achievement for the high achieving group is consistent with self-regulation, while the pattern for the low achieving students does not reflect the reality of their actual test scores. low achieving students maintain the same general expectations which may lead them to learned helplessness. future studies should explore the affective reactions of students who do not adjust their goals, their expectations in the face of continual failure, and their attributions for success and failure. one of the advantages of the present study is the longitudinal nature of the data set. since students are given weekly tests and the same data is collected for each test, we can examine the intra-individual differences for a number of variables. self-regulation involves small changes over time within, as opposed to across, individuals. to explore changes in selfregulation it is important to examine how the changes of one student week-to-week compare to the changes within another students. these intra-individual relationships can be examined as individual difference which can then be compared to variables such as final course grade. our study explored three such relationships. the first intra-individual difference we are calling “relative postdiction accuracy.” this is a measure of whether a student is able to predict whether they will do better on this test relative to other tests. the average student is somewhat accurate in estimating whether they will do better on this test compared to other tests, but the highest achieving students are better at judging when they will do better or worse on a test compared to their own performance across the semester. while teachers at all levels may view this as "common sense", the ability of students to judge how they have performed on this test compared to other tests is indicative of the mkm that is critical to academic success in college. high achieving students are not only better at estimating their score on an individual test, they are also sensitive to whether they will isaacson, r.m., and fujita, f. journal of the scholarship of teaching and learning, vol. 6, no. 1, august 2006. 51 do better or worse than they usually perform on the class tests. whether the reason is that they were not able to study as much as usual or they estimate that the material will be more difficult, high achieving students are aware that on this test they are likely to perform better or worse than they usually perform. self-regulated learning in zimmerman’s performance stage (1998) is dependent upon on-line monitoring of performance and when students are not able to assess whether they are doing better or worse than they normally do they are less likely to adjust their behavior. the second intra-individual difference is “self-efficacy constancy.” students were asked to estimate their confidence for achieving their satisfaction goal before they took the test, and after they took the test but before the test was graded. students who are metacognitive about the feedback they receive from taking a test adjust their self-efficacy based on this feedback. students who lack metacognitive awareness do not change their self-efficacy. high achieving students are more likely to modify their confidence for an individual test after having taken the test; "that test was harder/easier than i thought it would be." this is also evidence of the metacognitive awareness necessary for self-regulation. when students do not, or cannot, adjust their self-efficacy after taking a test, it is likely they are not metacognitively aware of how they have done. the third intra-individual difference is “reliance on effort.” many students link their test confidence (i.e., pre-test self-efficacy) to the number of hours they study for a test. when a student has studied four hours for a test they are usually more confident than when they have only studied one hour. but students who are truly self-regulating do not tie their confidence solely to the number of hours they have studied. self-regulating students make judgments of when to stop studying based on how well they know the material and whether they believe they will be able to achieve their goals. many students seem to have a pre-designated number of hours they plan on studying and if they complete those hours they believe they are sufficiently ready for the test. when students rely primarily on time to regulate their learning they may be less likely to succeed. the correlation of “reliance on effort” with total points indicates that the more a student relies on time to determine their pretest self-efficacy the lower their total points in the class. the relationship of metacognitive awareness, pre-test confidence, and self-regulation is an important variable to explore. many students depend on time as the leading indicator of learning and the result is often failure. how students decide they have spent enough time learning is an important question to be explored. the process of self-regulation depends on intra-individual changes in expectations, goals, and self-efficacy. students who are skillful at self-regulation modify what they expect based on the feedback they receive from self-monitoring and external input (e.g., tests). they also adjust their goals and self-efficacy based on these results. the process of learning self-regulation requires extensive time and feedback. the present study begins to follow the development of this process in a college classroom with frequent feedback over an entire semester. the variable weight variable difficulty test format in this study places a strong emphasis on student’s metacognitive awareness. given this test format it is possible for a student to get all 30 of the questions they choose correct and still only earn a c, if they choose the easiest test questions. students are told, and quickly learn, that their grade is dependant not only on the number of questions they get correct, but also choosing the correct test questions. this test format encourages students to take questions that are worth more points, which require higher level thinking skills, with the expectation that they will eventually learn to regulate their study time and strategies to go beyond the simple memorization of facts to the application, analysis, isaacson, r.m., and fujita, f. journal of the scholarship of teaching and learning, vol. 6, no. 1, august 2006. 52 and synthesis of information. but learning at this higher level is not the only skill necessary to succeed with this test format. it is also important for students to know-when-they-know. students who used elaboration and organizational strategies in learning would be more likely to do well if all students were required to take all the questions of highest value. if all students took all the higher level questions, the student who got the most questions correct would have the highest score. but since students can choose which questions to take their score is also dependant upon their ability to choose the question they will get correct. the question dependency (qd) variable helps to reveal a student’s ability, across the semester, to choose questions that will enhance their test score based on their mkm. students who lack mkm are less likely to be able to assess whether they understand, and can answer, each individual test item. for these students their test score is entirely dependent upon the number of questions they get correct because they do not make metacognitive choices on individual items on the test. for students with high mkm their test score is influenced by the questions they choose as well as the number of items they get correct. for example, a student with high mkm who, in a given week, is not able to devote as much time as usual to studying for the higher level thinking test questions is likely to choose lower level difficulty test questions on that test. these students are aware that they do not know the answer to the more difficult test questions and, instead of guessing, choose the easier questions for which they know the answer. their mkm influences their total score which makes them less question dependent. but students with poor mkm cannot make these choices because they do not know-when-they-know and therefore are guessing: guessing not only on the answer to the question, but also on whether they should choose the question. this finding invites further exploration of the relationship of metacognitive awareness and learning particularly when higher level thinking is required. many undergraduate students have difficulty when they first enter college because they are not familiar with the academic demands of higher level thinking. if they were effective in memorizing information in high school they were likely to have received good grades and also were likely to have believed their were good at learning. this would lead them to believe they would be successful in college if they were to use the same learning strategies. when they enter college and fail to meet their own expectation, they are then more likely to externalize the blame for their failure to the teacher or an unfair task. this may keep them from engaging in the reflection which is essential to changing their study behavior and improving their metacognitive awareness. are high achieving college students better able to predict their scores on difficult tests because the questions are more difficult (maki, 1995) or because the test requires higher level thinking skills? this study also raises the question of whether metacognitive awareness can be improved over time. it is clear that the highest achieving students in this study were better able to predict their test scores and also choose the right test questions. are the metacognitive skills that allow these students to choose the right test questions a stable characteristic, or can these skills be improved over time? if these skills can be improved, what instructional approaches would facilitate an improvement in their metacognitive awareness that would impact their self-regulated learning and success in school? this study and others (e.g., hacker et.al., 2000; tobias and everson, 2000, 2002; maki, 1995) are demonstrating a strong relationship between mkm and academic performance but the causal relationship is unclear. can an improvement in mkm lead to academic improvement or is improved mkm a result of improved academic performance? this study demonstrates that expert students are effective at estimating their understanding (postdicting their test scores) and they are more inclined to vary their goals and isaacson, r.m., and fujita, f. journal of the scholarship of teaching and learning, vol. 6, no. 1, august 2006. 53 self-efficacy based on past results and the feedback they receive from taking a test. expert students in this study were also more likely to make choices which demonstrate mastery and non-mastery of tasks of varying levels of difficulty (choosing the appropriate test questions.) this has important implications for the teaching-learning process in higher education and secondary education. as students are required to take on academic tasks of increasing difficulty it is critical that they have the metacognitive skills to assess their mastery of the material on a variety of levels. this metacognitive self-assessment is essential to the application of selfregulated learning. this study also invites a thorough examination of the relationship of mkm to learning. it seems clear that expert students are skilled in reflecting on their own learning but the origin and nature of these skills is not clear. does mkm facilitate student learning or does the mastery of a body of knowledge assist students in their judgement of their mastery of the material? does mkm change over time and can it be taught? if mkm is a skill that can be learned, what pedagogical changes in classroom practice are most likely to encourage students to be more reflective of their own learning? what types of assessment practices are most likely to encourage students to be more metacognitively aware? are there procedures that can be implemented in most postsecondary classrooms (e.g., frequent evaluation and immediate feedback) that would encourage students to be more metacognitively aware? can technology be used outside the classroom to assist students to improve their mkm? the author is presently implementing pedagogical approaches that are designed to make students aware of the impact of metacognitive knowledge monitoring and assist them in improving their self-awareness of their learning. this study also raises questions about how to assess metacognitive knowledge monitoring. this study demonstrates the challenges inherent in assessing the metacognitive awareness of students while they are taking a test which reveals the even more difficult task of assessing metacognitive decision making while students are studying which clearly is the more crucial connection between metacognitive knowledge monitoring and self-regulated learning. this study clearly demonstrates that low achieving students frequently over-estimate how well they know a body of information which leads to disengagement early during studying. this process typically results in failing the test which often leads them to blame the teacher/test for their failure instead of examining their own learning. a thorough examination of the impact of metacognition, and pedagogical approaches that might increase metacognition, are an important issue that should be addressed in the scholarship of teaching and learning. references bandura, a. (1997). self-efficacy: the exercise of control.. new york, ny: w. h. freeman. dembo, m., t.g. jakubowski (2003, april). “the influence of self-protective perceptions on the accuracy of test predictions.” presented at the annual conference of the american educational research association, chicago, il. garcia, t., p. r. pintrich (1994). “regulating motivation and cognition in the classroom: the role of self-schemas and self-regulatory strategies.” in d.h. schunk & b.j. zimmerman (eds.), self-regulation of learning and performance: issues and educational application. (pp. 127153). hillsdale, nj: erlbaum isaacson, r.m., and fujita, f. journal of the scholarship of teaching and learning, vol. 6, no. 1, august 2006. 54 hacker, d. j., bol, l., horgan, d. d., & rakow, e. a. (2000). “test prediction and performance in a classroom context.” journal of educational psychology, 92, 160-170. kelemen, w. l. (2000). “metamemory cues and monitoring accuracy: judging what you know and what you will know”. journal of educational psychology, 92(4), 800-810. koriat, a. (1997). “monitoring one's own knowledge during study: a cue-utilization approach to judgements of learning.” journal of educational psychology, 126(4), 349-370. locke, e. a., & latham, g. p. (1990). a theory of goal setting & task performance. englewood cliffs, nj: prentice-hall. maki, r. h. (1995). accuracy of metacomprehension judgments for questions of varying importance levels. american journal of psychology, 108(3), 327-344. maki, r. h., & berry, s. (1984). metacomprehension of text material.” journal of experimental psychology: learning, memory, and cognition, 10(4), 663-679. maki, r. h., j. m. foley, w. k. kajer, r. c. thompson, & m.g. willert. (1990). “increased processing enhances calibration of comprehension.” journal of experimental psychology: learning, memory, and cognition, 16(4), 609-616. maki, r. h., & m. serra. (1992). “the basis of test prediction for text material.” journal of experimental psychology: learning, memory, and cognition., 18(1), 116 126. maki, r. h., & serra, m. (1992). the basis of test prediction for text material. journal of experimental psychology: learning, memory, and cognition, 18(1), 116-126. nelson, t., & narens, l. (1990). “metamemory: a theoretical framework and some new findings.” in g. bower (ed.), the psychology of learning and motivation. san diego, ca: academic press. nelson, t. o., & leonesio, r. j. (1988). “allocation of self-paced study time and the "labor-invain" effect.” journal of experimental psychology, 14(4), 676-686. pintrich, p. r., walters, c., & baxter, g. p. (2000). “assessing metacognition and self-regulated learning.” in g. schraw and j. c. impara (ed.), issues in the measurement of metacognition (pp. 43 97). lincoln, ne: buros institute of mental measurement. pintrich, p., r. (1995). “understanding self-regulated learning”. in p. r. pintrich (ed.), understanding self-regulated learning (pp. 3 12). san francisco, ca: jossey-bass. schommer, m., & surber, j. r. (1986). “comprehension-monitoring failure in skilled adult readers.” journal of educational psychology, 78(5), 353-357. shaughnessy, j. j. (1979). “confidence-judgment accuracy as a predictor of test performance.” journal of research in personality, 13, 505-514. isaacson, r.m., and fujita, f. journal of the scholarship of teaching and learning, vol. 6, no. 1, august 2006. 55 sinkavich, f. j. (1995). “performance and metamemory: do students know what they don't know?” journal of instructional psychology, 22(1), 77. tobias, s., & everson, h. (2000). “assessing metacognitive knowledge monitoring.” in g. schraw & j. impara (eds.), issues in measurement of metacognition (pp. 147-222). lincoln, ne: buros institute of mental measurement. tobias, s., & everson, h. (2002). knowing what you know and what you don't: further research on metacognitive knowledge monitoring. college board report no. 2002-3. college board, ny. weaver, c. a., iii. (1990). “constraining factors in calibration of comprehension.” journal of experimental psychology: learning, memory, and cognition, 16(2), 214-221. zimmerman, b. j. (1998). “developing self-fulfilling cycles of academic regulation: an analysis of exemplary instructional models.” in d. schunk & b. zimmerman (eds.), self-regulated learning: from teaching to self-reflective practice (pp. 1 19). new york, ny: guilford. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006, pp. 56 – 74. connecting, making meaning, and learning in the electronic classroom: reflections on facilitating learning at a distance charlene johnson and william brescia, jr.1 abstract : the increasing use of technology to meet the vast educational needs of our expanding world has led to heightened concerns about learning experiences within educational environments that are removed from the immediate purview of instructors (duffy & kirkley, 2004). recent calls for use of more collaborative environments in which students interact with their instructors as well as with fellow students, have become more pronounced because of the purported learning benefits for students (bonk, 2002). constructivism, a theory of learning that is based on collaboration and interactions, provides such an environment (jonassen, 1991). we report on the experiences of one professor who maintained a constructivist approach while teaching a foundation course in education (classroom learning theory) in a distance education setting. emphasized are the challenges associated with creating the appropriate conditions for learning when moving from the face-to-face interactions of the regular classroom to the setting of compressed video. the implications of the medium for her role as facilitator, the establishment of a learning community, techniques of questioning and inquiry, and group collaboration are addressed. the impact of the medium and the greater cultural diversity of the distance education classes on how the tenets of constructivism are manifested and experienced by the students is also discussed. i. introduction there is considerable use and application of technology for instructional purposes in the electronic classroom. distance education including video conferencing, web-based courses, and compressed video, has greatly increased the breadth and scope of educational outreach (mangan, 2001). educators use distance education to span the distance between groups and to ensure equitable access to educational opportunities for those interested in receiving additional training and/or expanding their educational repertoire in a number of disciplines (mangan, 2001; raymond, 2000). accompanying the increased use of distance education and its technological tools is the clarion call for effective pedagogical strategies to ensure that the learning processes within this medium are as authentic and effective as those provided in traditional on-site, face-to-face (f2f) classrooms. to ensure that the learning experiences are equitable to those offered in f2f classrooms, bernard, derubacava & pierre (2000) recommend that courses offered via distance education be comparable in format and content to the f2f courses. given the structural and technological realities of teaching at a distance, adhering to this recommendation can be 1 assistant professor of preventive medicine and director of instructional technology, university of tennessee health science center, 930 madison avenue, room 836, memphis, tn 38163, 901/448-5506, bbrescia@utmem.edu. johnson, c. and brescia, jr., w. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 57 challenging. fostering and maintaining the interactions that are endemic to f2f classrooms further compounds the difficulty. interactive instructional approaches that complement and build upon students’ existing knowledge base and experiences are touted as being highly effective for enhancing the learning experiences for a wide range of students, and for fostering higher-order thinking and problemsolving skills (abdal-haqq, 1998; keiny, 1994). the student’s role in interactive instructional environments is to work with others to discover, construct, and participate in social collaborations that bring about meaning (crumpacker, 2001). this is true for courses offered f2f as well as those offered via distance (mcalpine, 2000). constructivism as a philosophy and approach to learning embodies these axioms. this article examines one instructor’s quest to remain true to her constructivist beliefs while leading a classroom of aspiring teachers via compressed video, an interactive video format. her experiences while teaching the course via distance to three groups of students over four years are compared across the three classes as well as with those of the students she taught f2f. the foci of the discussion are the contributions, challenges, cultural and logistical implications arising from maintaining an effective constructivist environment via distance; and, the influence these experiences have had on the professor’s beliefs regarding teaching and learning processes. the questions addressed in this article are: 1. can an instructor guided by constructivism remain true to the major precepts of this approach when physically distant from students? 2. how are learning community, collaboration, question/inquiry, and student voice manifested in distance-learning environments? 3. does distance differentially affect students’ reactions to a theoretical, foundational course that is taught based on constructivism? 4. how do students’ reactions to and understanding of material delivered via distance compare to that of students in the typical, f2f classroom setting? 5. at are the implications of cultural understanding and compatibilities for reactions to and learning within constructivist classrooms via distance? ii. constructivist theory the traditional transmission model of education views learning as an infusion of facts and information from one individual to another (abdal-haqq, 1998; keiny, 1994). constructivism diverges from this pedagogical model with its emphasis on collaboration and active participation of students as they seek to understand the material and resolve any inner conflicts it may cause. students play an active role in constructing meaning from material studied based on their experiences and background knowledge (scheme). their prior conceptions form a basis for determining the meaning of new knowledge. collaboration and interaction among peers to “test” and mediate the knowledge process is integral to the process of making meaning from new experiences. learning is considered a long-term phenomenon that requires discussion, debate, and opportunities to reconstruct ideas (brooks & brooks, 1999; watts & pope, 1989). because knowledge is generated in collaboration rather than transmitted from one (a teacher) to another (a student), the role of the instructor is one of facilitator as opposed to “transmitter” of knowledge (brescia, 2003, abdal-haqq, 1998). to effectuate these principles, collaborative learning is a major practice within constructivist approaches (bonk, 2002). these ideas concerning learning and its interactive nature are founded and endorsed by a number of theorists including jean piaget, lev vygotsky and paulo friere. johnson, c. and brescia, jr., w. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 58 for piaget, the development of scheme, the cognitive mental organization within the minds so fundamental to learning, hinges on experiences and the meanings individuals attribute to them (wadsworth, 1996). these meanings are mediated by interactions with others. vygotsky’s view on cognitive processes differed from piaget’s in that he did not see cognition as being influenced by interactions, but as being determined by them. the implications of culture for these interactions and the meanings given to them are an integral part of his ideas on learning (wertsch, 1985). freire (1970) brought another dimension to the interactive nature of learning by proposing the idea of critical pedagogy. rather than education serving as a mechanism to assimilate youth into the existing order of things, critical pedagogy seeks to provide young people with the mechanism to build anew and revise the existing order of ideas and concepts. freire was critical of the “banking” paradigm in education which asserts that we seek to “deposit” knowledge into children, with little to no recognition of their culture, i.e., what they bring to the educational experience, or the relevance of the new information for what they already know. this process of depositing is considered detrimental to the authentic, contextual learning which freire contended is necessary for the incorporation of information, lifelong learning, and praxis – action based on knowledge. for freire, teachers facilitate learning and do not determine or provide the learning. as with constructivism, the trend or tradition of teachers as transmitters of knowledge is antithetical to freire’s beliefs about learning and the transformative nature of the learning process. each of these theorists emphasized raising the level of cognitive processing by building and expanding on what students bring with them experientially, culturally, and educationally so that they become learners capable of analytical reasoning in any learning environment. these ideas/concepts are especially pertinent for distance education given the nature of the medium and the variety of contexts that converge in the virtual classroom. likewise, educators who believe in these principles should endeavor to ensure that authentic, transformative learning processes are germane to their instructional approaches. facilitating is an instructional technique which is essential to effectuate these principles. for preservice teachers, coursework that reflects these principles enhances their ability to make meaning of foundational theories and furthers their understanding, incorporation and subsequent use of these principles when they construct educational practices (keiny, 1994; abdal-haqq, 1998). to ensure that students/preservice teachers are exposed to these principles in action, instructional strategies such as group work and focusing on major ideas are regularly used (muirhead, 2001). in groups, ideas are discussed, debated, and negotiated with peers and instructors. these class dialogues are integral to students processing and formulating meaning of the material across varied perspectives. focusing on major concepts rather than facts in isolation and ideas out of context allows students to incorporate information in a more meaningful and contextualized manner (abdal-haqq, 1998; van looy, callaert, debackere, & verbeek, 2004). both practices reflect the belief that knowledge is constructed and predicated on the relevance or meaningfulness of the knowledge for the student. for distance learning these practices are even more pertinent given the nature of the medium with students who differ by location, experience, and educational background. negotiation of meaning is more complex and richer given these differences but its success hinges on the learning community that is developed. establishing a learning community is critical to the success derived from these interactive practices for student learning. the community ethos is an essential component of a constructivist classroom. it provides the safety net that frees students to share their experiences and ideas in an johnson, c. and brescia, jr., w. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 59 otherwise risky environment. establishing a distance environment in which students feel safe makes it possible for learners to share their knowledge across the various sites (brown, collins & duguid, 1989). these principles and practices are standard within classes taught by the authors whether they are f2f or via distance. in the next sections, the context, i.e., course, program and class activities; students and their reactions, including those on campus as well as those at a distance; and the instructor and her reactions are discussed. iii. constructivism f2f a. course information this course is part of a five-year, masters of arts in teaching (m.a.t.) licensure program. it is an upper-level (junior or senior) educational psychology/learning theory course required for all education majors. it is taken after a student has completed the introductory course in education. the instructor taught this course three years in a f2f setting; was among the inaugural group who developed and taught the course; and has been integrally involved in its development and evolution over the years. the instructors who collaborated on the design of the course ascribe to constructivist principles and agreed that the course would reflect these ideas. the activities were developed for f2f classes based on constructivist ideology. b. students the majority of students in the f2f classes are traditional in that they range in age from 19 – 23 years old, are single, have limited experiences outside high school, and have few responsibilities beyond themselves. most of the students are of european descent and grew up in ethnically homogenous areas of the state. the course is a core course for all students majoring in education, and, as a result, each class contains students from various concentrations: elementary, secondary, music, art, agricultural, and special education. c. classroom practices – student reactions over the years, it has become evident that students are basically unfamiliar with anything which approximates constructivism in an educational environment. to expeditiously introduce them to this approach, the following paragraph is included in the instructor’s syllabus for the course. course requirements: class attendance/participation: a constructivist approach is employed in class whereby students are encouraged to be actively involved in and responsible for their learning. students are encouraged to read the material, note any questions or areas where concepts were not fully understood, and raise these questions during the class. however, the readings will not be “rehashed” during class. the focus in class is on students’ interpretation and synthesis of the reading material; different dimensions of the issues/ideas will be explored. a variety of learning approaches are used within the class for interpreting and analyzing the material. johnson, c. and brescia, jr., w. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 60 during the first days of class, this passage is discussed including what is explicit in this paragraph and the implied expectations. to further familiarize students with the approach and its tenets, one of the topics assigned for group presentation is constructivism. peers (group members) give their interpretations of the approach and the underlying rationale for the instructor’s seemingly ambiguous responses and unstructured activities. although largely unfamiliar with the constructivist approach, some students do report they have “heard about” the instructor from other students who have taken the course from her (mixed reviews). based on these reports, the students have some preconceptions about the instructor. however, the instructor has no knowledge of them as students/learners. every class is novel with its own unique personality and needs; the ongoing challenge is to develop each distinctive personality into a community of learners whereby the principles of learning within constructivism’s parameters can be manifested. the development of a learning community is integral to the success of this approach because it is vital to elicit sincere, reflective thinking on issues (bonk, 2002). to facilitate this type of environment, a significant amount of attention is given to “reading” and knowing the students. the first and most basic way of becoming familiar with the students is to recognize them by name. becoming familiar with names and hearing students’ voices are initial steps to building community among the students. name cards are created (students write their names on folded index cards– 5” x 8”) the first day of class. these cards are placed on the students’ desks during the first few classes, until the instructor recognizes them by name. another practice that contributes to the association of names with faces is calling on everyone at least once during the course of a class session. in addition to gaining familiarity with the students’ names, this practice indicates that value is placed on hearing the different “voices” within the class. however, a student always has a right to “say no.” that is, if they do not wish to “share” when called upon, a student can say he or she does not wish to share their perspective on that issue. this norm is established at the beginning of the semester and adhered to throughout the term. groups are another means to build community and they also contribute greatly to the mediated process of learning. for each class, there is a major assignment to be part of a group presentation for the class. this assignment reflects collaborative learning’s principle of interdependence in that the same presentation grade is given to each member of a group. individually, students write a paper or an article critique related to the research done for their contribution to the presentation. this ensures some measure of individual accountability (johnson, holubec, & roy 1984). in addition to this assignment, groups are formed within class on a regular basis to discuss or explore the concepts being studied. the format and manner for responding within the groups varies. an example of an in-class group activity that is used during the study of piaget and his theoretical principles follows. students are placed into groups and directed to depict on newsprint (markers are provided), without using sentences or paragraphs (narrative), piaget’s major theoretical precepts concerning thinking and cognition (assimilation, accommodation, organization, scheme, equilibrium, and disequilibrium) (woolfolk, 2004). the idea is not to just restate a precept (e.g., “assimilation is fitting new information into existing schemes,”– repeating a statement from the book [woolfolk, 2004, p. 31]). students are required to somehow demonstrate/illustrate the meanings and relationships of piaget’s theoretical precepts for cognitive development. initially, most students express apprehension about this assignment and johnson, c. and brescia, jr., w. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 61 cite a lack of drawing ability and/or uncertainty about how to create a depiction as reasons why they lack the skill or expertise necessary to participate in the activity. however, with prodding and encouragement, they slowly begin to discuss the meaning of the terms. eventually the class is buzzing with ideas as students become engaged in the activity. discussions concerning the meanings of the terms/precepts and the best way to illustrate and present these meanings emerge and enable students to illustrate their interpretations of the precepts in insightful ways. this activity reflects constructivist principles in that students’ backgrounds, majors, interests and educational experiences are incorporated in these depictions. a wide range of motifs including animals, knitting, sports, farming, etc., become a backdrop for explaining these concepts and their relationships. group processes are monitored by checking with each group, asking clarifying questions about the depiction and its meaning, and addressing questions the group members could not resolve for themselves concerning the precepts and/or the appropriateness of their drawings. the activity and the illustrations allow the instructor to make a firsthand assessment of students’ understanding and interpretations of the material. peers also assist in addressing misconceptions. this activity also incorporates another supporting belief of constructivism: the importance of recognizing and valuing multiple intelligences, i.e., the varied ways in which students process information (gardner, 1983). often students who struggle with traditional learning activities which emphasize memorization and recitation excel in this activity. on the other hand, many students who excel in these traditional activities are challenged by this exercise. another area of angst for students is the inquiry nature of the course. students in general are initially uncomfortable with nature of the professor’s questions and her responses to their inquiries. questions that focus on the major concepts and how students incorporate them into their practices prevail in the classes. questions are asked concerning the relevance of the assigned readings (what idea or concept in the chapter “spoke” to you and why?); familiarity with the practices shown (have you ever experienced this practice? if so, what were your reactions?); and the implications of these ideas for educational practices (given the research on the effectiveness of this approach/idea/concept/practice, will you incorporate it into your practice?). follow-up questions are often asked to further explore students’ meanings, perceptions, and interpretations of the material. the intent of these questions is to get students to grapple with the ideas and assess the implications for them as educators. often, the professor responds to students’ questions with a question: “what do you think?” initially, students are frustrated with this response because they want the instructor, to “just tell me the correct answer.” they want to be imbued with the instructor’s understanding of a concept rather than develop their own. instead, they are told that “correct” is contextual and that, although they are expected to follow certain analytical processes when studying material, they are expected to arrive at their own conclusions concerning how effectively a model explains phenomenon. as facilitator, the instructor places the onus of responsibility for addressing their issues and concerns with students – which adds to their frustration. they are given assistance in the process of making decisions about the relevance and implications of the material studied, but this is not determined for them. the question and answer procedure allows the community – both instructor and students – to better recognize any confusion or misinterpretation of material. when a student explains his/her thoughts on an issue, others gain insight into how someone else interprets the concept. listeners can then assess that interpretation in light of their own perceptions of the concept. as johnson, c. and brescia, jr., w. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 62 the semester progresses, students tend to: become increasingly active in their learning process; address each other’s questions or concerns; and accept that answers are contextual and may differ based on perspectives. iv. constructivism at a distance a. program/course information the off-campus licensure program was initiated to support the training and local employment of new teachers, particularly those of african descent (because of the largely african american population of the area). reflecting national trends in education, the number of african american teachers in the state’s schools has declined (american council on education, 2000). rural, economically depressed regions such as the one targeted by this off-campus m.a.t. program, are usually the first to feel the effects of this decline. the region has difficulty attracting and creating employment opportunities to replace the agriculture-related jobs lost because of technological innovation. the program also contributed to the flagship university’s stated mission to meet the needs of all the citizens of the state. the program was set up so that students completed their first two years of coursework at the regional community college. the upper-level courses for the undergraduate degree leading to the m.a.t. program and the graduate courses in the m.a.t. program were offered via compressed video at a location 300-plus miles from the state’s flagship university which was the instructor’s home campus. initially, it was expected that the students would travel to the flagship university to complete the last two years of undergraduate coursework and courses for m.a.t. degree. however, most of the students were nontraditional (over 25 twenty-five years of age), had families, were employed full-time, and were not able or willing to leave their families and jobs to complete the degree program in residence. distance education (interactive video) was a viable alternative for the students to complete the degree program. the only degree/licensure program offered via distance education was elementary education. instructor the instructor shared a common ancestry, of african descent, with the majority of students within the program, but was not born or raised in the state. during her time at the university, she had worked as a consultant with the schools in the area where the m.a.t. program was directed. however, the distance education course was her inaugural instructional experience teaching a university course to a class composed entirely of students from the region. when first asked to teach this theoretical course using distance education, the instructor was hesitant, resistant, and apprehensive. could a constructivist approach be implemented effectively without physical presence and proximity? the opportunity to interact with a population of students from another part of the state who did not otherwise have access to the course was a strong incentive that overrode her initial hesitancy and trepidation. the challenge was to use the advantages provided by this medium without compromising the integrity of her beliefs about the roles of learner and teacher. the instructor’s initial impression of distance education and its learning expectations (lecture, transmission model) was based on the setup of the room from which the video was to originate. the setup resembled that of a principal’s office. there was a chair set up at a table with controls and a microphone. after a consultation with the technical-support people, the setup was changed to be more in line with the instructor’s preferred mode of instruction. the chair and a podium were placed next to the controls, which eliminated the appearance of a johnson, c. and brescia, jr., w. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 63 barrier separating the instructor from the students. it was a seemingly minor change, but it gave the instructor the freedom to move around during the class which conveyed a more interactive persona. a. students over the course of four years, the course was offered three times via distance education. although similar in some respects, each group has its own unique quality. the first class (group 1) was predominantly african american (12 of 15) students; nontraditional in that 80% were over 25 and had families and jobs. these students were selected based on their excellent academic records and high grade-point averages. they were the program’s pioneers who experienced the usual intricacies of a new program. these common experiences and frustrations bonded them to each other. the second class (group 2) was similar to the first in terms of ethnicity (16 of 19 students were of african descent) but differed in terms of age and experiences; less than 60% were under 25, had families and outside jobs. in addition, the educational backgrounds were less stellar than those of the first group. for the inaugural group, high performers were selected to ensure the success of the program. the program received a significant amount of publicity and public response which was very positive. people from the community, the region, and the state were impressed with the program’s intent–providing schools in the area with well-prepared teachers– and with the distance-education format. this support was a catalyst for continuing the program for subsequent cohorts of students. those who came after the first group were recruited less aggressively and with less stringent academic requirements. the third class (group 3) differed from group 1 and group 2 in terms of ethnicity. ethnically, the class was more diverse with only nine of the eighteen students being of african descent and nine of european descent). ages and experiences resembled those of groups 1 and 2; only three of the eighteen (17%) were traditional students. as with the f2f classes, each class had its own unique personality. the varied personalities affected reactions to the lessons and the climate that evolved in the class. b. classroom practices – student reactions the distance-education students were also unfamiliar with a constructivist approach to learning and initially had mixed reactions to it. most students responded amicably to being asked to bring their realities into the classroom and share their opinions and ideas about the material and its relevance for them (knowles, 1990). in general, there had been little (if any) attention given to their experiences or perspectives by other instructors (delpit, 1995; freire, 1970). they welcomed opportunities to relate the course material to their lives and varied realities. these opportunities albeit gratifying and somewhat unexpected, did not preclude the discomfort associated with the process. overall, these were high-caliber students whose high grade-point averages (pervasive in the first class, somewhat common in the second and third classes), were indicative of their success with the traditional transmission model of learning. similar to the students in the f2f classes, the students preferred the didactic approach to teaching and learning, which involved listening to lectures, taking notes, reading the assigned texts, studying the notes and readings for exams, and responding with memorized information to exam questions. tolerance of ambiguity was minimal. the focus was on completing assignments and getting correct answers. similar to johnson, c. and brescia, jr., w. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 64 the f2f students, the explore-and-find-meaning approach emphasized by the instructor was different than what they were accustomed and, therefore, a source of great frustration. for the distance-education students, this frustration was further compounded by obligations to families and employment. they wanted (sometimes demanded) to know the exact, “correct” answers so they could recite them on cue just as they had done in the past with great success. they were not inclined to analyze and grapple with issues. interacting with the material and analyzing concepts for relevance and meaning were not customary practices for most of the students and they were unsure of the steps in and/or the feasibility of the process. when confronted with this situation on campus, being physically present the instructor can note students’ reactions, more readily assuage any discomfort, and facilitate their thinking about and interacting with the material. being 300 miles away and viewing students via a monitor presented a significant challenge to this mode of accommodating students’ needs. establishing a learning community and nurturing risk-taking behavior within the classes via distance was a challenge that was met with mixed results. methods used successfully in the f2f environment were initially employed for the distance-education classes. name cards were constructed, but the faces and the cards were not always clearly visible on the monitor. as an accommodation, students were asked to state their names before speaking. this helped to familiarize the instructor with the names in a time comparable to that in f2f classes. for a short time, students were acknowledged via an attribute or characteristic – “the young man in the red shirt,” “the person sitting beside [student name],” or “the person sitting in the corner.” this method, due to its impersonal nature, was used only during the first class period for group 1. reactions to being called upon to share one’s perspectives were also mixed. although they welcomed the opportunity to relate the material to their experiences, culturally, they did not appreciate or accept the process for communicating their understanding of material to the instructor. african american response style influences how constructivist ideology and practice is perceived and implemented. delpit (1995) discusses the direct response style prevalent in african american communities and its incompatibility with obscure approaches to instruction, such as process approaches to literacy and constructivism. she posits that the ambiguousness of response that is endemic to constructivism is foreign to the direct/straight-forward style of communication and discussion prevalent in african american communities. this premise is mostly applicable to literacy approaches for children of color when trying to incorporate sounds and meanings of words that are incongruous with their backgrounds. the instructor was aware of and had reflected on this contradiction and its significance for her approach to teaching and learning. previous to teaching the distant education class, she had not had an opportunity to really grapple with this enigma given the majority european american student population she usually teaches. however, with the distant education students who embodied these realities, her beliefs on culture and its implications for meeting student needs were under scrutiny in conjunction with her constructivist philosophy. although somewhat in agreement with this premise, she had a different “take” on this seemingly contradiction. she believed that the constructivist approach to teaching and learning does not run counter to the response style prevalent in african american communities under certain conditions. students of color are constantly dealing with ambiguities in language and meaning as a matter of cultural style (delpit, 995). translating this into classrooms and using it to enhance student learning was the challenge. shared ancestral background helped in meeting this johnson, c. and brescia, jr., w. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 65 challenge. common understandings of words, interaction styles, and conventions, allowed the class and the instructor to interact directly as they capitalized on meanings and extended them – reflective of constructivist precepts. multiple responses that were recognized and valued to the seemingly vague questions (more than one “correct” answer) empowered and furthered students’ “voice.” they explored personal meanings and expressed their feelings about the course and its requirements. once the students were more familiar and comfortable with the instructor, they directly expressed their frustration with query and response in the expectation that she would be more forthcoming with the “correct answer.” when the instructor continued to probe and explore their meanings and interpretations of the materials, students addressed the questions and displayed their frustrations in varied ways. as in the f2f classes, many times when a student was asked a question, there was a seemingly long and uncomfortable silence while waiting for a response. using wait time, the question was not passed to another student until the first student indicated he or she did not know the answer or did not wish to answer. via distance, this uncomfortable silence hangs between the two locations and its manifestation is difficult to gauge with limited vision of students and their reactions. frequently, students attempted to address the discomfort by assisting each other in providing an answer to the question. they offered clues, hints, and, on some occasions, answers based on what they perceived the instructor wanted. the combined efforts of students in this situation were crucial in furthering the analysis and understanding of material by the distance classes. via distance, with the instructor limited to the capabilities of a monitor, students restating a question or providing cues/hints for a classmate greatly aided in the shared understanding of the material and its meaning. in addition, the instructor was able to hear how students collectively processed and interpreted the information. similar ancestry helped mediate the understanding process. however, given the differing backgrounds of instructor (midwestern, urban) and students (southern, rural), these clarifications were instrumental to understanding the context and meanings given to concepts by the students. the students’ common experiential knowledge helped them negotiate meaning in a context that was relevant for them (kim & hannafin, 2004). this process evolved in group 1 and was often observed in group 2, but not in group 3. the interdependence of group 1 based on their similar backgrounds and goal to successfully complete the program, intensified their sense of community and served to define a common purpose for the class as they sought to make collective meaning of the material. group 2 was not as solidified as a learning community, but their similar experiences within and outside the course served to mold them into more of a community than group 3. group 3 had little sense of community. it was more fractured on the basis of their experiences and backgrounds. additional effort was required to get them to collaborate on the meanings of the material. these experiences with the three classes underscored the benefits of establishing a learning community for the class if one is not already in place. the community serves the needs of the class in a number of ways. developing community is contingent on hearing and honoring student voice but more often than not, students resist this objective/focus. in group 1 a student requested that she not be asked questions in class. when the instructor explained the need to hear her voice and interpretations of the material, the student countered with she was shy and wanted to hear everyone else and learn from them. a compromise was reached, in which an attempt would be made to forewarn her before calling on her. this compromise had been used in f2f classes on rare occasions for students with similar issues and/or diagnosed special learning needs. over johnson, c. and brescia, jr., w. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 66 time, the interactions enabled the student to become more vocal in class, offering responses based on her interests. she realized that by speaking voluntarily on issues which held her interest, she was able to choose when and how she participated. for other students who expressed similar concerns, a compromise was struck whereby their voices were heard during class without (as they perceived it) being “put on the spot.” group presentations were required of the distance-education classes, but accompanied with additional challenges. the topics for the presentations were selected by the instructor and were broad in scope (vygotsky, multiple intelligences, constructivism, etc.) to allow each group considerable latitude in determining their focus and areas of interest within the category for their presentation. based on their collective interests, each group was to determine its focus; research the areas of interest; and, develop a presentation based on their collective research. maintenance and monitoring of groups to ensure they function effectively proved to be a major challenge given the distance and the available resources. in f2f classrooms, if a group had problems or concerns, students requested assistance after class. sometimes an intervention was made that same day or by the next class period. typically, issues were minor and pertained to differences in working styles or opinions concerning the appropriate manner of presentation. in the distance environment, students’ concerns were more difficult to address. students could not talk to the instructor after class because the audio and video feed connection between the locations was shut down at the ending time for the class. students used email, but in many cases that did not provide the desired level of immediate and personal interaction preferred in these situations. thus, for the distance learning classes, personality and working-style issues that were ameliorated early and easily in the f2f classes became major stumbling blocks to developing an effective presentation. an associated issue stemmed from the complex restraints on time and responsibility for the students with families and jobs. finding time to meet outside of class to decide on the focus and manner of a group presentation is problematic even for relatively unencumbered traditional college students. however, when one works all day and attends class in the evenings – distance education classes were scheduled from 4:00 to 9:00 pm, two or three days per week – meeting this requirement becomes even more complicated. it became obvious after the experiences with group 1 that more support from the instructor via email and phone was needed to ensure they were functioning adequately. subsequent to group 1, one or two persons in each group emerged as leaders and became the major contacts at the off-campus site through which the instructor monitored the process. additionally, individual group members were contacted to discuss any evident frustrations or concerns in order to support and encourage their involvement with the group. these measures helped to make the experience less stressful and more viable for the students. a major issue was access to library resources necessary to adequately investigate a presentation topic. technically the distance-education students were enrolled at the flagship university. however, for library access they were limited to the inadequate collection of reference books and periodicals at the local community college. theoretically, students had access to the ample resources, i.e., educational literature and periodicals, at the flagship university. realistically, students often did not have ready access to these collections due to problems with the local internet service through which they obtained online access to the university library. after several unsuccessful attempts to resolve library-access issues, the group presentations for group 1 were cancelled. instead, students wrote about their experiences johnson, c. and brescia, jr., w. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 67 working with their groups and trying to secure adequate literature for the presentation. for groups 2 and 3, library issues were addressed early via email and consistent updates in class. access to resources did improve in subsequent years, after the university library became involved and worked to ensure access, but the problem was never completely resolved. some students had access to more technologically updated servers and internet providers, while others remained on the wrong side of the digital divide. to address this challenge of equity and access, books were recommended by the instructor on the presentation topics which the group could consider reading and using as an alternative resource to research articles. these books were available at the local libraries, through purchase from the community campus bookstore and/or online sources. with these adaptations, the group presentations remained a requirement for groups 2 and 3. classroom activities requiring group collaboration, such as the one on piaget’s theoretical precepts, proceeded as with f2f classes but with some adaptations. displaying and presenting results for distance education classes were done via the document camera because the drawings were harder to decipher if just held up by students and shown on screen. moreover, additional description and explanation were often required. students explained their depictions with rich detail and defended them when questioned about whether there had been appropriate application of the theoretical precepts. reactions to questions about the non-written depictions and their meanings (how reflective of the precepts and their relationships) varied by class. in groups 1 and 2, students tended to display their drawings and have them questioned with little to no display of defensiveness or hard feelings. by this point in the term, the understood response style allowed students to take risks and directly state their perspectives on the instructor’s interpretations of their ideas. generally, the clarifications and explanations enhanced the shared understanding of the class on concepts being discussed. however, group 3 lacked a strong community ethos and generally were more defensive about any comments or questions that challenged their depictions and interpretations of concepts. eliciting the higher-order thinking required to address the questions raised in class was also a challenge to maintain and foster. when asked a question, students were hesitant to address it because of the possibility of follow-up questions requiring application and analysis. they feared they would not be able to address them adequately, thereby seeming incompetent to their peers. the community (peer) assistance was helpful, but as the class endeavored to explore implications of theoretical precepts for their communities and its students, it was obvious that many were unaccustomed to these types of questions and/or how to address them. according to freire (1970), the students were not accustomed to learning as a basis for praxis or action. in the f2f classes, the instructor walked around the classroom, stood next to students, probed and encouraged responses recognizing that students were not accustomed to investigative/analytical questions. they were most comfortable when repeating a concept or principle from the book than when thinking about its meanings for meeting differing students’ needs. with a sense of community established, students were more likely to move from their comfort zones and venture to answer questions and/or propose solutions to issues. when they shared, the instructor provided encouragement and assisted them in making connections between the theoretical underpinnings of the concepts and their practical experiences with them. distance-learning environments which by definition make physical proximity impossible changed the dynamics of these strategies resulting in them being less personal in format and function for the students. johnson, c. and brescia, jr., w. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 68 these impersonal realities of the distance education environment impact the role of facilitator and prove challenging when handling students’ frustration. similar world views and communication styles helped in finding the right degree of directness for addressing student concerns. this was not as successful with the f2f classes due to the ethnic and cultural differences between students (primarily european american) and the instructor (african american). with f2f classes, a more subtle approach was used to address student frustration. with the distance education students, frustration was discussed candidly. students openly complained to the instructor about the requirements and her “hard” grading. conversely, the instructor directly addressed the class regarding concerns she had about the level of response and their performance on assignments. the common understanding concerning directness of response without ill feelings served the instructor’s purposes with the class also. once, when concerned about the overall quality of their work (poor, inarticulate, and careless), the instructor discussed her frustration with this issue in the class. what came from the discussion was enlightening and reflected research findings for student populations in lower socioeconomic status areas. the students explained that the constructivist approach with its emphasis on collaborative learning, processing of information, and reaching higher levels of thinking, was new and unfamiliar to them. past educational experiences had emphasized the rudiments of learning and focused on lower levels of thinking – rote memorization, comprehension, etc., known to predominate in areas where the poor and lower socioeconomic levels of people preside (oakes, 1985). they were not “slacking” in their work but were finding it extremely difficult to engage in the analytical processes expected. their explanations helped the instructor better understand why their level of disequilibrium was high and somewhat debilitating. the class explored how and why these lower-level cognitive exercises prevailed in the locality as well as the implications of a link between low socioeconomic status and the predominance of these approaches to learning (apple, 2004). one student shared that she had recently observed these practices in her daughter’s classroom and had become more cognizant of them as a result of the class. she pondered the implications of these practices for area students’ future academic achievement and analytical abilities given that these higher-order thinking skills are necessary in higher education and in life. this was an extraordinary educational moment: the class had examined an issue and its implications for them making a difference as educators with their training – praxis (freire, 1970). based on this feedback, the instructor provided more opportunities in class for discussing concerns with the coursework. questions and issues were addressed as directly as possible, while still allowing students to analyze the material based on their own realities rather than those of the instructor. additional time for processing of information was recognized as necessary for the students to better fulfill required assignments. periodically conducting the class f2f with the distance-education students was significant for familiarizing the instructor with the students and the students with the instructor’s approach to teaching. during these visits the instructor walked around the classroom and asked questions just as she does in her classes on campus. students experienced constructivism in close proximity as opposed to having it modeled from the podium via distance education. they appreciated and understood the approach much better after those classes. they became more understanding and comfortable with the instructor and her approach after the visits. she tried to make at least three visits to the area: one to observe the group presentations, another to teach a concept, and the third for the presentation of the portfolio at the end of the semester. johnson, c. and brescia, jr., w. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 69 after the semester ended, the instructor continued her affiliation with some of the students. several emailed her to express gratitude and their thoughts about the value of the course. the most memorable email came from a student in group 1; the sentiments expressed embody her goals for the course and the students. she wrote: i just wanted to let you know how much i have enjoyed taking this class under your instruction. some of my fellow classmates feel that you expected too much from us. however, they fail to realize that through it all they met all deadlines and they fulfilled all requirements. therefore, they should realize something about themselves. i have really and truly enjoyed taking classroom learning theory. dr. johnson, you will never understand how much you have helped me this semester. you have helped me to explore avenues within myself that i never knew existed. you made me push myself to excel far beyond that of which i felt i was capable of. reflecting back over the semester, i realize more so now than ever before, the reason why you pushed us the way you did is because you want us to become the best educators that we possibly can. you want us to do more than just memorize answers, you want us to comprehend and fully understand the concepts and ideas. in my opinion, i honestly feel that you are the type of educator that i hope to become. an educator that cares about her students, both inside the classroom and outside of the classroom. even when i thought that no one noticed how i felt at the time, you noticed. you encouraged me to hold on. i wanted to say thank you. thank you for caring, thank you for pushing me to do the things of which you felt i was capable of doing. thank you for being “true” and thank you for being “real.” this email was received after grades were posted and the semester was done. other students have expressed similar sentiments, but not as eloquently. in subsequent years, some have emailed the instructor about the value of the course when preparing for the licensure exams, praxis ii. it seems that the skills gained from having their ideas, beliefs and perspectives challenged, and being prodded to articulate their rationales proved beneficial for analyzing issues that are a major emphasis of the test. additionally, case study is a part of the class, so it provides some exposure to this type of assessment. c. instructor’s reaction as a constructivist, the instructor’s questioning techniques were enhanced, and her commitment to groups remained steadfast in spite of its challenges. her initial reaction to the idea of distance education has changed. she sees the value and efficacy of distance education for reaching students who would not otherwise be able to fulfill degree-program requirements. the exposure to another reality–i.e., rural, economically depressed area – was enlightening and led to a careful reexamination of her perspectives on difference and its manifestations. she especially appreciated having an opportunity to become privy to and interact directly with the different contexts and realities of student learning. every class began with current events – this is true in f2f as well as distance classes. the instructor’s rationale, which is shared with the class, is that events worldwide have johnson, c. and brescia, jr., w. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 70 implications for classrooms. in both f2f and distance education, local and world news are discussed and evaluated for relevance to the classroom, students, and educational issues. however, for the distance-education students, this activity became a forum for assessing local news and events from their perspectives. local realities and histories brought different dimensions to the material being studied. some of these realities have become part of the instructor’s educational scheme and continue to be incorporated into course activities when teaching precepts f2f. in this instance, distance education has informed the practices on campus. educationally, the instructor has benefited as a facilitator and as a learner. her questioning, active listening, and facilitation skills were enhanced as a result of these experiences. she has learned that her methods for detecting and handling student discomfort and disequilibrium in a f2f setting are not as effective when used in a distance learning environment. however, with attention to students’ needs and realities, a few alterations, some flexibility, and considerable patience, it can be just as productive. a dominant area of intelligence for the instructor is interpersonal intelligence (gardner, 1983). although aware of its influence on her as a learner and an instructor, the distance education experience further validated that the feedback and energy from the class are fundamental to her functioning in the classroom. there is a symbiosis of energy and involvement between instructor and students. the distance caused the instructor to be livelier and more involved in an effort to invoke this symbiotic relationship with the students. this is an area warranting additional attention as it relates to distance education – how do different intelligences and learning styles impact receptiveness to this medium as a learner and as an instructor? although f2f interaction is the instructor’s preferred style, she understands that the attributes she dislikes most about the medium – lack of personal interaction, anonymity, and exclusive use of written responses, are the ones that some students find most suitable for their learning. to better meet student needs and “hear” the multiplicity of voices and tones, more familiarity and ease with this medium is warranted. a major benefit of the instructor’s involvement with the distance-education classes is her increased use and familiarity with technology for educational purposes. she has developed listservs and uses email much more frequently than before becoming involved with distance education. the areas of interpersonal intelligence and technology converge when the instructor emails students. humor is a major tactic used by the instructor to maintain focus and interest in material considered to be “dry”, i.e., theories and their tenets by most students. in f2f classes when students misunderstand the instructor’s humor, her response, or manner of responding; their body language usually signals this and she immediately addresses the confusion. email and listserv posts do not include these clues. the instructor became acutely aware of the importance of “tone” in email. based on students’ reactions to content of her emails and on students’ perceptions concerning the “tone” of her messages, she has discovered that her audience may receive a message other then the one she intended to convey. she has worked on this aspect of her communication with some of the technical advisors at the institution and implemented several of their recommendations to use emoticons as indications of humor (smile, ☺, grin, etc.) in email responses. improvement in this area has enhanced her communication skills for all classes. johnson, c. and brescia, jr., w. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 71 v. conclusion and implications the initial hesitancy, frustration, and bafflement concerning the constructivist approach and its expectations were as common at a distance as f2f. with each succeeding class, the instructor learned more about the importance of the sense of community within the class for the student interaction and involvement which are fundamental to the effectiveness of the pedagogical methods and student assignments. over the years she has learned how to be more effective at mediating interpersonal and systemic challenges, to ensure that students receive equitable assignments and exercises (anderson, 2001; hardwick, 2000). for each class the degree to which a learning community is established needs to be assessed, and in the case of there not being one, it needs to be facilitated. for distance education this challenge is especially cogent because the instructor is less knowledgeable about all the realities and/or experiences of the students who are taking the course. the experience of distance education students with the library sources is characteristic of the logistic problems and the “hidden curriculum” embedded within distance education that has received limited attention in the literature (apple, 2004). to ensure that the experiences of distance education students especially those in rural, less-developed areas are equitable with students on campus, access arrangements must be made for technological linkages (anderson, 2001). calling on students and asking several questions related to or associated with the local realities helps to establish community and build interpersonal relationships. listening and using information from previous responses regarding the area and its realities when referring to a theoretical idea also invoke insightful responses. students are often surprised that an instructor listens, values their responses, and builds on them. active listening in any class is important when sincere, continued responses from students are a goal. however, at a distance, this is even more critical to the success of the collaboration and sharing of ideas. instructors need to receive focused training and be given opportunities to practice using the relevant technologies before they face students who are relying on them for instruction. in addition to practicing with the technology, practicing appropriate distance-learning techniques is necessary for all instructors who want to foster fruitful collaborations and interactions. the establishment of community is necessary for collaboration and for constructivism to be effective in classrooms whether they are f2f or distant (bernard et al. 2000; hardwick, 2000; keiny, 1994; mcalpine, 2000; raymond, 2000). the necessity and benefits of designing instruction for distance education with these philosophical beliefs in mind have been documented (bernard et al. 2000; hardwick, 2000; mcalpine, 2000; raymond, 2000). creating appropriate environments involves more labor and time, and has its own distinctive set of challenges. however, doing so is beneficial for higher-order learning. ensuring that learning experiences develop analytical and problem-solving skills equitable to those nurtured in students in f2f classes is essential for those who do not have direct access to university classrooms. otherwise, distance education becomes another means by which the societal hegemony is continued by providing less for those most in need. given that the focus or purpose of distance education is to extend educational equity and access, assurances must be made that students who receive their education via this medium are sufficiently skilled to compete and achieve. it is expected that in the future there will be an even greater need for higher-order thinking skills, problem-solving skills, and effective interaction skills with a range of individuals. concurrently, the use of technology to meet the growing needs johnson, c. and brescia, jr., w. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 72 of our students is growing exponentially. therefore, further study on how to use and implement collaboration and constructivism more effectively via distance is needed. additionally, better understanding of the impact of culture on the distance education learning processes is integral to developing effective environments that are equitable for a wide range of students with differing realities. references abdal-haqq, i. (1998, december). “constructivism in teacher education: considerations for those who would link practice to theory.” eric digest, eric clearinghouse on teaching and teacher education. american council on education (2000). annual status report: minorities in higher education. washington, dc: author. anderson, t. (2001). “the hidden curriculum in distance education.” change, 33(6) 28 – 35. apple, m. w. (2004). ideology and curriculum, 25th anniversary 3rd edition. new york: routledge. baec, y. k. & westrom, m. (2001). “building a cyber training center for teachers in korea.” journal of technology and teacher education, 9(2), 171-179. bernard, r.m., derubalcava, b.r., & st. pierre, d. (2000). “collaboration online distance learning: issues for future practice and research.” distance education, 21(2), 260 – 277. bonk, c. j. (2002, november/december). “collaborative tools for e-learning.” chief learning officer, pp. 22-24, & 26-27. retrieved october 1, 2004: http://www.clomedia.com/content/templates/clo_feature.asp?articleid=41&zoneid=30 brescia, w. f. “a support taxonomy for improving online discussions.” journal of public affairs education, 9(4) 289-298, 2003. brooks, j.g. & brooks, m.g. (1999). in search of understanding: the case for constructivist classrooms (2nd ed). alexandria, va: association for supervision and curriculum development. brown, j. s., collins, a., & duguid, p. (1989). “situated cognition and the culture of learning,” educational researcher, 18(1) 32-42. crumpacker, n. (2001). “faculty pedagogical approach skill and motivation in today’s education milieu.” online journal of distance learning administration, 4(4) retrieved online october 19, 2004: http://www.westga.edu/~distance/ojdla/winter44/crumpacker44.html. delpit, l. (1995). other people’s children: cultural conflict in the classroom. new york: the new press. duffy, t. & kirkley, j. (2004). “learning theory and pedagogy applied in distance learning: the case of cardean university.” in t. duffy & j.r. kirkley (eds.), learner-centered theory and johnson, c. and brescia, jr., w. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 73 practice in distance education: cases from higher education (pp. 107-141). mahwah, nj: lawrence erlbaum associates, inc. freire, p. (1970). the pedagogy of the oppressed. new york: continuum publishing gardner, h. (1983) frames of mind: the theory of multiple intelligences. new york: basic. hardwick, s.w. (2000). “humanizing the technology landscape through a collaborative pedagogy.” journal of geography in higher education, 24(1) 123 – 129. johnson, d.w., johnson, r., holubec, e.j., & roy, p. (1984). circles of learning: cooperation in the classroom. alexandria, va: association for supervision and curriculum development. jonassen, d. h. (1991). “objectivism versus constructivism: do we need a new philosophical paradigm?” educational technology research and development, 39(3), 5-14. keiny, s. (1994). “constructivism and teacher’s professional development.” teaching and teacher education, 10(2), 157 – 167 kim, m. & hannafin, m (2004). “designing online learning environments to support scientific inquiry.” the quarterly review of distance education, 5(1), 1-10. knowles, m.s. (1990). the adult learner: a neglected species (4th ed.). houston, gulf publishing. mangan, p. (2001). “what is distance learning?” management quarterly, 42(3), 30 – 35. mcalpine, i. (2000). “collaborative learning online.” distance education, 21(1), 66 – 80. muirhead, b. (2001). “practical strategies for teaching computer-mediated classes.” ed at a distance. 15(5). retrieved october 14, 2004 from http://www.usdla.org/html/journal/may01_issue/article02.html oakes, j. (1985). keeping track: how schools structure equality. new haven, ct: yale university press. raymond, f.b. iii (2000). “delivering distance education through technology: a pioneer’s experience.” campus-wide information systems, 17(2), 49 – 55. van looy, b., callaert, j., debackere, k. & verbeek, arnold. (2004). “patent related indicators for assessing knowledge-generating institutions: towards a contextualized approach.” journal of technology transfer. 28(1) 53-61. retrieved october 14, 2004 from http://journals.kluweronline.com/issn/0892-9912/contents wadsworth, b.j. (1996). piaget’s theory of cognitive and affective development: foundations of constructivism (5th ed.). white plains, ny: longman publishers. watts, m. & pope, m. (1989). “thinking about thinking, learning about learning: constructivism in physics education.” physics education, 24, 326–331. johnson, c. and brescia, jr., w. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 74 wertsch, j.s. (ed.) (1985). culture, communication and cognition: vygotskian perspectives. new york: cambridge press. woolfolk, a. (2004). educational psychology (9th ed.). needham heights, ma: allyn and bacon. journal of scholarship of teaching and learning, vol. 6, no. 1, august, 2006, pp. 75 – 87. rapid prototyping as method for developing instructional strategies for supporting computer-mediated communication among university students dave s. knowlton1 abstract: because rapid prototyping results in the quick development of curriculum, materials, and processes, it is a form of design that could be particularly useful to professors in higher education. yet, literature documenting the use of rapid prototyping in higher education is scarce. this paper offers a case example of rapid prototyping being used as a design process. after presenting the case, the author points to necessary considerations for other faculty members who are considering using rapid prototyping. these considerations include the need to gain perspective on the roles of instructional strategies and computers within teaching and learning processes, understand the distinction between traditional research rigor and design rigor, and the importance of approaching design systematically. i. introduction and purpose designing meaningful learning experiences is difficult for professors. true instructional design often is too expensive of a process to be viable in higher education; and while carefullyconstructed constructivist learning environments are becoming more widely used across the academy, such environments, too, require a high level of detailed planning, particularly when computers are involved. college professors simply cannot enter each semester with a solid constructivist design of all assignments and course activities. sometimes, then, the best a professor can do is to design “something” as a part of a new course preparation and tweak it over time. in general, this process of designing and tweaking is referred to as rapid prototyping (reiser, 2001). while a professor’s efforts to use rapid prototyping can result in the quick development of instructional materials or activities (resier, 2001), the quality of resulting materials and activities often is suspect. why? models of rapid prototyping are surprisingly complex and are largely based on “progressive refinement”—“putting a first version of a design into the world” and then revising that design “until all the bugs are worked out” (collins, joseph, & bielaczyc, 2004, p. 18). the process is not complete, then, when materials have been developed. revising implies a detailed and systematic process; it is the iterative nature of designing that makes rapid prototyping a successful design approach (jones & richey, 2000). in fact, rapid prototyping often involves an entire support team to manage the design process (cf., lohr, javeri, mahoney, gall, li, & strongin, 2003), but most professors do not have access to such a level of human capital. the purpose of this paper is to offer a case example of the rapid prototyping process that i used to develop assignment guidelines for supporting students’ use of an online discussion board. notably, this paper emphasizes the systematic development of the assignment guidelines 1 assistant professor of instructional design & learning technologies at southern illinois university edwardsville, www.siue.edu/~dknowlt. knowlton, d.s. 76 across three semesters of implementation. perhaps this type of case example can be illustrative for other professors who need to systematically prototype assignments, sans formal training in rapid prototyping and support team. this paper begins with a description of the context in which the assignment was prototyped and then temporally describes the prototyping process. in the last section of this paper, i offer generalized principles for using rapid prototyping to develop assignments for the higher education classroom. ii. context in which rapid prototyping occurred part of my teaching responsibilities within a school of education at a midwestern university included serving as a member of a faculty team that supported the efforts of preservice teachers (undergraduate students majoring in elementary or secondary education) in a two-year, field-based teacher-certification program. the preservice teachers who were enrolled in this program were assigned to k-12 classrooms in partnership schools. because this was only the second implementation of the entirely field-based certification program, much of the context supporting the program was still developing. during the first semester of the two-year program, the preservice teachers often assumed a periphery role within the classroom—serving more as a teacher’s aide than as a practicing teacher. during the last semester of the two-year program, though, the student teachers participated in a formal “student-teaching” experience. the fieldbased program was designed to support the preservice teachers’ development from aide to professional teacher. throughout the two years, a team of university faculty supervised weekly content seminars. within the seminars, faculty members sometimes resorted to lecture as a means of orienting the preservice teachers to various educational theories and methods; more often, though, within these seminars, preservice teachers were given opportunities to discuss their experiences in the classroom. during each of the first three semesters of their field experience, the preservice teachers were enrolled for one credit hour of educational psychology—the content that i was responsible for overseeing. in principle, though, “courses” were non-existent. instead, each courses’ content was integrated into seminar activities and discussions. while the faculty team and preservice teachers came together for the weekly seminars, communication throughout the rest of the week was difficult. most of the professors on the faculty team had other responsibilities that prevented them from spending substantive time within the partnership schools, and the preservice teachers were placed in a variety of schools across three school districts. therefore, the team of faculty determined that since webct’s (the university’s approved online course management tool) discussion board depended on neither face-to-face communication nor real-time interactions, it would be a useful and efficient tool to help the preservice teachers stay connected with each other and with the faculty team. the rapid prototyping process that is the basis of this paper involves the design of strategies to support the effective use of bulletin board discussions. table 1 provides (a) an overview of the factors that influenced the development of each version of the discussion assignment, (b) the characteristics of each version, and (c) a summary of evaluation findings for each version. iii. the first-semester use of the electronic bulletin board the first-semester guidelines supporting the preservice teachers’ use of the bulletin board proved ineffectual as a tool for promoting communication, much less learning. within this knowlton, d.s. 77 table 1: factors contributing to assignment design, assignment characteristics, and evaluation. first semester version second semester version third semester version factors influencing design factors influencing redesign factors influencing redesign • need for flexible and efficient communication tool • emerging nature of the field experience • lack of information about the preservice teachers’ knowledge and skills • need to introduce basic educational psychology principles • first version was ineffectual • shifting responsibilities of preservice teachers • changes to the use of weekly seminar time • need for preservice teachers to gain skill in using web-based communication tools • evaluation of revised version • elimination of seminar time for educational psychology • continued shifting responsibilities of the preservice teachers initial design characteristics characteristics of redesign characteristics of redesign • laissez-faire • preservice teachers were simply made aware that discussion board existed. • preservice teachers assigned to two groups • discussion based on three-week cycles • discussion centered on student-initiated problems and proposals for practical solutions • addition of a privacy statement and job aid emphasizing conventions of cmc • additional direction to focus on “instructional problems”; more scaffolding to support “good” contributions • added reflection writing and self report form evaluation of first evaluation of second evaluation of third • ineffectual and rare use • preservice teachers reported that they didn’t see practical value of using cmc • problems were narrow in scope • interaction among the preservice teachers was limited • grading was cumbersome • perservice teachers noted workload was heavy and contrived • scaffolding of third week contributions did broaden the types of input from the preservice teachers knowlton, d.s. 78 section, the factors contributing to the first-semester guidelines and a description and evaluation of those guidelines are discussed. a. factors contributing to the first-semester guidelines commonly, careful analysis of both the educational context and learners precedes rapid prototyping (jones & richey, 2000). because the context of the partnership school was still emerging, analysis was based largely on generalization. from a macro-perspective, it seemed that the unique context of the field experience would continue to emerge as implementation progressed. this symbiosis between context and implementation required that i give the preservice teachers plenty of latitude in their use of the bulletin board, which might include their decision not to use it at all. furthermore, as the initial guidelines needed to be in place the day that i met the preservice teachers, i had no knowledge of the skill of the learner for which i was designing. had they used a bulletin board before? did they even know how to find the university’s webct site and log on? i did know that these preservice teachers had never before taken educational psychology. some content, then, needed to be transferred to these preservice teachers. in a pedagogical age of open-ended learning environments and within the context of a field experience, i recognize the vulgarity of suggesting the need for knowledge transfer. nevertheless, because of both the school of education’s accreditation process and certification tests that the preservice teachers would need to pass, the preservice teachers needed to obtain a basic understanding of educational psychology concepts and principles. this requirement further accentuated the need to de-emphasize the use of the bulletin board (which often is more wellsuited for promoting open exploration than for supporting direct concept attainment) and emphasize activities and assignments that were more likely to promote direct knowledge transfer. b. characteristics of the first-semester guidelines at the start of the first semester of the field-based program, the faculty team simply made the preservice teachers aware that webct had a discussion board where they could share ideas with each other and ask questions in a forum that would expedite communication. once the preservice teachers were aware of webct, i informally suggested that they might use the bulletin board to collaboratively make sense of assigned readings and prepare for seminar activities. admittedly, this laissez-faire approach contradicts much of the practical advice for using bulletin board discussions. some literature suggests that if professors do not scaffold the “hows,” “whens,” and “whys” of using asynchronous discussion then students will not use it effectively, or even at all (e.g., knowlton, knowlton, & davis, 2000). c. evaluation of the discussion board’s first-semester use predictably, the bulletin board was used rarely. when it was used, the contributions were most often in the form of close-ended questions: “what chapters are we supposed to have read by next week’s seminar?” several preservice teachers noted that it was nice to know the bulletin board was available, but they did not have a need to use it often. that is, they did not see how sharing ideas on the bulletin board would help them prepare for their day-to-day activities in the k-12 classrooms. after all, their argument went, they daily had access to their mentor teachers— knowlton, d.s. 79 the full-time teachers in the classroom to which each preservice teacher was assigned—who could guide them in their decision-making processes. iv. the second-semester use of the electronic bulletin board the laissez-faire approach to support learning through the bulletin board was not effective. to aim for more educational effectiveness, i shifted the emphasis toward a computermediated communication (cmc) assignment by designing instructional strategies that would more likely secure the preservice teachers’ participation. the formalized design showed some promise, but evaluation suggested the need for refinements to the assignment’s design. a. factors contributing to the second-semester guidelines both the preservice teachers’ “readiness” for a higher level of professional thinking and their shifting responsibilities in the classroom necessitated formalized guidelines to support the use of the computer-mediated discussion. during the first semester of the partnership program, i had assigned readings from the adopted educational psychology book (see eggen & kauchak, 1997). these readings served the purpose of introducing the preservice teachers to the large issues that fall within the domain of educational psychology. once the preservice teachers had been exposed to key educational psychology concepts, they needed experience applying those concepts by making connections between textbook theory and real-world classrooms. such connections can be useful in supporting students’ problem-solving efforts in field experiences (beckett & grant, 2003). this shift from “knowing” to “applying” seemed further appropriate because it paralleled the preservice teachers’ shift within the partnership school. the preservice teachers slowly were moving from serving as paraprofessionals—by taking class attendance and organizing materials, for example—to participating as true professionals—by designing lesson plans and teaching the entire class. a second contextual factor also created the need for more exact guidelines to support the use of the discussion board. the team of faculty members who supervised the weekly seminars decided that more organization was needed within the seminars. no longer would the faculty team collectively guide discussion and facilitate activities; rather seminar time was divided among content areas—“today is an educational psychology seminar; next week will be a reading methods seminar.” such a shift was problematic because it violated one of the very foundations of a field-based program—that content should be integrated and directly based on the preservice teachers’ field experiences (cf., beckett & grant, 2003; scanlon & ford, 1998; weber, 1996). successful professionals must learn to think holistically about their experiences, not about “courses” from a program of studies. designing and implementing more exact strategies to support cmc served as a means for prompting the preservice teachers to continue making integrated connections, even though seminar time was less integrated. b. characteristics of the second-semester guidelines participants were divided into two groups and the electronic discussion was based on three-week cycles of sharing and response. assessment of students’ efforts was integrated across the cycle. at the end of each cycle, roles were reversed so that preservice teachers in group one performed the responsibilities of the preservice teachers in group two and vice versa. this general approach has been supported elsewhere in the literature (cf., knowlton, 2002). knowlton, d.s. 80 week one of the discussion cycle. preservice teachers assigned to group one were responsible for describing a problem that they were experiencing within their partnership school. the assignment guidelines noted that the “problem might be interpersonal (e.g., a conflict with a mentor teacher or parent); instructional (e.g., students not meeting the objectives of a lesson); behavioral (e.g., a student who constantly ignores classroom rules); or contextual (e.g., a lack of adequate textbooks or other supplies).” this emphasis on a variety of problem types was purposeful. as i have noted, one goal of the assignment was to help the preservice teachers see the ways educational psychology was integrated with other “content,” such as cultural foundations of education, instructional methods, and classroom management. if the scope of the problems that students shared was broad, then opportunities for connections to content beyond educational psychology equally would be broad. week two of the discussion cycle. preservice teachers in group two were responsible for using the index and table of contents of the educational psychology textbook to theoretically frame the problems that had been shared during week one. the textbook, then, became a learning-on-demand resource, where preservice teachers were self-selecting readings that would most likely contribute to an analysis of the problem-at-hand. in addition to making connections between the problem and educational psychology, preservice teachers were encouraged to make connections to content areas that were the basis of their weekly seminars. these connections were designed to help the preservice teachers understand that neither the problems that they encountered nor educational psychology were discrete. instead relationships existed among problems encountered in classrooms, educational psychology, and other content areas. week three of the discussion cycle. all of the preservice teachers—regardless of whether they were assigned to group one or group two—were responsible for three contributions to the computer-mediated discussion. the assignment guidelines dictated that not all three contributions should be posted on the same day of the week. the purpose of this criterion was to build in reflection time for the preservice teachers. they were to consider the discussion in its entirety before adding further to the discussion. the assignment guidelines directed the preservice teachers’ efforts with week three contributions by noting that they should “further define and work to solve the problems-at-hand through dialogue.” they should “read what [their colleagues had written] within a ‘thread’ of discussion and interact by responding to [each other’s] ideas.” assessment across the discussion cycle. the assignment guidelines noted that the preservice teachers would “receive most credit for the number of contributions that [they offered].” the rest of the credit would be earned by meeting the stated purpose of each week’s contribution. for example, a stated purpose of week two and week three contributions was that the preservice teachers should theoretically frame the problem with citations. the assignment guidelines did offer a caveat to this loose assessment structure, however: “after the first two cycles, if we all feel that we are doing more than ‘going through the motions,’ then the assessment criteria can stay equally ‘loose.’ that is, we all should be working as professionals to help classmates solve real problems. if, however, i sense—or we agree as a class—that the criteria are not rigorous enough to foster collaborative problem solving, i will [offer] additional criteria (for example, specifying the length of contributions) to enhance the educational potential of this assignment.” knowlton, d.s. 81 c. evaluation of the second-semester design two data sources served as a basis for evaluation. first, my assessment of the preservice teachers’ efforts served as a basis for determining additional changes that were needed to improve the efficacy of the assignment. second, during a weekly seminar, i engaged the preservice teachers in a discussion about the use of cmc. assessment as evaluation data. as i assessed the preservice teachers’ participation in the computer-mediated discussion, i made judgments about the design of the assignment itself. this approach of combining assessment with evaluation to determine the educational viability of cmc is not unprecedented. in fact, “only the integration of assessment [with] evaluation can produce a clear picture of an online discussion’s educational viability” (knowlton, 2001, p. 164). numerous observations suggested the need for additional change. first, i found that students were relatively successful at articulating problems that they were experiencing, but the problems were extremely narrow in scope. of the approximately ninety posted problems, most focused on discipline problems among the k-12 students. one or two of the posted problems focused on interpersonal conflicts, such as conflicts with their mentor teachers or a parent. two of the posted problems focused on instructional concerns. second, most contributions during week three of the discussion were replies to the original problem posted during week one. in other words, the preservice teachers were not discussing the problems by interacting; they merely continued to offer solutions to the original problem. in fact, it often was unclear as to whether or not the preservice teachers were reading the threads of discussion in which they were responding. while repetition of various ideas across contributions within the same thread of discussion was common, consensus building and synthesis of ideas were scarce. third, as i assessed the preservice teachers’ participation, i recognized the inordinate amount of time i was spending on grading, as opposed to engaging in activities that were related to assessment but more productive toward creating continued learning among students—such as reacting to their discussion contributions, highlighting common themes among their interactions, and offering contributions to the discussion as an authentic participant. certainly, it was within my purview to grade the preservice teachers’ efforts, but grading should not dominate the assessment process (bauer & anderson, 2000). input from preservice teachers. at the end of the second semester, i solicited input from the preservice teachers about the use of cmc. i used two questions as prompts to promote feedback from the preservice teachers. notably, these prompts emerged as i assessed the preservice teachers’ success during the second semester: • how could the assignment guidelines be adjusted to emphasize the value of collaborative analysis and inquiry in an attempt to solve real-world problems? • how might the assignment guidelines be structured to foster an environment where the preservice teachers “forget” that cmc is being used as an “assignment” that will be graded and instead remember the need to act professionally and help their colleagues, even if that means offering more input than the minimum expectations require? during this discussion, two points emerged. first, the preservice teachers noted that many of them still usually were engaged in activities that did not directly relate to teaching. certainly, they felt that by the third semester of their partnership experiences they would have shifted even more into a key role as the “teacher” of the class. this shift, they argued, would make it easier knowlton, d.s. 82 for them to participate in the discussion because they would have richer experiences on which to base their contributions to the discussion. second, the preservice teachers noted that criteria governing week two and week three contributions were hindering their participation. they urged me, for example, to reduce the number of required contributions in both weeks two and three. this would give them more time to research and find appropriate resources to support the perspectives that they offered within the discussion. one preservice teacher noted that there were so many contributions to each thread that there was nothing left to add for late-comers to the discussion; reducing the number of required contributions would solve this problem. another preservice teacher agreed and noted that she did not read the threads before contributing because she did not want to know whether she was duplicating ideas that had already been offered. for similar reasons, students suggested the need to eliminate any criterion that specified on what days of the week they should participate. once they planned their contribution they returned to the discussion board only to find that someone else had offered their idea. also, several preservice teachers noted that they were printing out discussion contributions and sometimes even entire threads of discussion and reading them. so, while their actual contributions might come on a single day of the week, they were spending time considering the discussion across numerous days of the week. v. the third semester cmc assignment the third-semester version of the assignment included several changes from the previous semester. notably, these changes were based on feedback from the preservice teachers, which was reported in the previous section of this paper. in this section, i describe the milieu that contributed to the development of the third-semester assignment guidelines, the changes that were implemented, and evaluation. a. factors contributing to the third-semester design the feedback that i had solicited from the preservice teachers did contribute to the prototyping of the third-semester design, but other factors contributed, as well. notably, the format of the weekly seminars once again changed during the third semester of the partnership. it was determined that certain content areas—educational psychology being one such area—would not be given any formal emphasis during seminars. yet, i was still accountable for assessing the preservice teachers and giving an educational psychology grade to each of them at semester’s end. because of this dilemma between, on the one hand, needing to assess the preservice teachers and, on the other hand, not having formal seminar time to assess them, continuing to formally use cmc seemed appropriate. b. characteristics of the third-semester cmc guidelines the assignment was still based on the idea of the preservice teachers sharing real problems that they were experiencing and cycles of theoretically framing and solving those problems. three changes to the assignment guidelines were made in an effort to overcome some of the weaknesses evident in the second-semester version. the first was an administrative change. the second was a change in the types of problems that the preservice teachers should knowlton, d.s. 83 share. the third change concerned guidelines governing week two and week three discussion contributions. administrative change. during the second-semester version of the assignment, i had spent large amounts of time grading the preservice teachers’ contributions, as opposed to assessing and providing participants with the types of authentic feedback that could improve their problem-solving and analysis skills. to shift my own role within the discussion from a grader—which is often viewed as punitive—to a facilitator—which offers the opportunity to be more constructive—i developed a self-report form. at the end of each cycle of discussion, the preservice teachers completed and submitted this self-report, which allowed them to report factual information about their participation. for example, they could list the subject line of the threads in which they participated and cite the various resources that they used in theoretically framing a problem to which they responded. when the preservice teachers submitted their report, i had a list of threads in which i could find their contributions. this made the process of “grading” less time consuming. changes to the types of problems offered for discussion. the third-semester version of the assignment guidelines required that all problems contributed to week one of the discussion cycle must be “instructional problems”—as opposed to the type of behavior and discipline problems that dominated the second semester. specifying that the problems should be “instructional” in nature was designed to broaden the preservice teachers’ thinking regarding what constitutes a classroom problem that was worthy of analysis. changes to guidelines governing week two and week three contributions. i reduced the number of required contributions during week two of each cycle from three to two. the preservice teachers had advocated for the need to lower the number of required discussion contributions. they argued that lowering the quantity of required contributions would allow them to be more thorough in their analysis of the problems contributed to the discussion. while skeptical of such claims, i implemented this change in the hope that my skepticism would be unfounded. also, i specified that week three contributions had to be replies to week two contributions, not replies to the original problem discussed during week one of each cycle. i hoped that this criterion would improve interaction among the preservice teachers within the computer-mediated discussion and promote a deeper analysis of the issues embedded within the problems, not just continued (and often redundant) “solutions” to the original problem. because of this more specific purpose of week three contributions, i developed a list of possible strategies that the preservice teachers might use as they contributed to the discussion during week three. (see table 2.) c. evaluation of the third-semester prototype evaluation consisted mainly of the preservice teachers completing an open-ended survey about their views of using cmc. space limitations prevent a full explication of the survey results. i focus here, though, on feedback that directly related to changes made in prototyping the final version of the cmc assignment. changes governing the types of problems shared during week one. several preservice teachers noted that describing an instructional problem was more difficult than describing problems with student behavior or interpersonal conflicts. many of them acknowledged, though, that being asked to describe instructional problems forced them to look at their own curriculum development and instructional practices in a more detailed way. as one preservice teacher noted, knowlton, d.s. 84 table 2: strategies for replying to week three contributions as you write contributions to the discussing during week #3, you should work to add a deeper level of analysis to the discussion, not to simply repeat what has already been offered. if you are stuck for ways to contribute during week #3, consider the following possible strategies. • pick two replies to the same problem and discuss why you think one would work better than the other. • pick a reply to a problem and discuss the strengths and weaknesses of the proposed solution • pick a theory that someone mentioned as a help to understanding week #2 and apply that theory differently (or more thoroughly). • discuss your experiences with how a solution has/has not worked in the classroom. • write a summary of responses to your own problem and describe what the biggest things that you are taking away from your problem are. “i was surprised that it was more difficult to relate an instructional problem to ed[ucational] psych[ology]. the behavior problems stood out more and the connections were more obvious. because we had to share instructional problems, i think that i learned how interwoven ed[ucational] psych[ology] and teaching truly are.” changes to week two and three contributions. the preservice teachers on average tended to agree that the changes to week two and three contributions were positive. one preservice teacher noted that the changes to the guidelines allowed her to “actually relate different theories and sources of information to the problems [that] others [were] experiencing.” her point was that by being required to offer fewer contributions she could consider those contributions more carefully. other students seemed to indicate that the suggestions for week three responses were useful. many students noted that from these possible responses they realized that they could share their own experiences to a problem. one preservice teacher phrased it this way: “the most helpful responses were not the ones that said, ‘on page 276 of the text book, it states.…’ instead, responses that described what [others] were doing in their classrooms to help with similar problems were . . . much more helpful.” from this and several similar comments, i infer that the suggestions for week three responses (as shown in table 2) were useful to students in guiding them toward offering more salient contributions to the cmc discussion. vi. implications of this prototyping approach in this paper, i have offered a case example of rapid prototyping as a design approach for developing a cmc assignment for the higher education classroom. the details of such an example should provide faculty members with new perspectives about the iterative nature of development processes. specifically, several implications of this case cut across many higher education scenarios and are worthy of comment. knowlton, d.s. 85 a. instructional strategies influence learning noticeably absent from the case example is a discussion of the importance of customizing webct to improve the educational utility of the discussion. instead of focusing on the prototyping of improved media, this case focuses on the development of instructional strategies. such a focus is fully appropriate, as it is consistent with a view supported in the literature. namely, instructional strategies, not computers, are the cause of learning (e.g., clark, 1983, 1994a, 1994b). admittedly, such a perspective is not without detractors (e.g., kozma, 1991), but even these detractors agree that there is no credible evidence to suggest that computers influence learning. professors who are using rapid prototyping to design media-based assignments would do well to consider their own philosophy among media, instructional strategies, and positive learning outcomes. if, in fact, computers do not influence learning, then prototyping should focus on strategy development more so than on media development. b. consideration of design rigor for readers of this article who come from a traditional empirical background, this case example may have proved a frustrating read. no method of data collection and analysis was offered and applied, and no discussion of “significant” results was provided. but edelson (2002) distinguishes between traditional research and design research. with this distinction comes a distinction in approach. for the professor who is interested in achieving a level of understanding to justify change within a course or assignment, empirical rigor is not needed, and may even be misleading. instead, pedagogical rigor can provide insights sufficient to adjust assignments so that they promote a stronger opportunity for learning among students. the point is that faculty members across disciplines should gravitate toward design processes that allow for functional revision of assignments, even if such gravitation limits one’s ability to publish more scientific claims that are indicative of traditional research. c. nature of systematic design this third implication builds largely from the second. to suggest that empirical rigor indicative of the positivist research paradigm is unnecessary is not to suggest that design is haphazard and non-systematic. the case example noted here serves as a worthy model for professors across disciplines because it illustrates the relationship between the prototyping process and a dependence on inputs and outputs, which is one characteristic of “systematic” design. consideration was given, for example, to the macrocontext of the field-based program. consideration was also given to the changing needs of the preservice teachers. as professors adopt rapid prototyping procedures, they, too, should consider the role of context as a factor that influences and informs their design. as a learning context evolves, design practices must become increasingly iterative and flexible. furthermore, the evaluation of design is one unique stage of the design process that is particularly dependent on inputs and outputs. professors who are prototyping assignments across semesters or even within a single semester should plan for evaluating the quality of their own designs. as can be noted within the case described in this paper, the professor’s judgment was involved in evaluating the assignment, but the prototyping of the assignments did not stand on the professor’s judgment alone. student input was a part of the evaluation process and the assignment was prototyped—at least to some extent—based on that student input. within the knowlton, d.s. 86 case reported in this paper, perhaps i had an obvious advantage in that my students (i.e., the preservice teachers) remained the same across each semester of implementation. this allowed me to develop a rapport with them, and they perhaps felt more invested in assisting with the prototyping of the strategies, since they knew that they would be engaged in cmc discussion in future semesters. professors who do not have such an advantage might have to go to greater lengths to account for student input as they are prototyping assignments. the use of additional formal surveys, focus groups, or other opportunities for student input may be useful to this end. references bauer, john f., and anderson, rebecca s. (2000). evaluating students’ written performance in the online classroom. in renee e. weiss, dave s. knowlton, and bruce w. speck (eds.), principles of effective teaching in the online classroom (pp.65-72). san francisco: jossey-bass. beckett julia, and grant, nancy k. (2003). guiding students toward solutions in field experiences. in dave s. knowlton & david c. sharp (eds.), problem-based learning for the information age, (pp. 67-72). san francisco: jossey-bass. clark, richard e. (1983). “reconsidering research on learning from media.” review of educational research, vol. 53, no. 4, pp. 445-459. clark, richard e. (1994a). “media will never influence learning.” educational technology research and development, vol. 42, no. 2, pp. 21-29. clark, richard e. (1994b). “media and method.” educational technology research and development, vol. 42, no. 3, pp. 7-10. collins, allan, joseph, diana, and bielaczyc, katerine (2004). “design research: theoretical and methodological issues.” the journal of the learning sciences, vol. 13, no. 1, pp. 15-42. edelson, daniel c. (2002). “design research: what we learn when we engage in design.” the journal of the learning sciences, vol. 11, no. 1, pp. 105-121. eggen, paul, and kauchak, donald (1997). educational psychology, 5th ed. columbus, oh: prentice hall. jones, toni s. and richey, rita c. (2000). “rapid prototyping methodology in action: a developmental study.” educational technology research and development, vol. 48, no. 2, pp. 63-80. knowlton, dave s. (2001). “determining educational viability in online discussions: a student-centered approach.” academic exchange quarterly, vol. 5, no. 4, pp. 162-168. knowlton, dave s. (2002). “promoting liberal arts thinking through online discussion: a practical application and its theoretical basis.” educational technology & society, vol. 5, no. 3, pp. 189-194. knowlton, d.s. 87 knowlton, dave s., knowlton, heather m., and davis, camela. (2000). “the whys and hows of online discussions.” syllabus: new directions in educational technology, vol. 13, no. 10, pp. 54-58. kozma, robert b. (1991). “learning with media.” review of educational research, vol. 61, no. 2, pp. 179-211. lohr, linda, javeri, manisha, mahoney, chris, gall, jim, li, kathy, and strongin, dawn (2003). “using rapid application development to improve usability of a preservice teacher technology course.” educational technology research and development, vol. 51, no. 2, pp. 41-55. reiser, robert a. (2001). “a history of instructional design and technology: part 2: a history of instructional design.” educational technology research and development, vol. 49, no. 2, pp. 5767. scanlon, patricia a., and ford, michael p. (1998). grading student performance in real-world settings. in rebecca s. anderson and bruce w. speck (eds.). changing the way we grade student performance: classroom assessment and the new learning paradigm, (pp. 97-105). san francisco: jossey-bass. weber, alan (1996). “professional development schools and university laboratory schools: is there a difference?” the professional educator, vol. 18, no. 2, pp. 59-65. journal of scholarship of teaching and learning, vol. 6, no. 1, august, 2006, pp. 88 – 109. developing the scholarship of teaching through transformative learning carolin kreber1 abstract: following a cognitive-developmental perspective, the scholarship of teaching & learning is understood as a process of knowledge construction whereby knowledge claims are validated through reflection on teaching experience and educational theory. these reflective processes can be documented and peer reviewed. teaching portfolios allow for the documentation of indicators of reflection. indicators can be developed for each of three domains of teaching knowledge: (1) what we consider to be meaningful goals and purposes of higher education; (2) what we know about student learning and development in relation to these goals; and (3) what we know about the teaching and instructional design processes needed to bring about academic learning and development. keywords: learning about teaching, transformative learning, reflection, professionalism and citizenship, scholarship, documentation and peer review i. introduction: the scholarship of teaching as professionalism guided by citizenship “we develop a scholarship of teaching when our work as teachers becomes public, peerreviewed and critiqued. and exchanged with members of our professional communities so they, in turn, can build on our work” (shulman, 2000). far from having remained “an amorphous term, equated more with commitment to teaching than with any concrete, substantive sense of definition or consensus as to how this scholarship can be recognized” (menges & weimer, 1996, p.xii), the scholarship of teaching and learning has gained much clearer contours and recognition in recent years (kreber, 2003). often linked to the notion of professionalism in university teaching, the scholarship of teaching and learning is progressively associated with a form of knowledge about teaching and student learning that can be rationally verified through disciplined inquiry. “professional knowledge” thus construed is knowledge oriented towards “best practices”. while the question of what constitutes “best practices”, ultimately, is a philosophical one, there remains little doubt that we can observe a trend in the educational policy arena to equate the idea of “best practices” increasingly with notions of effectiveness and efficiency. applied to the scholarship of teaching and learning, professional practices (or “best practices”), then, are identified by exploring the question “which teaching innovations produce the best results (i.e., more learning, better/deeper learning, or a closer fit of learning outcomes with those required by the job market, etc)?”. no one would dispute that this is a significant question to delve into; however, it is just one question that the scholarship of teaching is (or should be) concerned with. a second way of exploring university teaching in a scholarly (or if you will professional) way is to turn to its moral and civic purposes. thus construed, the scholarship of teaching (and 1 director of the centre for teaching, learning and assessment, moray house school of education, paterson’s land, university of edinburgh, uk. questions and comments may be emailed to carolin.kreber@ed.ac.uk. kreber, c. journal of scholarship of teaching and learning, vol. 6, no. 1, august, 2006. 89 learning), or professionalism in teaching, is more appropriately associated with the notion of citizenship rather than with “effectiveness or efficiency”. while understanding how students learn and finding ways to optimize their learning, preferably through replicable and publishable forms of scientific inquiry, is clearly important, this alone cannot be the essence of the scholarship of , or professionalism in, teaching. “best (teaching) practices”, therefore, are no more than the means by which to bring about desired educational outcomes. and so a question to be addressed early on in any deliberations on university teaching ought to be “what are the ends that the contemporary university serves through its teaching practices and curriculum?” and, more importantly, “are these the same that it should serve”? are we concerned with training and socializing researchers into our discipline, preparing students for specific jobs or for lifelong learning, facilitating their personal development, promoting their successful participation in a democratic society, or perhaps none, or all, of these? analyses of these latter questions are just as essential aspects of practicing the scholarship of teaching as are explorations of how well certain teaching methods work and how, or how well, students learn (hutchings & shulman, 1999). the scholarship of teaching and learning, or professionalism in teaching, therefore, needs to be conceptualized broadly and integrate the notion of professionalism with the notion of citizenship (see also walker, 2001). by asking (1) what do we consider to be meaningful goals and purposes of higher education, (2) what do we know about student learning and development in relation to these goals, and (3) how can we promote such learning and development (kreber & cranton, 2000; kreber, 1999), the scholarship of teaching and learning could lead to changes that go beyond the development and implementation of instructional innovations but are expressed also in the larger curriculum and co-curriculum (kreber, 2005a) i am stating what is obvious to everyone, of course. certainly goals are important and no one would dispute this. in recent years, many scholars have highlighted the university’s role in promoting moral and civic education (e.g., colby, ehrlich, beaumont, & stephens, 2003; lempert, 1996; piper, 2002; orr, 1993; rhoads, 2000) and numerous higher education policy documents directly speak to the importance of these goals (e.g., dfes, 2003; world conference of higher education, 1998; national panel report, 2002). more over, the focus for the 20032004 cohort of higher education carnegie scholars organized by the carnegie academy for the scholarship of teaching and learning, was on liberal learning. the carnegie foundation also recently initiated the “political engagement project” to address the problem of young people increasingly disengaging from politics. this initiative builds on the earlier work by colby et al. (2003) on moral and civic responsibility. clearly, all these projects demonstrate careful consideration of the goals and outcomes of higher education and a concern with student learning that goes beyond the development of discipline-experts, or training for specific jobs. however, the links between these moral and civic goals and the scholarship of teaching and learning, though possibly assumed by many, has not always been made explicit. even though many have discussed the importance of educational goals and purposes over the years, until recently these discussions occurred largely outside rather than within the discourse on the scholarship of teaching and learning (by which i mean a discourse found in softl specific journals and conferences). i suggest that it is perhaps for this reason that many faculty and administrators associate the scholarship of teaching and learning still primarily with the notion of “best (teaching) practices” rather than a broader notion of professionalism that would integrate the idea of “citizenship”. though the latter is possibly taken for granted by some, it still occupies kreber, c. journal of scholarship of teaching and learning, vol. 6, no. 1, august, 2006. 90 somewhat of a secondary role in conceptions of what the scholarship of teaching and learning is, should, or could be. in this article, then, i suggest that the scholarship of teaching and learning involve (1) careful consideration of educational goals and purposes suitable for addressing the various political, social, cultural, environmental and economic challenges of our times, (2) understanding how students learn and develop toward these and other academic goals, and (3) identifying ways to best facilitate this learning and developmental process. i further contend that the notion of the scholarship of teaching and learning implies that we approach our teaching practice with the same sense of skepticism that guides our research. as researchers, we habitually provide arguments or reasons for our assertions. depending on our discipline or subject area, we engage in the process of hypothesis testing, interpretation or critical analysis routinely. moreover, we recognize that it is important to share with colleagues the evidence we generated for our point of view and invite them to follow, and possibly critique, our lines of argumentation. it has been proposed that we engage in similar processes with respect to our teaching; however, traditional ways of sharing such as conference papers and journal publications are but two of several possibilities. later in this paper i will discuss how and why teaching portfolios are particularly suitable for this purpose. in the remainder of this article i will build on these observations and discuss how the scholarship of teaching and learning may be developed through transformative learning (mezirow, 1991), a process by which faculty construct knowledge about teaching and learning through reflection. in line with the earlier arguments, i suggest that faculty construct knowledge in three different domains. the first domain of knowledge relates to what we consider to be meaningful goals and purposes of higher education (curricular knowledge). the second refers to what we know about student learning and development in relation to these goals (pedagogical knowledge, or perhaps more appropriately referred to as psychological knowledge). the third pertains to what we know about the teaching and instructional design processes needed to bring about student learning and development (instructional knowledge). questions that will be examined in this article include: • how is reflection on teaching and learning valuable? • what role do experience and theory play in reflection on teaching? • what is transformative learning? • are there different kinds or levels of reflection, and if so, are all levels of reflection equally conducive to fostering change and development in higher education teachers? • how is reflection, and transformative learning, linked to the scholarship of teaching? • how can transformative learning on teaching and student learning be demonstrated and reviewed? ii. how is reflection on teaching and learning valuable? for more than two decades researchers have explored the role of reflection in teacher training (e.g., hatton & smith, 1995; calderhead, 1989; zeichner et al., 1987) and more recently, it has become a buzzword also in higher education. time and again faculty are reminded of the importance of reflecting on their teaching (e.g., brookfield, 1991, 1995; cranton, 1998; ramsden, 1992; schön, 1995), though the process of reflection itself remains poorly understood kreber, c. journal of scholarship of teaching and learning, vol. 6, no. 1, august, 2006. 91 (e.g., moon, 2000). reflection also has been identified as a key process in the scholarship of teaching and learning. andresen (2000), for example, argued that the scholarship of teaching, should be inquirydriven, involve critical reflectivity, and scrutiny by peers. two other studies, one with “experts” in the scholarship of teaching (kreber, 2002b) and one with “regular faculty” (kreber, 2003) showed that both “experts” and regular academic staff consider the scholarship of teaching to be inquiry driven and to involve critical reflection. trigwell et al. (2000) identified five qualitatively different conceptions of the scholarship of teaching in a study with faculty at an australian university following the research tradition of phenomenography (marton 1981). these five conceptions were shown to differ in terms of four dimensions, one of them being the focus that reflection on teaching can take. finally, glassick et al (1997) proposed reflective critique as one of six criteria or standards by which to judge the scholarship of teaching. clearly, reflection is recognized as an important aspect of the scholarship of teaching. however, just what precisely it is that we hope reflection will accomplish is not always made clear. at the same time there perseveres a deeply-held belief that through reflection we can enhance our teaching practice, and by extension, the learning experiences of our students. such conclusions presuppose that reflection will lead to valid and valuable forms of knowing. for if the outcomes of reflection on teaching were not assumed to be valid, how would such reflection be meaningful? it is this idea of checking whether what we think actually makes sense, or is “valid”, given the context we find ourselves in, that is at the core of mezirow’s (1991) theory of transformative learning. in emphasizing the importance of validity testing, mezirow is inspired by the work of german sociologist jürgen habermas. borrowing heavily from habermas’s (1971) theory of knowledge-constitutive interests, mezirow distinguishes three different forms of learning: instrumental, communicative and emancipatory. it is through these three forms of learning that we can test the validity of our assumptions. a.three different forms of learning when engaged in instrumental learning we verify a belief or assertion by subjecting it to the empirical-analytical method; that is by posing it as a hypothesis that then can be tested by gathering data that will either support or contradict it. communicative learning, on the other hand, relies on the notion that through communication with others we can reach a common understanding on what is true. an assertion or belief is valid, therefore, if agreement on it can be reached within a community. such a consensus then is based on what the community at some point has accepted as the norm. while communicative learning is very valuable if the goal is to reach greater understanding within a framework of given norms, this form of learning does not concern itself with the question of how these norms have come about. put differently, through communicative learning we do not ask “why did we ever conclude that things should be this way?” or “why does it matter that we do them this way?” in response to the limits of communicative learning, mezirow (1991), leaning on habermas (1971), suggests that important aspects of learning do not occur on the basis of subjective understanding and consensus within a given social context. instead the most significant forms of learning involve a critical analysis of the processes and conditions by which certain norms we have come to take for granted have evolved and how “consensus” was reached. this is the nature of emancipatory learning. kreber, c. journal of scholarship of teaching and learning, vol. 6, no. 1, august, 2006. 92 whether or not the assumptions or conceptions we hold about university teaching are valid, therefore, can be determined through instrumental, communicative, or emancipatory learning or any combination of these. iii. what role do experience and theory play in reflection on teaching? when we think of reflection on teaching, we typically have in mind teachers reflecting on their personal teaching experiences rather than on research findings (see also hiebert, gallimore & stigler, 2002; huberman, 1985). this notion is supported by an extensive body of literature which argues that instructors who reflect on their teaching experiences acquire knowledge that is useful to them in the contexts in which they teach (e.g., cochran-smith & lytle, 1990; munby & russell, 1994; schön, 1983). moreover, these scholars suggest that the teachers’ personal knowledge, constructed on the basis of teaching experience, is more valuable than theoretical or research-based knowledge on teaching. theoretical knowledge about teaching, such as that found in books and academic journals, is, according to these scholars, more or less irrelevant or worthless as it cannot be directly applied to practice. an intriguing question, however, is whether the reflective process might also be directed to theoretical knowledge about teaching, and if so, under what circumstances would theoretical knowledge be of value to teachers? norris (2001) suggests that the value of educational theory for the practice of teaching depends on how teachers engage with theory. educational theories, he argues, surely will not seem particularly useful to teachers if they are wrongly expected to serve as situational or context-specific problem-solving strategies. this cannot be the purpose of any theory. instead, teachers need to understand the value of theories as “general models” which they need to adapt to their specific context. whether and, if so, how such research-based knowledge applies to a teacher’s given situation is a question that only those who know the particulars of the situation can answer. “when the situation is the classroom, teachers know the most about them” (norris, 2001). hiebert et al. (2002) also emphasize the importance of local hypotheses that teachers develop and test across specific contexts thereby working in collaboration with researchers “to digest and transform their general findings into professional knowledge for teaching “ (p.13). it follows that while reflection is certainly associated with experience (see also boud, keoch & walker, 1985; dewey, 1991; kelly, 1955; kolb, 1984; mezirow, 1991; moon, 2000) it also plays a significant role in determining the usefulness of theoretical or research-based knowledge. jarvis (1999) summed it up most succinctly when he argued that theories serve as information that practitioners need to transform into situationspecific knowledge as they try them out in practice. in doing so they create valid knowledge. the view that there are at least two equally important sources of reflection on teaching – educational theory and teaching experience--, has been repeatedly articulated also in the higher education teaching and learning literature (e.g., kreber, 2002; kreber & cranton, 2000; menges & weimer, 1996; paulsen, 2001; rando & menges, 1991). as rando and menges (1991) suggested more than a decade ago: “articulating a rationale for one’s instructional world…requires reflection about personal theories, knowledge of formal theories, and blending of the personal and formal” (pp. 13-14). while it is indeed important to consider both experience and theory in the discussion of reflection on teaching, doing so does not in and of itself address the question of how reflection enhances the practice of teaching. as we have just seen, jarvis proposed that theories are validated through practical experience but the idea of validation would benefit from a more thorough analysis. as discussed above, mezirow (1991) suggests that we come to know things as being either “true” or “false” through instrumental, kreber, c. journal of scholarship of teaching and learning, vol. 6, no. 1, august, 2006. 93 communicative, and/or emancipatory learning. these forms of learning are linked to different kinds of reflection. we will turn to these different forms of learning and kinds of reflection next. iv. what is transformative learning? mezirow’s theory of transformative learning, to a large extent, is informed by the cognitive-developmental tradition (e.g., kelly, 1955; piaget, 1964). this particular strand of psychology assumes that individuals develop intellectually as they encounter events that cannot be interpreted through their existing mental frames of reference. intellectual development occurs as frames of reference get revised as a result of reflection. a frame of reference, often called a conceptual structure, is interpreted as an interrelated set of assumptions, constructs, or conceptions, individuals actively form through experience. as specific assumptions are questioned and possibly revised (or transformed) in light of contradicting evidence, this can lead to a more substantial change in frame of reference (or “conceptual structure”). since individual assumptions are understood to be hierarchically organized, it would depend on their position or relative importance within the larger frame of reference, whether a revised assumption leads to a transformation in the frame of reference itself. to illustrate this point by means of an example, think of a new faculty member whose assumption that all students would prepare the readings assigned for class is challenged early in the semester when she notices that this holds true for only about 50 % of students. clearly, she now realizes that undergraduate students are not equally ready to assume responsibility and control over their learning. whether or not the transformation of this one assumption (or conception) will actually promote a more substantial transformation in frame of reference (or conceptual structure) will depend on the importance she attributes to this new knowledge. it would be possible, for instance, that she begins to question related assumptions and reflects on how she could better facilitate the process of self-regulated learning for different students, and whether, and if so, why, it matters that students learn to take on more responsibility for their learning. since, as was noted, the assumptions (or conceptions) we hold about teaching and learning are in some ways interrelated, a transformation of one assumption may promote reflection on other assumptions. not in all cases, however, will reflection lead to a drastic change in frame of reference for, through reflection, we may also find our assumptions to be confirmed or validated. let us now look at the different forms “reflection” (as conceived by mezirow) can take. v. are there different kinds or levels of reflection, and if so, are all levels of reflection equally conducive to fostering change and development in higher education teachers? in distinguishing between different kinds of reflection, mezirow put clearer parameters on the rather vague term “reflection” and, hence, made a significant contribution to our understanding of the reflective process and teacher thinking. the three kinds of reflection mezirow identified are content, process, and premise reflection. he describes the differences between the three forms of reflection as such: the critique of premises or presuppositions pertains to problem posing as distinct from problem solving. problem posing involves making a taken-for-granted situation problematic, raising questions regarding its validity. … the term “critical reflection” often has been used as a synonym for reflection on premises as distinct from reflection on kreber, c. journal of scholarship of teaching and learning, vol. 6, no. 1, august, 2006. 94 assumptions pertaining to the content or process of problem solving (mezirow, 1991, p.105). it follows that content, process and premise reflection are very different in terms of their nature and what they can achieve; indeed, one could say that they represent different levels of reflection. the term content reflection is at times confusing to people, particularly when discussed in the context of teaching and learning. contrary to our intuitive understanding, the term “content reflection” does not refer to reflection on the content of the courses we teach. what mezirow means by content reflection is having a clear sense of, and providing a description of, the content of the problem that we need to solve. in short, content reflection asks “what’s really the problem here and what do i need to do?”. in content reflection, we do not question the presuppositions underlying our argument but simply use our present knowledge, that is the assumptions or conceptions we presently hold, to describe a problem and how it is habitually solved by us. according to mezirow, content reflection, is a process in which we “are not attending to the grounds or justification for our beliefs but are simply using our beliefs to make an interpretation” (mezirow, 1991, p.107). to be clear, the question of whether our knowledge is valid is not one posed by content reflection. all we ask through content reflection is “what do i presently know about how to solve this problem?” process reflection, on the other hand, is focused on the effectiveness of the problemsolving strategy itself. here we ask “how do i know that i am effective (or was conscientious) with what i do?”. finally, in premise reflection, we call into question the presuppositions on which our present knowledge is based and ask “why is it that i choose to attend to this problem— is there an alternative?” how these forms of reflection are linked to the three forms of learning discussed earlier-instrumental, communicative and emancipatory-is illustrated next. it has become evident that content reflection does not address the question of validity of the outcomes of reflection. through process and premise reflection, however, we test the validity of our assumptions or conceptions. in process reflection we find out whether what we do works by seeking some form of evidence for its “effectiveness” (which in some case is better interpreted as “meaningfulness” or “conscientiousness”). this evidence might be found through published research we read about, research we conduct ourselves, or through experience such as talking to others. reflection then can be informed by the two sources of knowledge construction discussed earlier: personal teaching experience and educational theory. process reflection then occurs through either instrumental or communicative forms of learning, or both. in case of instrumental learning we might validate our knowledge by posing it as a hypothesis that we then test (for example, “students achieve better test scores if i give them the opportunity to choose between two assignments”). in case of communicative learning, we might validate our knowledge as we discuss what we assume to be true with a community of peers to achieve consensus (for example, as we discuss the meaningfulness and relevance of certain goals or values which guide our curriculum planning). obviously, it is only when we are engaged in premise reflection, that is the questioning of presuppositions of what we believed to be true, that our learning would become also emancipatory. we may question, for example, why we ever decided on certain goals and values and examine the processes and conditions by which these came about. kreber, c. journal of scholarship of teaching and learning, vol. 6, no. 1, august, 2006. 95 if our goal is to enhance university teaching, it is vital that we encourage also process and premise reflection rather than just content reflection on teaching. let us now explore what it might look like when people engage in content, process or premise reflection on teaching. v. how is reflection, and transformative learning, linked to the scholarship of teaching? kreber and cranton (1997, 2000) suggest that the scholarship of teaching and learning involves learning about three equally important domains of teaching knowledge: (1) knowledge about the goals and purposes of our teaching (labelled curricular knowledge), (2) knowledge about how students learn (back then was labelled pedagogical knowledge but should perhaps have been labelled psychological knowledge), and (3) knowledge about instructional design and the instructional process (labelled instructional knowledge). this taxonomy is not unrelated to other models describing the knowledge base of teaching (e.g., rice, 1991; shulman, 1987). the main difference is that the soft model is not limited to identifying knowledge domains but is concerned with the construction of knowledge, through reflection, in each domain. another important aspect of the model is that it stresses the critical examination of goals and purposes of higher education as an integral part of the scholarship of teaching and learning. mezirow’s three levels of reflection (on content, process and premise) serve to explain how faculty might construct knowledge in the domains of curriculum (what are the goals and purposes of our teaching?), pedagogy (how do students learn and develop toward these goals?), and instruction (what can be done to optimize this learning or developmental process?). it follows that individuals may actually be involved in as many as nine different kinds of reflection (namely content, process and premise reflection in the three domains of instructional, pedagogical and curricular knowledge) with each of the nine kinds of reflection generating a slightly different form of teaching knowledge. in summary, as higher education teachers are involved in any of these nine forms of reflection, they draw on their teaching experience or educational research, or both. clearly then, reflection, whether informed by experience or theory, leads to knowing, indeed, as many argue, is a process of active knowledge construction. the knowledge higher education teachers construct through these forms of reflection can be tested for its validity through instrumental, communicative or emancipatory learning processes. when our assumptions become validated as a result of process or premise reflection, we can present some sound arguments by which to justify our practice. to borrow dewey’s (1933) words, through reflection we carefully considered “any belief or supposed form of knowledge in light of the grounds that support it and the further conclusions to which it tends” (p.9). when individual assumptions were not validated as a result of reflection, transformative learning (a revision of assumptions or conceptions) might occur as a result. premise or critical reflection on a single assumption may or may not lead to a transformation of one’s larger frame of reference (or conceptual structure) on teaching2. table 1 summarizes what has been argued and provides some concrete examples of possible responses to the reflective questions posed by the scholarship of teaching (soft) model. tables 2 to 4 illustrate the process of content, process 2 it might be useful to think of the conceptual structure as something similar to prosser and trigwell’s (1999) “conceptions of teaching” (see also martin & balla, 1991; ramsden, 1992; kember & gow, 1994), which can be predominantly teacher-focussed or increasingly more student-focussed and oriented towards promoting students’ conceptual change in the understanding of subject matter. for instructors to change their conceptions, transforming one single assumption may or may not trigger a more drastic change or transformation in “conceptions of teaching”. kreber, c. journal of scholarship of teaching and learning, vol. 6, no. 1, august, 2006. 96 and premise reflection for the domains of curricular knowledge, pedagogical (or rather psychological) knowledge and instructional knowledge respectively. clearly, the goals we identify as a result of reflection within the domain of curricular knowledge influence the reflective processes in the other two domains. the three goals discussed in the tables --selfmanagement, autonomy and social responsibility-are examples only (though they are, of course, consistent with the understanding of the scholarship of teaching and learning as a “professionalism” that is inclusive of the notion of “citizenship”). obviously there are other important goals of higher education including those that are more subject, discipline or program specific. an essential aspect of the soft model is its emphasis on justifying educational goals through process and premise reflection. following this model, faculty can provide evidence of engagement in the scholarship of teaching. this involves demonstrating that we made efforts to validate our knowledge of teaching in learning in the three knowledge domains, and acted on the results of our reflection. given appropriate criteria, it could also be assessed through a process of peer review (kreber, 1998; 2001). how the various reflective processes could be demonstrated in a teaching portfolio i will discuss below. vii. how can transformative learning on teaching and student learning be demonstrated and reviewed? the idea of the teaching portfolio originated in canada in the early 1970s (knapper, mcfarlane, & scanlon, 1972) and later resulted in a publication sponsored by the canadian association of university teachers (caut), entitled "the teaching dossier: a guide to its preparation and use" (shore and others, 1980, 1986). teaching portfolios are meant to have both formative and summative evaluative purposes (edgerton, hutchings, & quinlan, 1991; knapper, 1995; smith, 1995). this is to say that they are meant to promote teacher growth and provide a basis for judging teaching effectiveness. typically, teaching portfolios include various documents among them a philosophy statement, outlines of courses taught, unsolicited comments from students, written feedback from colleagues, examples of course work completed by students, summary of teaching evaluation from students, and so forth. evidently, sources such as these, particularly if compiled together, provide a broader and more objective picture of teaching effectiveness than, for example, student ratings of instruction alone. clearly, teaching portfolios can be very useful in demonstrating teaching effectiveness to an evaluation committee. however, it is less obvious how teaching portfolios thus construed serve their formative purpose. most teaching philosophy statements that i have read over the years provide “thick” descriptions (e.g., lincoln & guba, 1985) of present practices but demonstrate little evidence of critical reflection on the underlying assumptions themselves. these descriptions are good examples of what is meant by “content reflection”. surely, this does not necessarily mean that faculty are not critically reflective; it is equally possible that many simply do not know how to demonstrate their reflection. my purpose in this essay is to show how teaching portfolios could be used to document engagement in the various reflective processes associated with the soft model. the portfolio then could be a means not only for stating the assumptions we hold about instructional design issue, student learning and development and goals and purposes but also for documenting the processes by which assumptions were constructed. the basic idea behind such a portfolio is that both authors and reviewers can form judgements regarding the validity of the outcomes of reflection by exploring and assessing the extent to which stated assumptions are the result of kreber, c. journal of scholarship of teaching and learning, vol. 6, no. 1, august, 2006. 97 table 1 the scholarship of teaching (soft) model (adapted by kreber & cranton, 2000): content, process and premise reflection in the three knowledge domains (examples of possible responses) curricular knowledge pedogogical/psycholog ical knowledge instructional knowledge validity testing involved? form of learning sources of knowledge used in reflection content reflection “what are the goals and purposes of my teaching?” “what do i know about how students learn and develop?” “what instructional strategies should i use?” the problem is described “the problem i need to solve here is clarifying my goals.” “the problem i need to solve here is how to promote moral development and social responsibility.” “the problem i need to solve here is (for example) how to provide students with real-life problems.” i make explicit what i already “know” (what i believe to be true) no challenging of assumptions is taking place at this point. largely experience-based but could also be research-based possible habitual response “my main goal is to promote in students a greater sense of moral development and social responsibility.” “i can do this by providing them with opportunities to solve real-life ill-structured problems.” “i can do this by incorporating a service learning component.” no process reflection “how conscientious have i been in identifying this goal?” “how effective am i in learning how to promote moral development and social responsibility?” “how effective has my use of service learning been in providing students with opportunity to solve real-life problems?” yes instrumental and/or communicative experience-based and/or researchbased premise reflection “why does my goal of promoting social responsibility matter – what possible alternatives are there?” “why does it matter that i offer opportunities to solve real-life and illstructured problems – what possible alternatives are there?” “why does it matter that i use this approach (here service learning). is there an alternative? yes emancipatory (possibly preceded by instrumental and/or communicative learning) experience-based and/or researchbased kreber, c. journal of scholarship of teaching and learning, vol. 6, no. 1, august, 2006. 98 table 2 examples of possible responses to content, process and premise reflection questions in the domain of curricular knowledge. first goal second goal third goal content reflection “what are my goals” process reflection “how effective or conscientious have i been in identifying these goals” or “how did these goals come about”? premise reflection “why does it matter that i promote these goals?” “my goal is to promote selfmanagement.” “the need for students to develop the skills, abilities and attitudes needed for continuous learning has been repeatedly emphasized in the educational literature.” “while there are other important goals, students need to acquire the capacity to engage in continuous adaptive learning because such learning has become a reality in our rapidly changing world. without this capacity individuals will be seriously limited in their opportunity to make changes necessary to improve their lives.” “my goal is to promote personal autonomy” “philosophers, educators and social critics have identified the cultivation of autonomous individuals as an important purpose of higher education.” “while there are other important goals, students need to be able to distinguish mere habit and convention from what they can defend by argument in order to solve the most pressing problems of our times. without this ability individuals easily become puppets and automatons of hegemonic forces that take control of their lives.” “my goal is to promote social responsibility” “people arguing that higher education should bring about autonomous thinkers also emphasize the need for responsible citizens. carnegie identified encouraging civic engagement as an important educational purpose.” “while there are other important goals, students need to develop a sense of responsibility towards the community and the environment because our most pressing problems can be solved only by recognizing that we are in this together. without this sense of stewardship and citizenship we open the door to conflict as marginalized groups and social and environmental issues easily get ignored.” kreber, c. journal of scholarship of teaching and learning, vol. 6, no. 1, august, 2006. 99 table 3 general questions asked in content, process and premise reflection in the domain of pedagogical (or “psychological”) knowledge. self-management autonomy social responsibility content reflection process reflection premise reflection “what do i know about how students develop a sense of self-management?” “how do i know that what i believe about how to promote self-management (for example, promote selfregulated learning, deep level approaches, self-evaluation, information finding, etc), is actually true?” and “how effective have i been in identifying, or learning, about how students develop selfmanagement?” “why does it matter that i focus on these constructs, ideas or theories to help students develop selfmanagement – what possible alternatives are there?” “what do i know about how students develop a sense of autonomy?” “how do i know that what i believe about how to promote autonomy (for example, promoting intellectual development, critical thinking, exercising freedom of choice, etc) is actually true?” “why does it matter that i focus on these constructs, ideas or theories to help students develop autonomy - what possible alternatives are there?” “what do i know about how students develop a sense of social responsibility?” “how do i know that what i believe about how to promote social responsibility (for example, promoting moral development, social commitments, cultural and environmental sensitivity, etc) is actually true?” “why does it matter that i focus on these constructs, ideas or theories to help students develop social responsibility -what possible alternatives are there?” kreber, c. journal of scholarship of teaching and learning, vol. 6, no. 1, august, 2006. 100 table 4 general questions asked in content, process and premise reflection in the domain of instructional knowledge. self-management for example, linked to self-regulated learning deep level learning self-evaluation finding information autonomy for example, linked to critical thinking deep level learning intellectual development exercising freedom of choice social responsibility for example, linked to moral development cultural and environmental sensitivity content reflection process reflection premise reflection “what do i know about the strategies needed to help students develop a sense of self-management?” “how do i know that these strategies are effective?” “why does it matter that i use these strategies -what possible alternatives are there?” “what do i know about the strategies needed to help students develop a sense of autonomy?” “how do i know that these strategies are effective?” why does it matter that i use these strategies -what possible alternatives are there?” “what do i know about the strategies needed to help students develop a sense of social responsibility?” “how do i know that these strategies are effective?” why does it matter that i use these strategies -what possible alternatives are there?” kreber, c. journal of scholarship of teaching and learning, vol. 6, no. 1, august, 2006. 101 engagement in process and premise reflection, or instrumental, communicative and emancipatory learning. in order for us to demonstrate our reflection on the various aspects of the scholarship of teaching model, it is not enough to tell others that we reflected. demonstrating reflection more convincingly, would involve providing indicators of engagement in the reflective processes underlying the scholarship of teaching and learning (here the soft model). indicators of engagement in the scholarship of teaching are concrete actions we took from which engagement in the various reflective learning processes can be inferred. while kreber and cranton (2000) proposed a list of indicators based on a conceptual analysis, a recent study with thirty-six science faculty (kreber, 2005) identified indicators empirically. the list below draws on both records but makes some additional suggestions. the items in the list are concrete actions faculty can take and also describe and document in the portfolio. 1. describing the instructional strategies one uses (content reflection/instructional knowledge--experience-based); 2. asking for peer review of course outline (process reflection/instructional knowledge--experience-based); 3. collecting data on students' perceptions of methods and materials (process reflection/instructional knowledge—experience-based); 4. experimenting with alternative teaching approaches and checking out results (premise reflection/instructional knowledge—experience-based); 5. comparing different instructional strategies for their suitability in a given context (premise reflection/instructional knowledge—experience-based) 6. paying attention to end of term teaching evaluations (process reflection/instructional knowledge—experience-based) 7. writing critiques on “how-to teaching books” (premise reflection/instructional knowledge—research-based); 8. administering learning styles or other inventories to students (process reflection/pedagogical knowledge—research-based/experience-based); 9. writing an article on how to facilitate learning in the discipline and submit it to a scholarly journal (content/process reflection/pedagogical knowledge—researchbased); 10. gathering feedback from students on their learning of discipline-specific concepts (process reflection/pedagogical knowledge—experience-based); 11. comparing research-based insights gained from courses on teaching and learning to one’s knowledge of how students learn (process reflection/pedagogical knowledge—research-based) 12. listening to others, observing how others learn, and discussing or writing about it (process reflection/pedagogical knowledge—experience/research-based) 13. reading articles or books on learning and developmental theory (content reflection/pedagogical knowledge—research-based); 14. observing others teach and observing the reactions of their learners (process reflection/pedagogical knowledge—experience-based) 15. conducting an action research project on student learning (process reflection/pedagogical knowledge—research-based); kreber, c. journal of scholarship of teaching and learning, vol. 6, no. 1, august, 2006. 102 16. presenting findings from classroom teaching experiments at teaching-related sessions at conferences (process reflection/instructional knowledge—researchbased); 17. showing how goals of one’s teaching relate to what students need to live successful lives (process reflection/curricular knowledge—experience-based) 18. consulting with an educational development specialist (process reflection/pedagogical knowledge—research-based); 19. comparing classroom experience to formal research results on student learning (process reflection/pedagogical knowledge—research-based); 20. explaining how and why goals have changed over time (premise reflection/curricular knowledge—experience-based); 21. consulting with employers to see what goals they have in mind (premise reflection/curricular knowledge—experience-based). 22. participating in a curriculum review committee (premise reflection/curricular knowledge—experience-based). 23. participating in philosophical discussions on student learning, for example through a listserv or with colleagues (premise reflection/pedagogical knowledge—experience-based) 24. reading books on the goals of higher education and comparing goals to those underlying the programs offered in the department (process reflection/curricular knowledge—research-based) 25. writing articles that compare the usefulness of textbooks in one’s field and compare outcomes of analysis to own text and course content (process reflection/curricular knowledge—research-based) prior to discussing this list it might be useful to make one further observation. it has been suggested that there is a difference between scholarly teaching and the scholarship of teaching (e.g., richlin, 2001) and some feel that this distinction is one of critical importance. to the extent that we adopt this view, we will conclude that most of the above indicators reflect scholarly teaching but not the scholarship of teaching. however, does interpreting the scholarship of teaching exclusively as “publication of research on teaching and learning” (and considering any practices that do not meet this criterion as scholarly teaching) not unnecessarily reduce the scholarship of teaching to the scholarship of discovery on teaching3? the latter, i propose, is just one way by which to engage in and demonstrate the scholarship of teaching. if the scholarship of teaching is aimed at enhancing the quality (and recognition) of teaching and supporting student learning, is informed by knowledge of the field, is inquiry-driven, involves critical reflectivity and scrutiny by peers, as many have argued over the years (e.g., andresen, 2000), does it then not follow that the above indicators are indeed indicators of the scholarship of teaching, particularly if they themselves are shared with peers? essentially, the idea of using indicators is that they allow us to make inferences about the kind of learning or reflection the faculty member has engaged in. to the extent that the indicators suggest engagement in process and premise reflection (through instrumental, 3 i assume that advocates of this view would suggest that the difference between what educational researchers do and what scholars of teaching do, is that for scholars of teaching the research problem originates in their own teaching practice, whereas for educational researchers the problem originates on the basis of theory. this distinction, however, would not hold true for all educational researchers either. kreber, c. journal of scholarship of teaching and learning, vol. 6, no. 1, august, 2006. 103 communicative or emancipatory learning), we can infer that assumptions about teaching and learning were tested for their validity. to be clear, not every indicator on this (or any other) list of suggestions needs to be addressed. the idea is that one would want to see some evidence of process and premise reflection in the domains of curricular, pedagogical and instructional knowledge. such evidence might be produced through traditional forms of inquiry and sharing such as studies of how students learn that are then published, but many of the above indicators do not involve publication. without a doubt, the scholarship of teaching can be demonstrated in many different ways (see for example, theall & centra, 2001). other indicators than the ones suggested here are clearly possible and the development of further indicators by those who practice the scholarship of teaching is both necessary and encouraged. for an outsider it is not always easy to decide whether an indicator (i.e., a concrete action that a faculty member may take) gives evidence of content, process or premise reflection on the part of a faculty member. a higher degree of trustworthiness in the interpretation of indicators might be possible only by talking to the faculty member him or herself. when teaching portfolios are used as described in this essay, namely with the intent to record and demonstrate reflection of different kinds, their greatest value may lie in the formative purposes they serve. conversations based on the portfolio between educational developer and faculty member would hold great promise for further reflection and continued growth in teaching. as for summative purposes, the portfolio holds potential as well. as was noted, it is neither likely nor necessary that reviewers of portfolios will be able to decide for each recorded instance whether reflection was focussed on content, process or premise. nonetheless, reviewers can still arrive at an overall impression of whether the faculty member went beyond content reflection on goals, his or her understanding of student learning and development, and how to promote academic learning and development. results from a recent study suggest that faculty engage primarily in content reflection on teaching, followed by process reflection and that premise reflection across all three knowledge domains is not as common (kreber, 2005b). viii. concluding comments i argued in this essay that the scholarship of teaching and learning needs to be informed by a broader conceptualizations of professionalism, one that is not limited to “best teaching practices” but is inclusive of the notion of citizenship (contributing to the university’s moral and civic purpose). the scholarship of teaching (soft) model (kreber & cranton, 2000) suggests that faculty develop in the scholarship of teaching and learning as they actively construct and validate their knowledge through reflection in three domains of teaching knowledge: (1) knowledge about the goals and purposes of university teaching (curricular knowledge), (2) knowledge about student learning and development toward those goals (pedagogical knowledge, or rather psychological knowledge), and (3) knowledge about how to optimize this learning and developmental process (instructional knowledge). reflection is informed by knowledge gained through personal experience and/or through formal inquiry faculty conduct themselves or read about. content, process and premise reflection are three qualitatively different kinds of reflection. only in process and premise reflection are assumptions or conceptions questioned for their validity. in content reflection we merely make our present assumptions explicit (i.e., we state what we believe to be true); however, content reflection is not irrelevant as identifying kreber, c. journal of scholarship of teaching and learning, vol. 6, no. 1, august, 2006. 104 assumptions is a critical first step in reflection. both process and premise reflection go beyond this and involve “validity testing”. process reflection occurs through instrumental and/or communicative forms of learning and focuses on the problem-solving strategy (“how do i know that what i’m doing makes sense?”). premise reflection can lead to emancipatory learning and focuses on the presuppositions underlying our practices and how they came to be taken for granted (“why does it matter that i/we focus on this problem?”). reflection on assumptions (or conceptions) we hold does not always lead to a transformation of assumptions (or conceptions) to be meaningful (kelly, 1955; mezirow, 1991), as through reflection we may find justification for our practices. only if, through reflection, assumptions are not validated might transformative learning take place. however, even then it will depend on a combination of factors -including personal (for example, willingness to change), social (for example, support by others for change) and contextual ones (for example, external constraints to change) -whether reflection will lead to a transformation of assumptions and, ideally, changes in practice. as well, as was stated earlier, whether or not one transformed assumption leads to a transformation in the larger frame of reference (or conceptual structure) on teaching depends on the importance we attribute to our new insight or knowledge. the scholarship of teaching and learning thus construed involves both learning and knowing about teaching. teaching portfolios offer the opportunity to document or publicize our engagement in the scholarship of teaching and learning and to share or exchange the insights we gained through the various reflective processes with the larger academic teaching community so that they, in turn, can review and critique our practices. this exchange or sharing of indicators of reflection with members of the academic teaching communities may encourage others to build on our work. it might make most sense to start this sharing within our own departmental contexts where our insights can immediately be built upon to improve practice. in an ideal scenario there would be teaching environments in place in all departments across the university that encourage faculty to support each in other in the process of content, process and premise reflection on educational goals and purposes, learning and student development and instruction design. but small groups of faculty who start a weekly or monthly discussion group can make a difference. and if this group decides to go together to conferences on teaching and learning to share their own work more widely or to learn from that of colleagues, they have even greater insights to share with their own departmental colleagues when they return. i should not conclude without noting that the three forms of reflection described here (on content, process and premise) are also useful in the planning of educational development program initiatives. questions program planners could ask themselves include: ● to what extent are participants in our program encouraged to engage in content, process and premise reflection on the goals and purposes of the courses they teach specifically and on the goals and purposes of a university education more generally? ● to what extent are they encouraged to engage in content, process and premise reflection on learning and student development? ● to what extent are they encouraged to engage in content, process and premise reflection on instructional design (including teaching and assessment methods)? ● to what extent are they encouraged to reflect on their personal experience and on education theory? ● to what extent are they encouraged to contribute to educational theory? ● how do we evaluate these kinds of learning? kreber, c. journal of scholarship of teaching and learning, vol. 6, no. 1, august, 2006. 105 i suggested that faculty can record the indicators of their reflections (i.e., the concrete activities they engaged in that made them reflect) in the form of a teaching portfolio for formative and summative evaluation purposes. however, the list of indicators introduced earlier also serve a second purpose: these indicators are useful also for planning educational development assignments or activities for faculty and gtas as they are concrete examples of activities that can be planned that would involve course participants directly in the desired forms of reflection. references andresen, l.w. (2000). a usable, trans-disciplinary conception of scholarship. higher education research and development, 19(2), 137-153. boud, d., keoch, r., & walker, d. (1985). reflection: turning experience into learning. london: croom helm. brookfield, s. (1991). the skillful teacher. san francisco: jossey-bass. brookfield, s. (1995). becoming a critically reflective teacher. san francisco: jossey-bass. calderhead, j. (1989). reflective teaching and teacher education. teaching and teacher education, 5(1), 43-51. cochran-smith m. & lytle, s.l. (1990). research on teaching and teacher research: the issues that divide. educational researcher, 19(2), 2-11. colby, a., ehrlich, t., beaumont, e. & stephens, j. (2003). educating citizens. preparing america’s undergraduates for lives of moral and civic responsibility. the carnegie foundation for the advancement of teaching. san francisco: jossey-bass. cranton, p.a. (1998). no one way. toronto: wall and emerson. dewey, j. (1933). how we think. new york: heath. dfes (2003). the future of higher education (white paper). norwich, uk. edgerton, r., hutchings, p., & quinlan, k. (1991). the teaching portfolio: capturing the scholarship of teaching. washington, dc: american association for higher education. glassick, c.e., huber, m.t. & maeroff, g.i. (1997). scholarship assessed. evaluation of the professoriate. san francisco, ca: jossey-bass. habermas, j. (1971). knowledge and human interests. boston: beacon press. hatton, n., & smith, d. (1995). reflection in teacher education: towards definition and implementation. teaching and teacher education, 11, 33-49. kreber, c. journal of scholarship of teaching and learning, vol. 6, no. 1, august, 2006. 106 hiebert, j., gallimore, r., & stigler, j. (2002). a knowledge base for the teaching profession: whaat it would like it and how we can get there, educational researcher, 31(5), 3-16. huberman, m. (1985). what knowledge is of most worth to teachers? a knowledge-use persepctive. teaching and teacher education, 1, 251-162. hutchings, p., & shulman, l. s. (1999). http://www.carnegiefoundation.org/elibrary/docs/sotl1999.htm (also published in change magazine, september/october 1999. volume 31, number 5, pages 10-15) jarvis, p. (1999). the practitioner researcher: developing theory from practice. san francisco: jossey-bass. kelly, g.a. (1955). the psychology of personal constructs, vol. 1. new york, ny: norton and company. knapper, c.k., mcfarlane, b., & scanlon, j. (1972). student evaluation: an aspect of teaching effectiveness. caut bulletin, 21(2), 26-34. knapper, c.k. (1995). the origins of teaching portfolios. journal of excellence in college teaching, 6(1), 45-56. kolb, d. (1984). experiential learning. englewood cliffs, nj: prentice hall. kreber, c. (2005a). charting a critical course on the scholarship of university teaching movement. studies in higher education. volume 30, number 4, 389-407. kreber, c. (2005b). reflection on teaching and the scholarship of teaching. higher education. volume 50 (2), 323-359. kreber, c. (2003). the scholarship of teaching: conceptualizations of experts and regular academic staff. higher education,46 (1): 93-121. kreber, c. (2002a). teaching excellence, teaching expertise, and the scholarship of teaching. innovative higher education, 27(1), 5-23. kreber, c. (2002b). controversy and consensus on the scholarship of teaching: a delphi study. studies in higher education, 27 (2), 151-167. kreber, c. (2001). designing teaching portfolios based on a formal model of the scholarship of teaching. to improve the academy. the professional and organizational development network in higher education. bolton, ma: anker publishing com, vol 19, pp. 285-305. kreber, c. (1999). a course-based approach to the development of teaching-scholarship: a case study. teaching in higher education, 4(3),309-325. kreber, c. journal of scholarship of teaching and learning, vol. 6, no. 1, august, 2006. 107 kreber, c. (1998). perfection through reflection: the teaching portfolio to evaluate and professionalize university teaching (in german). in, handreichungen zur hochschuldidaktik (section 18, pp.1-24), handbuch hochschullehre. bonn, germany: raabe verlag. kreber, c. & cranton, p. a. (2000). exploring the scholarship of teaching. journal of higher education, 71(4), 476-496. kreber, c. & cranton, p.a. (1997). teaching as scholarship: a model for instructional development. issues and inquiry in college learning and teaching, 19(2), 4-13. kember, d., & gow, l. (1994). orientations to teaching and their effect on the quality of student learning. journal of higher education, 65, 59-74. lempert, d.h. (1996). escape from the ivory tower. student adventures in democratic experiential education. san francisco: jossey-bass. lincoln, y., & guba, e. (1985). naturalistic inquiry. beverly hills, ca: sage. martin, e., & balla, m. (1991). conceptions of teaching and implications for learning, research and development in higher education, 15, 148-55. marton, f. (1981). phenomenography-describing conceptions of the world around us. international science, 10, 177-200. menges, r. j. & weimer, m. (1996). teaching on solid ground: using scholarship to improve practice. san francisco: jossey-bass. mezirow, j. (1991). transformative dimensions of adult learning . san francisco: jossey-bass. munby, h. & russell, t. (1994). the authority of experience in learning to teach: messages from a physics methods class. journal of teacher education, 45(2), 86-95. moon, j.a. (2000). reflection in learning and professional development. london: kogan page. national panel report (2002). greater expectations: a new vision for learning as a nation goes to college. washington, dc: association of american colleges and universities. norris, s. p. (2001). the pale of consideration when seeking sources of teaching expertise. the american journal of education, 108 (3), 167-196. orr, d.w. (1990). the liberal arts, the campus, and the biosphere. harvard educational review, 60(2), 205-217. paulsen, m. b. (2001). the relation between research and the scholarship of teaching, in kreber, c. (ed.) revisiting scholarship: perspectives on the scholarship of teaching. new direction for teaching and learning, no. 86. san francisco: jossey-bass. kreber, c. journal of scholarship of teaching and learning, vol. 6, no. 1, august, 2006. 108 piaget, j. (1964). the child's conception of the world. london : routledge & kegan paul. piper, m. (2003). special feature: building a civil society. a new role for the human sciences. 2002 killam annual lecture. the canadian journal of higher education, 33(1), 113-130. prosser, m., & trigwell, k. (1999). understanding learning and teaching. the experience in higher education. society for research into higher education: open university press. ramsden, p. (1992). learning to teach in higher education. london: routledge. rando w.c. & menges, r.j. (1991). how practice is shaped by personal theories, in: menges, r.j. & svinicki, m.d. (eds) college teaching: from theory to practice. san francisco: josseybass. rhoads, r.a. (2000). democratic citizenship and service learning: advancing the caring self. in, m. baxter magolda (ed), teaching to promote intellectual and personal maturity: incorporating students’ worldviews and identities into the learning process (pp. 27-44). new directions for teaching and learning, no. 82. san francisco: jossey-bass. rice, r.e. (1991). the new american scholar: scholarship and the purposes of the university, metropolitan universities, 1, 7-18. richlin, l. (2001). ‘scholarly teaching and the scholarship of teaching’, in kreber, c. (ed), revisiting scholarship: perspectives on the scholarship of teaching, new directions for teaching and learning, no 86. san francisco, ca: jossey-bass. schön, d. (1983). the reflective practitioner. san francisco: jossey-bass. schön d (1995, november, december). the new scholarship requires a new epistemology. change, 27-34. shore, b. m., foster, s. f., knapper, c. k, nadeau, g. g., neill, n., & sim, v. (1980, 1986) the teaching dossier. (revised edition). montreal: canadian association of university teachers. shulman, l.s. (1987). knowledge and teaching, harvard educational review, 57, 1-22. smith. r.a. (1995). creating a culture of teaching through the teaching portfolio. journal of excellence in college teaching, 6(1), 75-100. theall, m.,& centra, j.a. (2001). assessing the scholarship of teaching: valid decisions from valid evidence. in c. kreber (ed.), revisiting scholarship: perspectives on the scholarship of teaching. new directions for teaching and learning, 86. san francisco: jossey-bass. trigwell, k., martin, e., benjamin, j, & prosser, m. (2000). scholarship of teaching: a model. higher education research and development, 19(2), 155-168. kreber, c. journal of scholarship of teaching and learning, vol. 6, no. 1, august, 2006. 109 walker, m. (ed) (2001). reconstructing professionalism in university teaching. society for research into higher education: open university press. world conference on higher education (1998). higher education in the twenty-first century. vision and action. volume i, final report. paris: unesco publishing. zeichner, k.m., tabachnik, b.r., & denmore, k. (1987). individual, institutional, and cultural influences on the development of teachers’ craft knowledge, in j. calderhead (ed) exploring teacher thinking. london, uk: cassell. journal of scholarship and teaching, vol. 6, no. 1, august 2006, pp 110 117. bridging the theory-practice divide: a creative approach to effective teacher preparation jacqueline a. hughes, ph.d.1 abstract: teacher educators need to remain current regarding the challenges that prospective teachers are going to face in their classrooms. one way to maintain this currency is for teacher educators periodically to spend some time in the k-12 classroom testing the theories they teach. this paper will discuss the benefits both teacher educators and prospective teachers will derive from engaging in such an activity. i. introduction preparing prospective teachers for the realities of today’s classrooms is a complex and challenging undertaking for teacher educators. this complexity and challenge is a result of the changing nature of the classroom. schools today face an increasing number of language learners, the mainstreaming of special population students, and, working with a standards driven curriculum, all of which present new challenges for the teacher as they attempt to meet their students educational needs. as a result of this “new classroom environment” and the educational needs they present teacher educators must now seek different approaches to prepare prospective teachers to meet these needs because the traditional (e.g. coursework independent of fieldwork) approaches to teacher preparation are no longer effective in equipping teachers to address these issues. it has been my observation that some teacher educators are so far removed from the k-12 environment that what they teach sometimes does not reflect the realities their students face. additionally, there is the belief that “learning to teach is a two-step process of knowledge acquisition and application or transfer” (feiman-nemser & remillard, 1996, p. 79). the latter view infers a mutually exclusive relationship between the teacher educator, the prospective teacher and the classroom. in this approach to teacher preparation, the teacher educator provides the knowledge and the prospective teacher applies it. however, the teaching of theories or knowledge to prospective teachers and expecting that they will effectively apply them is an inadequate approach (stuart & thurlow, 2000;wideen, mayer-smith & moon, 1998; adams, shea, liston & deever, 1998) to teacher preparation. the assumption lying herein is that prospective teachers not only acquired the knowledge and theories in their program but the wherewithal to apply it in their classrooms. this, of course, might be true if the process of learning to teach were linear rather than dynamic; free of extraneous influences and circumstance rather than a complex mélange of variables this thought is best captured by britzman (1991): “learning to teach is not a mere matter of applying decontextualized skills or of mirroring predetermined images: it is a time when one’s past, present and future are set in dynamic tension. learning to teachlike teaching itselfis always the process of becoming: a time of formation and transformation, of scrutiny into 1 assistant professor, school of education, california state university, bakersfield, (661) 665-6807, jhughes4@csub.edu hughes, j.a. journal of scholarship and teaching, vol. 6, no. 1, august 2006. 111 what one is doing and who one can become…learning to teach is a social process of negotiation rather than an individual problem of behavior.” since preparation of teacher candidates is, at best, a complicated process, teacher educators must consider adopting new practices: as such, for teacher educators to better prepare prospective teachers three things must occur: examination of their teaching practices and “the process of learning to teach” (szabo, scott & yellin, 2002, p.1); utilization of field work to aid prospective teachers in their process of meaningful reflection and construction of practical knowledge (perry & power, 2004); and, finally, inculcating prospective teachers’ understanding of the relationship between theory and practice (szabo, scott & yellin, 2002). it is the consistent interconnection and persistent engagement in the above facets of learning-to-teach that show promise of more effectively preparing teachers. in this paper, i will discuss a field-based approach i implemented to examine my own teaching practices in linking theory with practice and regaining currency in the real world of public school. i approached this project with the tentative optimism that my experiences would assist me in preparing prospective teachers for their “multiple roles and [the] contextual complexities of life in schools” (knowles & cole, 1996, p. 648). ii. a theoretical framework the teaching of theory must be (or should be) inextricably linked to its application (brunner, 1997). when students are exposed to theoretical concepts for the first time, they must be introduced to these ideas in a manner to which they can best relate (brunner, 1997). brunner (1977) refers to this “as grasping the structure of a subject.” he further argues that “teaching specific topics or skills without making clear their context in the broader fundamental structure of a field of knowledge is uneconomical in several deep senses” (pg. 31), in that: “such teaching makes it exceedingly difficult for a student to generalize from what he has learned to what he will encounter later…the best way to create interest in a subject is to render it worth knowing, which means to make the knowledge gained usable in one’s thinking beyond the situation in which the learning have occurred. third, knowledge one has acquired without sufficient structure to tie it together is knowledge that is likely to be forgotten. an unconnected set of facts has a pitiably short half-life in memory.” similarly, theories cannot be taught in a vacuum; prospective teachers must understand the relationship between the ideas they are taught and the applications they will encounter. one way to develop this skill is to arrange for this connection to be made in the context of their “lived” realities. such learning-in-context will provide prospective teachers with the opportunity of questioning what they do and think (brookfield, 1995). it is during this process of inquiry, thinking about their practice, that teacher transformation occurs. another valuable theoretical approach which informs this process is the constructivist approach to learning, which derives its name and its power from the belief that knowledge is best constructed when the learner actively interacts with the environment and, hence, constructs meaning from that experience (hausfather, 2001). similarly, hall-quest asserts in the editorial foreword to dewey’s experience and education, that “sound educational experience involves, above all, continuity and interaction between the learner and what is learned” (dewey, 1938, hughes, j.a. journal of scholarship and teaching, vol. 6, no. 1, august 2006. 112 p.10). the end result of this nexus is the teacher’s ability to transfer teacher-knowledge to effective practice. in sum, the theories that drive this project are those of constructivism, the theory of integration, and the rigorous application of critical reflective thinking. iii. the role of fieldwork in teacher preparation field experiences are significant means through which to develop prospective teachers’ understanding of the why, what, and how of teaching and learning. however, simple placement of student teachers in the field does not automatically result in a valuable experience for the teacher candidate (zeichner, 1990). after all, not “all experiences are genuinely or equally educative” (dewey, 1938, p. 25). dewey (1938) asserts that “it is not enough to insist upon the necessity of experience, nor even of activity in experience [emphasis mine]. everything depends upon the quality of the experience which is had” (dewey, 1938, pg. 27). although, there is little doubt among teacher educators about the role of fieldwork in preparing better teachers, “there is persistent concern that such experiences do not reach their full potential value” (bowman & mccormick, 2000, p.256). several circumstances may account for this: traditional structures of student teaching (zeichner, 2002), which are often developed out of “convenience or tradition” (guyton & mcintyre, 1990, p 517) rather than innovative practices; limited resources to carry out field work (goodlad, 1990; darling hammond, 1999); the individualized nature of fieldwork (goodlad, 1994); the quality of the field placement (laboskey & richert, 2002); and a traditional approach to university supervision (bowman & mccormick, 2000). empirical evidence and current thinking suggest that many time-honored and time-worn field experience practices, such as those referenced earlier, need to be either refurbished or abandoned all together. one viable solution to teacher-educator lack of currency lies in the periodic return of teacher-educators to the public school environment with the intent of gaining practical experience. a return to the living laboratory of the k-12 classroom will allow teacher educators to test the theories and concepts they teach as well as to examine their own teaching practices while making pertinent and necessary revisions and adjustments in their practice. iv. approach from several years of informal conversations with teacher candidates during office hours, class discussions, brief encounters with them in the hallways, and reading their observation journals, i have concluded that for teacher preparation to be effective it must take place in the context in which it occurs, the school environment. over the course of several semesters, i utilized various approaches to field work, such as focused observation activities with required critical reflective inquiry of their observation and case study development and analysis of issues of interest to the prospective teacher. although these exploratory attempts at “teaching in context” yielded some satisfying results, i became increasingly restless with my methodology. searching for something new and innovative, i assumed the role of a 9th grade algebra-i teacher, sharing teaching responsibilities with the teacher of record. this pilot project was conducted at a high school located in a predominantly hispanic community in southern california. the high school population consisted of approximately 95% hispanic with asians, caucasians, and african americans constituting the remaining 5%. i selected this school because of its cultural location (a high percentage of minority students, hughes, j.a. journal of scholarship and teaching, vol. 6, no. 1, august 2006. 113 mexican and south-american, of low socio-economic status) as well as the current commitment between the university and the school district.2 four students fully participated in this pilot project. because the majority of my students were themselves classroom teachers (on emergency permit) or holding jobs outside of the field of education, the remainder of my college class members participated in a somewhat more limited way. the project full-participants observed my classroom experience for a minimum of two class periods twice weekly. in addition, a thirty to forty minute critical and reflective debriefing was conducted immediately after the observation. if time constraints prevented this from happening, the requisite discussion session took place in my university office later that same day. during these discussion sessions, participants were encouraged to speak freely about my teaching strategies and my interaction with the algebra students, as well as the general classroom atmosphere; they provided insights into the events of the period, explored various perspectives on the relationship between classroom practice and education theory, highlighted various concepts already covered in class discussions at the university, interpreted and analyzed particular incidents and offered suggestions for solutions; further, they discussed with me the reasons i had handled an issue or situation in the way that i had. in addition to these discussions, all participants maintained an observation journal in which they reflected on the various concerns and issues they encountered, some entries of which are noted below: it is important to note here that the 9th graders received a combination of before and/or after school tutoring; tutoring from the student teachers; exam review activities; and, homework to practice algebraic concepts. a. student #1 first reflective entry: how do i adjust piaget’s cognitive theories to meet the different learning stylesin my classroom? it does not seem as if it applies. especially, when i have to interpret the curriculum standards in order to teach it to my students who have such grave disparities in their learning levels. second reflective entry: i also find that learning the concept takes so much time and i don’t have the luxury to study it in a way that would help me put it into practice due to the dayto-day demands of my classroom environment. i know that the things that i learn in this course are important and informs my teaching in some way. but, i don’t see it, yet. i know that students are at different stages cognitively, intellectually, socially, and so forth. that’s evident! i see it demonstrated in my students everyday. so i don’t need a theorist to tell me this. my problem is how these seemingly relevant concepts, theories, help me motive, j--e. how does it help me teach my academically diverse students complex curriculum standards while being cognizant of their overall development – social, emotional, behavioral? third reflective entry: i find myself not wanting to learn these concepts because i don’t know yet howto readily apply it to my class. i hear my peers speak of how they have applied cooperative learning in their classroom and how successful it was. but, i am afraid i don’t share those experiences. dr. hughes – struggled to illustrate this concept in class last night and there were no videos except from her high school class regarding this issue. so i was really stuck. 2 human subjects clearance was not needed for this research. hughes, j.a. journal of scholarship and teaching, vol. 6, no. 1, august 2006. 114 b. student #2 reflective entry: i think i got it! the theory of motivation. i so want my students to want to learn. then it occurred to me that (as i watched dr. hughes struggle with how best to motivate a----a and some of the other low motivated students) that she missed something important – what interest them). she talked about it in class but she didn’t do it. i don’t know why! i will ask her later. reflective entry – a week later: this week dr. hughes planned to review for the upcoming exam. s----n and i have worked hard all weekend putting together a jeopardy algebra game. we weren’t sure if it would work but dr. hughes let us try anyway. it worked well! i was so relieved. the students were excited and very much into the game. i thought it was because the method was fun, innovative and so forth. i was worried about no real reward (e.g. no homework, etc.) except for the points they received for the correct answers in each category. much to our surprise (dr. hughes, too). the students were really into the activity. they didn’t seem to mind that all they would have at the end of the period was just points for first, second, or third place. i am not sure if the students learned much in preparation for the exam. but, they indicated that they liked the game, that it was fun. dr. hughes informed us the next week that a majority of the students had passed the exam… we had hope that the game would motivate the students to go home and study as a result of the positive feelings of giving the right answer and being praised by their team-mates and teacher. these sessions (as well as several days of my teaching) were videotaped; these tapes were then used in my university class as a teaching tool to enhance the curriculum and to demonstrate best or worst practices. this procedure provided a rich foundation for discussion by both full and limited participants. the limited participants benefited from watching the video of my teaching as well as listening to the discussions between me and their student colleagues. the retrospective discussions allowed all students the opportunity to form sound and valid arguments, to make explicit their practical knowledge, and to apply this new knowledge to current educational issues. throughout the entire experience, i provided extensive feedback to guide interpretations and encourage critical reflective inquiry. it is important to note here that the 9th graders received a combination of before and/or after school tutoring; tutoring from the student teachers; exam review activities; and, homework to practice algebraic concepts. v. reflections i found from my experience that when both the teacher-educator and prospective teachers are actively involved in both the college class and the k-12 environment, the ability for everyone to teach and learn simultaneously was enhanced. furthermore, this process made it possible for me and my prospective teachers to identify and examine the convergence of theory and practice. the goal of exposing students to the process of teaching and learning through such organized field experience was met. i fervently believe that a process similar to that which i have described has the potential to augment the quality and merits of the field observation requirement. as such it contributes immeasurably to the overall ability of the students to teach effectively in the k-12 environment. a summative entry from my own journal follows: hughes, j.a. journal of scholarship and teaching, vol. 6, no. 1, august 2006. 115 what i learned as a teacher educator. as an educator i learned that it wasn’t just about teaching the theories of motivation, piaget’s theories of cognitive development, or vygostky’s zone of proximal development in my educational psychology course. rather it was about teaching motivation as it came to life in my algebra class instead of in abstract form. i was challenged with how best to motivate my 9th grade students. i tried no homework if they completed their worksheet, etc. but, i was faced with the fact that most of my students did not understand the concepts. so with the help of the collaborating teacher we divided the class into several smaller groups and assigned them work that met their skill and comprehension level. this approach worked very well. the next semester i used the experience to exemplify how piaget’s theory of cognitive development might inform high school teaching. my college students remarked how very beneficial this was. of course, i did not do away with teaching the stages of the theory of piaget’s cognitive development, etc. what this experience did for me was to transform how i teach and how i constructed my course curriculum. theoretical concepts were paired with some real life classroom experience i had encountered. my text selection was based on how well the authors presented complex theoretical concepts (that is their approach to illustrating these complex theories in the hope of increasing comprehension, and possible later transfer of this understanding). as a result, i looked for the level of language used to introduce complex concepts, case studies, activities that would facilitate understanding and skill acquisition. vi. suggestions for change the serious nature of the process of preparing students to meet the demands of their profession requires a critical examination of the professional requirements. one such requirement is the familiarity with the changing nature of the profession. meeting this requirement will likely provide insights on how to interpret and integrate professional standards and expectations into program course work and requirements. educators can meet this requirement in one of two ways. the first is, a return to the environment of professional practice for a semester or a year-long reintroduction to the field. the educator may serve as a social worker, counselor, or teacher performing duties similar to what their current students would do once in the field. the second approach to gaining familiarity with the current practices in one’s field is to allow former students to return to the college classroom to share their lived experiences and current practices as it relates to theory. the above approaches can be combined or employed separately. i prefer the approach wherein the educator returns to the field. implementation of one or both of the above approaches or a variation will no doubt demonstrate how serious we are in bridging the divide between what our students are learning in their college classrooms and what they do in their jobs. as a result of my experiences, i am now even more convinced that faculty currency is critical to the development of well-grounded students who can effectively respond to the demands and challenges of their profession. hughes, j.a. journal of scholarship and teaching, vol. 6, no. 1, august 2006. 116 references adams, n., shea, c., liston, d., & deever, b. (1998). learning to teach: a critical approach to field experiences. new jersey: lawrence erlbaum. baker, b. (1997). “anthropology and teacher preparation: some possibilities and precautions.” queensland journal of educational research, 13(2), 41-58. bullough, r. v., jr. young, j., erickson, l., birrell, j.r., clark, d.c.’ egan, m.w., berrie, c. f., hales, v. & smith, g. c., (2002). “rethinking field experiences partnership teaching versus single-placement teaching.” journal of teacher education, 53(1), 68-80. bowman, c., & mccormick, s. (2000). “comparison of peer coaching versus traditional supervision effects.” journal of educational research, 93(4), 256-261. britzman, d. (1991). practice makes practice. albany, ny: state university of new york press. brookfield, s. d. (1995). becoming a critically reflective teacher. san francisco: jossey-bass. brunner, j. (1977). the process of education. massachusetts: harvard university press darling-hammond, l. (1999). “educating teachers for the next century: rethinking practice and policy.” in g. griffin (ed.), the education of teachers. chicago: university of chicago press. dewey, j. (1933). how we think. new york: houghton mifflin. dewey, j. (1938). experience and education. new york: macmillan. feiman-nemser, s. & remillard, j. (1996). “perspectives on learning to teach.” in f. murray (ed.), the teacher educator’s handbook (pp.63-91). san francisco: jossey-bass. goodlad, j. (1990). teachers for our nation’s schools. san francisco: jossey-bass. goodlad, j. (1994). educational renewal better teachers, better schools. san francisco: josseybass. guyton, e., & mcintyre, d. j. (1990). “student teaching and school experiences.” in w.r. houston (ed.), handbook of research on teacher education. new york: macmillan. hausfather, s. (2001). where’s the content? “the role of content in constructivist teacher education.” educational horizons, 80(1), 15-19. knowles, j. & cole, a. (1996). “developing practice through field experiences. in f. murray (ed.), the teacher educator’s handbook (pp. 648-688) san francisco: jossey-bass. laboskey, v.k., & richert, a. e. (2002).” identifying good student teaching placements: a programmatic perspective.” teacher education quarterly, 29(2), 7-34. hughes, j.a. journal of scholarship and teaching, vol. 6, no. 1, august 2006. 117 perry, c. m. & power, b. m. (2004). “finding the truths in teacher preparation field experiences.” teacher education quarterly, 31(2), 125 –136. stuart, c., & thurlow, d. (2000). “making it their own: preservice teachers’ experiences, beliefs and classroom practices.” journal of teaching education, 51(2), 113-121. wideen, m., mayer-smith, j. & moon, b. (1998). “a critical analysis of the research on learning to teach: making the case for an ecological perspective on inquiry.” review of educational research, 68(2), 130-178. zeichner, k. (1990). “changing directions in the practicum: looking ahead to the 1990s.” journal of education for teaching, 16(2), 105-132. zeichner, k. (2002). “beyond traditional structures of student teaching.” teacher education quarterly, 29(2), 59-64. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006, pp. 118 126. teacher candidates’ conceptual understanding of conceptual learning: from theory to practice ellen a. sigler and julie saam1 abstract: education researchers suggest that teacher education candidates be taught that meaningful learning is essential and that conceptual understanding be infused into all lessons. however, many teacher candidates are unable to successfully develop conceptual level lesson plans and some are unable to differentiate between skills and concepts. the purpose of this study was to enhance the conceptual understanding of teacher candidates and in turn help to develop their understanding of the learning process. this was accomplished by presenting the candidates with ideas concerning conceptual learning in a way that facilitated more meaningful learning and higher-level thinking. from survey analysis and evaluation of course projects, this method seems to achieve the goal of breaking down the cognitive barriers and allowing teacher candidates to conceptualize the difference between instruction that focuses on skills and instruction that focuses on conceptual understanding. i. introduction the notion that skills instruction is so essential to students’ ability to achieve (especially in the field of mathematics) has been prevalent in the u.s. for many years (tournaki, 2003). drill and practice ideologies have taken the forefront in many instructional plans due to the perceived efficiency and effectiveness of the processes (swanson & sachse-lee, 2000). especially with the onslaught of high stakes testing in the classroom, teachers often feel that direct instruction of skills is the only way to ensure that students are to achieve academically (case, 2002; gulek, 2003). the research, however, downplays the effectiveness of rote learning and suggests a more meaningful, conceptual understanding in order to promote problem solving and increase knowledge generalization. in addition, teachers who (as students) learned using drill and practice may find it difficult to break free of this more rigid paradigm. the purpose of this study was to enhance the conceptual understanding of teacher education students (teacher candidates) and in turn help to develop their understanding of the learning process. the intent was that the candidates develop insight into the critical difference between skills and concepts and the notion that conceptual understanding leads to richness and flexibility in problem solving both within and outside the classroom. ii. background concept learning focuses on the aspect of category formation and the use of concepts to interpret experiences and solve problems (ormrod, 2001). a variety of theories have been developed proposing processes by which concepts such as schemas, exemplars, prototypes, etc. 1 ellen a. sigler, division of education, indiana university kokomo; julie saam, division of education, indiana university kokomo. correspondence concerning this article should be addressed to ellen a. sigler, division of education, indiana university kokomo, 2300 south washington street, kokomo, indiana 46904. e-mail: elsigler@iuk.edu sigler, e.a., and saam, j. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 119 are formed (anderson, 2000). information processing theory embraces the ideas of conceptual understanding because it is the essence of meaningful learning and is the key to higher-level thinking. additionally, in education, proponents of constructivism weigh heavily on the need for conceptual learning since it is the basis of all knowledge constructed by the learner (mestre, 2002). in cognitive psychology, a concept is a group or category of similar events or objects (ormrod, 2003). the ability to form concepts allows an individual to make sense of the vast amount of information processed everyday. students taught to develop a conceptual understanding of various domains will be more proficient at problem solving, abstract reasoning, generalizing their knowledge to new situations and more likely to make connections to related information (ormrod, 1999). the converse of conceptual learning is rote memorization and drill and practice techniques (snowman & biehler, 2003) which tend to emphasize a skill acquisition approach. “skill learning” is the term used to define learning a set of steps or memorization of a procedure in order to accomplish a task (mayer, 2002). though there is acceptance of drill and practice methods for some domains, the consensus of many professionals in the k-12 classroom and at the university level is that learning must entail conceptual understanding for it to be meaningful and for it to facilitate problem solving (stingler & stevenson, 1994; fuys & liebov, 1997; carlson, 1995). convincing teacher candidates, that meaningful learning is vital and that conceptual understanding is essential for all lessons is part of almost every educational psychology text. the difficulty lies, however, in training teacher candidates to develop lessons that promote conceptual understanding versus ones that promote purely routine skills training. iii. specific programmatic concerns in our teacher education program, when teacher candidates were asked to develop conceptual lessons they consistently produced skill level lessons, even though the need for conceptual learning was part of their college curriculum. in this particular situation, all teacher candidates received training in conceptual learning early on in their program during the educational psychology course that preceded all teaching methods courses. then, while in the methods classes, teacher candidates were expected to apply this information in the development of a variety of lesson plans, specifically in the areas of math and science. for the most part, they were unable to do so. not only were the teacher candidates unable to develop conceptual level lesson plans but more simply, they were unable to differentiate between skills and concepts. it became our task to determine a reason for this lack of understanding, and then to develop a course of study to remedy this situation. it was immediately apparent that information concerning the differences between concept learning and skills acquisition was not fully understood by these candidates. therefore, it was necessary to determine what lesson changes should be made to facilitate better understanding of these essential points. iv. specific conceptual concerns while delving into the specific college level lessons it became clear that one of the main problems concerning concept formation was that the lessons themselves were not taught conceptually. by its very nature, the “terms and definitions” format presented in the educational psychology course lead candidates to learn material in more of a rote fashion, despite the sigler, e.a., and saam, j. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 120 instructor’s efforts to initiate meaningful learning. in many instances throughout the course, techniques such as real life examples, discovery learning and imagery helped to develop the candidate’s understanding of the concepts at hand. however, when specifically dealing with the concept of concepts candidates regressed to old habits of rote memorization. that is, they were able to define the term “concept”, list examples and even recall theories, but when it came to higher level thinking skills; application, analysis, synthesis, etc. they were unable to utilize previous learning. a second observation regarding the problem of concept learning was the candidates’ inability to differentiate a concept from a skill. this seems more prevalent when working with less complex material. for example, when developing a lesson plan on a more sophisticated topic such as gravity, the candidates seemed to understand the concept of gravity must be demonstrated and understood by children before moving on to higher-level aspects of this scientific idea. however, when dealing with more simplistic notions such as addition, the concept of “what is addition?” did not occur to candidates as an essential part of the lesson. candidates consistently moved directly into the manipulation of symbols without regard for the child’s level of understanding of the concept of addition. apparently, the concept of addition was not perceived as novel most likely due to the candidate’s familiarity with the concept. as students often complain about professors teaching at too high a level, the candidates did not perceive a similar corresponding difference between their conceptual knowledge and that of the child in their classroom. based on this information it seems imperative to present teacher candidates with ideas concerning conceptual learning in a way that facilitates more meaningful learning and higherlevel thinking. additionally, it is important to ensure that candidates identify the concept to be taught and develop lessons for children that clearly center on concept formation. in this study, our goal was to determine a way to integrate concept learning into the teacher education curriculum in a way that will facilitate more meaningful learning and assist in application of conceptual level lesson planning. to accomplish this, we developed a three-part intervention. in the first part, we created a “mock” elementary school lesson, acting out a skillsapproach arithmetic lesson and utilizing a numeration system foreign to the candidates. for the second part, we developed a college-level lesson outlining the properties of conceptual understanding. in the last part, we assisted the candidates in applying the conceptual ideas to mathematical methodologies. v. methodology this was an experimental program that was carefully developed, presented and reviewed at various stages. this study was developed in order to “acquire in-depth and intimate information about a smaller group of persons <and> …to learn about how and why people behave, think, and make meaning as they do” (ambert, adler, adler & 1995, p.880). however, this study would not strictly be classified as either quantitative or qualitative in nature. brown (1992) referred to such studies as design experiments, which center more specifically on improvement of instruction as opposed to hard empirical evaluations. according to kember (2003), designs such as this one are more likely to yield useful results if triangulation of data occurs. triangulation is a method for utilizing multiple sources of data in order to “establish claims beyond a reasonable doubt” (kember, 2003, p. 99). the data utilized in this study came from survey responses obtained immediately after the intervention and projects developed by the students at a later time. sigler, e.a., and saam, j. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 121 a. subjects candidates from two teacher preparation classes were combined to participate in the experimental class. fourteen of the candidates were from the 200 level educational psychology class, and the 32 were from the 300 level mathematics methods class. there were several reasons for combining the classes. first, as it was indicated earlier, candidates learn the basic framework for concept learning in the 200 level class, yet they must apply these theories in the 300 level class. it was unclear at the time which group would benefit more from this instruction, so both groups were presented the material simultaneously. b. procedure we felt it essential that candidates, in order to have a more meaningful learning experience, must recall the experience of learning novel information. as stated earlier, candidates found it difficult to “think like children” and approach a familiar topic as a novice learner. it was also necessary that the candidates experience the information and come to their own conclusions without being told specifically what they should be learning. in other words, in order for the information to be meaningful, the candidates needed to construct their own understanding of what it means to be a novice learner. this would take more than simply giving examples of concept level lessons or identifying developmentally appropriate practice. the candidates needed to discover for themselves what a concept is and what it is not. lastly, it was vital the concept taught be simplistic enough that when the lesson was complete the candidates could see the parallels between their concept learning and the concept learning desired by schoolchildren. if the candidates spent too much effort attempting to learn a sophisticated concept, they might possibly miss the point of the entire lesson. in order to do this we developed a basic, base-ten numeration system that consisted of unfamiliar symbols instead of the well-known arabic system. we used the wingding font (microsoft, 2000) and simply replaced each number in the base-ten system with a symbol, as shown in table 1. table 1. base ten symbolic numeration system. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 then we developed a few demonstration problems requiring addition and regrouping. we scripted the class lesson intentionally to model a less than adequate classroom lesson for elementary school children. an example of this script is: “we are now going to attempt some two digit addition problems. does everyone remember how to regroup {short pause}, good. now, let’s try one together, dot circle plus dot diamond equals square dot circle, alright, good, and when we regroup it leads to square plus square plus sigler, e.a., and saam, j. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 122 dot square… which of course equals…anyone {short pause}, good, dot circle”. the instructor presenting the mathematics lesson did so as a confederate. that is, in no way did she indicate the lesson was scripted, and the information presented was anything less than authentic. after completing the sample lesson, the second instructor proceeded with the standard educational psychology lecture that normally accompanies a unit on conceptual learning. this included definitions and specific examples of the presentation of concepts in a classroom, and included such terms as feature lists, exemplars, prototypes, and schemas. the specific text used was ormond (2001). after the presentation, the students were then “debriefed”. we explained that their frustration with the wingding system is similar to the frustration school children have when teachers teach only the skill of addition and not the concept of addition. to illustrate, we showed table 1 to the candidates and explained the construction of the wingding system. we also practiced a few more addition problems. within minutes, the candidates were as fluent in wingdings as their instructor. we anticipated candidates would understand the connection between their frustration and their schoolchildren; however, they would still need guidance in applying the conceptual learning theories to instructional planning. we then gave an additional explanation of sample concepts specifically with simple classroom methodologies in mind, with results as shown in table 2. c. analysis candidates completed a short likert-scale questionnaire on various aspects of the class. several of the questions were directed to the overall quality of the presentation. one question specifically asked the candidates about their perception of the need for such information as a teacher. several questions (targeted the candidates in the methods class) asked candidates to rate their own understanding of concepts in the classroom and their ability to apply the information learned in the class. table 2 . sample lecture notes from class debriefing. exemplar view • exemplars are actual memories of specific members of a category, and we use these to compare to an item in question to see if that belongs to that category • intersections and roads seen everyday as we travel to school schema view • we recognize some objects simply by our own experiences, regardless of their group membership or their attributes. • using measuring cups and spoons to “experience” fractions. candidates were also invited to write in comments about this class at the end of the evaluation. these comments were collated and analyzed. in addition, the methods’ instructor sigler, e.a., and saam, j. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 123 evaluated the lesson plans and mathematics education methodological projects created by the methods candidates. vi. results and discussion a. general results for the question “this class helped me develop a skill i will need as a professional”, a chi square analysis was calculated for both classes combined. there were a total of 46 candidates. the χ2 obtained was significant at the α = .01 level. all students indicated agreement or strong agreement with this statement. for the upper-level methods candidates, 32 candidates participated in the class and completed the survey. for the questions: i remember studying concept learning in p250. i understood concept learning and its application prior to this class today. i found this class enriched my knowledge of concept learning compared to what was presented in p250. candidates indicated either agreement or strong agreement for all three questions using a five point likert scale. again, response were significant at the α = .01. some of the candidates’ comments included: “great review and reminder for actual use in the classroom. it increased application knowledge.” “very helpful class, helped apply and put into perspective previous info learned.” “it helped give me more concrete examples of concept learning.” “this was a great experience – to see theory apply realistically to what i hope and will strive to do as a teacher.” b. specific results during the remainder of the semester, the methods instructor evaluated candidates’ work for examples of conceptual understanding and applications of conceptual learning. examples were prevalent in lesson plans, presentations, and reflective papers. after viewing a model of the constructivism approach to teaching in the combined class, candidates seemed to understand that in order to teach schoolchildren concepts they needed to also allow these children to discover the concepts. one candidate shared an instructional strategy on how to teach the concept of factors. he asked children to use lincoln logs (stackable rods for building structures) and group them into even groups. for example, using ten logs, a child can group those logs into 1 group of 10, 10 groups of 1, 2 groups of 5, or 5 groups of 2. these group numbers are the total number’s factors. the number 10 has factors of 1, 2, 5, and 10. in another discovery-approach lesson plan a candidate used the process of grouping objects to teach the concept of division. the instructions for this lesson are reported below: give each student manipulatives and a compartmentalized craft box or divided cardboard box. the manipulatives should be similar in some attributes and different in others (i.e. m&m’s of various colors, beads with different colors/shapes, cars and trucks, blocks with numbers/letters/colors, cards, or fruit). let the students then play with/ manipulate the items. walk around the room and sigler, e.a., and saam, j. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 124 observe. ask questions, “why did you group these this way?” and “what is the same/different about these?” “how many do you have in each group?” and “how many did your start with?” “how did your separate the blocks, toys, etc.?” bring the class together with a discussion about how they sorted/divided up their toys. the lesson ended with the concept presentation that sorting is division. other candidates’ lesson plans included hands-on activities such as using measuring cups to help students understand the concept of equivalent fractions and using paint rollers and paint to understand the concept of a line. one candidate’s reflection indicated a complete understanding of the importance of using learning theories, particularly concept learning to enrich instructional planning. “percents are so widely used in day to day situations, it is imperative that students develop a good understanding. unfortunately, studies show that students and even adults do not understand the basics of percents. this tells me as a future educator that more time needs to be spent developing an understanding of percents before jumping ahead to calculating percents.” these lessons overall were far advanced conceptually, compared to projects from previous semesters where the majority of the lessons, presentations and projects consisted of skill-level lessons. candidates in previous classes would develop lessons that consisted of showing students how to add 2-digit numbers, how to multiply fractions, or how to do long division, using only symbol manipulation for demonstration purposes. the students were required only to duplicate the pattern accurately. these lessons lacked conceptual learning and discovery, while the lessons produced by the conceptually trained students were rich in discovery, conceptualization and number sense. vi. discussion the purpose of this study was to determine a way to integrate concept learning into the teacher education curriculum in a way that will facilitate more meaningful learning and assist in application of conceptual level lesson planning. it seems apparent from the content of the methods candidates’ projects after the conceptual learning class, candidates glean conceptual understanding necessary to teach children utilizing the constructivist approach. it also appears from the survey analysis, the method used to teach these lessons effectively achieved the goal of breaking down the cognitive barriers and allowing teacher candidates to conceptualize the difference between instruction that focuses on skills and one that focuses on conceptual understanding. there are many sources that cite the importance of concept learning and others that demonstrate examples of concept-level lessons for schoolchildren (ormrod, 2001, 2003). however, there is little research concerning effective methods for instructing teacher candidates on how to develop concept level lessons. it is essential in the field of education that teachers have the skill of analyzing the methods used to teach children, and assess the goal of these methods to ensure meaningful learning is occurring (mayer, 2002; snowman & biehler, 2003). in future studies, a wide range of cognitive domains (outside of mathematics) could be evaluated, ensuring that teacher candidates generalize the information concerning conceptual knowledge beyond the one presented in this study. additionally, a further analysis about the sigler, e.a., and saam, j. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 125 timeliness of teaching concept learning in the pre-professional teacher education courses should be investigated. vii. conclusion it takes a leap of faith for many teachers and teacher candidates to leave the security of the, “teach to the test” philosophy. for so long, teachers’ understanding of learning has encompassed skills instruction, drill and repetition, that they no longer have the faith children can be taught to think. “many studies document a preoccupation with transmission of information and rote application of ‘skills’, and a paucity of class time devoted to promoting students’ ability to think critically” (case, 2002, p.11). this study demonstrated aspects of the process of teaching conceptual understanding in the classroom. it demonstrates how teacher candidates, when given a lesson that focuses on the development of their conceptual understanding, are able to see ways to incorporate conceptual understanding in their own lesson plans. with equal importance, however, this study demonstrated the need for the discovery approach, a conceptually based lesson for the teacher candidates themselves. without the epiphany and the realization of their own roots of conceptual learning, the candidates slip too easily into a purely skills approach. with this method candidates were able to gain the realization that they know what they know, not from exacting drill and practice, but through constructing their own knowledge based on meaningful learning and conceptual understanding. references ambert, a. & adler, p.a. (1995). understanding and evaluating qualitative research. journal of marriage & family, 57(4). anderson, j.r. (2000). cognitive psychology and its implications (5th ed.). new york: worth publishers. brown, a. (1992). design experiments: theoretical and methodological challenges in creating complex interventions in classroom settings. the journal of the learning sciences, 2(2), 141178. carlson, a.d. (1995). letters, numbers, shapes and colors. school library journal, 41(5), 3034. case, r. (2002). partnering to promote critical thinking. canadian statistics in the classroom, 22(1), 11-12. fuys, d.j. & liebov, a.k. (1997). concept learning in geometry. teaching children mathematics, 3(5), 248-252. gulek, c. (2003). preparing for high stakes testing. theory into practice, 42(1), 42-50. kember, d. (2003). to control or not to control: the question of whether experimental designs are appropriate for evaluation teaching innovations in higher education. assessment and evaluation in higher education, 28(1), 89-101. sigler, e.a., and saam, j. journal of scholarship of teaching and learning, vol. 6, no. 1, august 2006. 126 mayer, r.e. (2002). the promise of educational psychology. volume ii: teaching for meaningful learning. new jersey: merrill-prentice hall. mestre, j.p (2002). cognitive aspects of learning and teaching science. in s.j. fitzsimmons & l.c. kerpelman (eds.) teacher enhancement for elementary and secondary science and mathematics: status, issues, and problems. washington dc: national science foundation. ormrod, j.e. (2003). educational psychology: developing learners. (4th ed.) new jersey: merrill-prentice hall. ormrod, j.e. (1999) human learning. new jersey: merrill-prentice hall. snowman, j. & biehler, r. (2003). psychology applied to teaching. boston: houghton mifflin. stigler, j.w. & stevenson, h.w. (2001). how asian teachers polish each lesson to perfection. in m. gauvin & m. cole (eds.) readings on the development of children (3rd ed.). new york: worth publishers. swanson, h.l. & sachse-lee, c. (2000). a meta-analysis of single-subject design intervention research for students with ld. journal of learning disabilities, 33(2), 114-136. tournaki, n. (2003). the differential effects of teaching addition though strategy instruction versus drill and practice to students with and without learning disabilities. journal of learning disabilities, 36(5), 449-458. 127 josotl mission the journal of scholarship of teaching and learning seeks to encourage all instructors to engage in the discussion of the scholarship of teaching and learning (sotl), and to become involved in the sharing of knowledge and learning about the teaching-learning process. any report about an investigation into what works (or doesn't work) for a particular teaching-learning context will be considered for publication. those submissions that include reflective commentary about the result of the investigation will be considered of greater value to our readership and more appealing for publication. the journal shall also consider submissions that offer opinion, thoughtful reflection, commentary, or theoretical ideas related to sotl. while we welcome submissions from people working in any discipline, we most strongly encourage submissions that provide useful insights and a significant potential impact across disciplines. papers narrowly focused on a single discipline are unlikely to be accepted; authors of such papers should probably target discipline-specific journals. the journal is intended to provide support for those already engaged in exploring sotl, as well as encouraging those new to the topic to become involved. the support will come in a variety of forms: • shared results as with any journal,the readership shall gain by learning through the experiences of others published in josotl • editorial process submissions will be reviewed by at least two reviewers using a double-blind process. feedback gained from the reviews, as well as comments from the editorial board, should help guide authors as they continue work in the field • examples published work provides models for those considering involvement in this field • community the readership will have a means for interacting with editorial staff, authors, and each other in a convenient and timely manner. the community of scholars interested in a shared field should be greatly enhanced by the presence of the journal on the internet. classroom action research, descriptive or qualitative research, quantitative studies, case studies, and other forms of research addressing sotl are all acceptable for consideration in josotl. brief reports of projects are welcome and will be considered for publication in the interest of fostering community comment on the work. the editorial board will evaluate the general quality of the work, the value of the reflective content included by the author, and the relative appeal of the report for the readership of the journal. sotl is an evolving topic for study. there is continued need to simply think about it, suggest theory regarding it, or comment on existing notions about it. therefore, josotl will also accept submissions that do not report on completed research or experimentation. these submissions will also undergo a double-blind review process and editorial evaluation 128 submissions please see the mission statement for a discussion of the editorial philosophy for josotl. authors are encouraged to submit work in the following categories: traditional research reports reports on classroom action research essays on sotl teaching portfolios despite their differences, all of these types of submissions should include the author's expression of the implications their work has for the teaching-learning process. this reflective critique is central to our mission in furthering understanding of sotl. authors are encouraged to review the guidelines for reviewers in order to understand how their submissions will be evaluated. authors are strongly encouraged to study the reviewer's rubric that reviewers shall apply in evaluating their submitted work. authors are encouraged to submit papers, teaching dossiers, and other potential publications to donald a. coffin. submissions must be prepared in an electronic format using microsoft word. submissions are preferred as files attached to an e-mail. however, 3.5" disks sent through snail mail are also acceptable. josotl will not return disks to authors. research papers and essays should be according to the josotl style sheet. there is no page limit, but authors should consider brevity a virtue as they prepare their submissions. authors are expected to include proper referencing for their sources, especially urls for web sites that might contain material of interest to our readership. authors are encouraged to provide a few key hyperlinks that might offer readers additional information concerning their topic. every submission must include a title page with the following information: • title • for each author: o name and affliation o postal address o email address o telephone number • a brief (75-100 word) abstract • a brief (more than two, less than eight) list of keywords related to your work. do not include any author's name on any page other than the title page. please number the pages. the abstract will be included in the publication's title page, and the keywords are important for proper archiving of published articles. we also welcome the submission of electronic teaching portfolios for josotl. portfolios should be posted on the author's web site, and the url should be forwarded to the managing editor. reviewers will be asked to view the site and provide evaluation. if 129 accepted, josotl will post a hyperlink from the journal to the author's site. please note that it is impossible to provide double blind review for portfolios--although the reviewers' identities will not be revealed, the author's identity cannot be removed from an entire portfolio. paper submissions accepted for publication will require authors to sign a copyright agreement with the trustees of indiana university. authors must be prepared to sign this agreement upon acceptance of their work for publication in the journal. for more information regarding copyright, please see the statement of copyright and terms of use. if you have any questions regarding the submission policy, please e-mail or call a member of the editorial board. 130 editorial board all members of the josotl editorial board are affiliated with facet, the faculty colloquium on excellence in teaching, at indiana university. donald coffin associate professor of economics dcoffin@iun.edu http://www.iun.edu/~bnwdac/ school of business & economics indiana university northwest, gary, in 219.980.6913 eugenia fernandez associate professor of computer & information technology efernand@iupui.edu http://www.engr.iupui.edu/~efernand purdue school of engineering and technology indiana university purdue university indianapolis 317.274.6794 joan e. lafuze professor of biology jlafuze@indiana.edu instructional programs indiana university east, richmond, in 765.973.8246 david j. malik associate executive vice president, indiana university facet director chancellor's professor of chemistry dmalik@iupui.edu http://chem.iupui.edu/faculty/malik.html school of science indiana university purdue university indianapolis 317.274.6884 julie saam assistant professor of secondary science education jsaam@iuk.edu division of education indiana university kokomo 765.455.9302 ellen a. sigler assistant professor of educational psyhcology elsigler@iuk.edu division of education indiana university kokomo 765.455.9419 131 style sheet for the journal of scholarship of teaching and learning donald a. coffin1 abstract: this paper provides the style sheet for the journal of scholarship of teaching and learning. manuscripts submitted for publication should adhere to these guidelines; finished manuscripts must be in this style for final acceptance. i. general guidelines for the manuscript. the final manuscript should be prepared in 12-point, times new roman, and single spaced. submissions should be double-spaced. margins should be 1” top and bottom, 1” left and 1” right. the text should be left-flush (ragged right). the title (in 16 pt. bold) and author’s name (in 12 pt. bold) should be at the top of the first page (the author’s name should be followed by a footnote that provides the author’s institutional affiliation and any acknowledgements), followed by an abstract. the abstract should be indented 1” left and right from the margins, and should be in italics. paragraphs should have a 0.5” first line indent. we will renumber pages of final manuscript. authors should number their pages so that they appear in the bottom right of the page. we will also insert a header on the first page of the article, as above. authors need not insert headers/footers. references should be incorporated in the text as authors name and date of publication [coffin (1993)], with a reference section at the end of the manuscript (see below for the desired format for the references). footnotes should incorporate material that is relevant, but not in the main text. ii. section and sub-section headings. a. major sections. major section headings should be flush-left, bold-faced, and roman-numeral numbered. major section headings should have a one-line space before and after. b. sub-sections. sub-section headings should also be flush-left, in italics, and alpha-numbered. subsection headings should have a one-line space before and after. sub-sub-sections should appear at the beginning of a paragraph (i.e., with an 0.5” indent, followed immediately by the text of the sub-sub-section), with the heading also in italics. 1 associate professor of economics, indiana university northwest, 3400 broadway, gary, in 46408. dcoffin@iun.edu. 132 iii. tables and graphs. tables and graphs should be inserted in the text where the author believes they best fit. (they may be moved around a little to better correspond to the space requirements of the journal.) if necessary, tables and graphs may occupy an entire page and may be laid out in portrait or landscape orientation. insofar as possible, tables should fit onto a single page. all tables and graphs should be germane to the paper. references coffin, donald a. (1993). “using ‘the competitive edge’,” journal of economic education, vol. 24, no. 1, pp. 62-69. garcia, jaume, and rodriguez, placido (2002). “the determinants of football match attendance revisited: empirical evidence from the spanish football league.” journal of sports economics, vol. 3, no. 1, pp. 18-38. pappas, doug (2004). “fixing the fan cost index: a more realistic view.” http://www.baseballprospectus.com/article.php?articleid=2790. april 21, 2004. v6n1, hoyert.pdf v6n1, howard.pdf v6n1, isaacson.pdf v6n1, johnson.pdf v6n1, knowlton.pdf v6n1, kreber.pdf v6n1, hughes.pdf v6n1, sigler.pdf v6n1, mission.pdf v6n1, submissions.pdf v6n1, editorialboard.pdf v6n1, stylesheet.pdf microsoft word v9n2sullenberger-8019.doc volume 9 number 2 june 2009 josotl is sponsored by the mack center at indiana university for inquiry on teaching and learning in association with the faculty colloquium on excellence in teaching (facet). 
 i
 
 
 
 
 
 
 volume
9
 number
2
 june
2009
 sabrina
williamson
 sullenberger
&
valerie
 chang
 enhancing
the
success
of
sotl
research:
a
 case
study
using
modified
problem‐based
 learning
in
social
work
education
 1
 flávia
vieira
 developing
the
scholarship
of
pedagogy‐
 pathfinding
in
adverse
settings
 10
 helen
i.
cannella‐malone,
 judah
b.
axe
&
edward
d.
 parker
 interteach
preparation:
a
comparison
of
 the
effects
of
answering
versus
 generating
study
guide
questions
on
quiz
 scores
 22
 ruth
benander
 experiential
learning
in
the
scholarship
of
 teaching
and
learning
 36
 bryan
taylor
&
michael
 kroth
 a
single
conversation
with
a
wise
man
is
 better
than
ten
years
of
study:
a
model
for
 testing
methodologies
for
pedagogy
or
 andragogy
 42
 hollis
glaser
&
shereen
 bingham
 students'
perceptions
of
their
 connectedness
in
the
community
college
 basic
public
speaking
course
 57
 kenneth
davis
&
scott
 weeden
 teacher
as
trickster
on
the
learner's
 journey
 70
 tony
docan‐morgan
 relational
turning
point
events
in
college
 teacher‐student
relationships
 82
 robin
k.
morgan
 student
stalking
of
faculty:
impact
and
 prevalence
 98

 
 josotl
mission
 117
 
 submission
guidelines
 118
 
 editorial
board
 120
 
 style
sheet
 121
 
 journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009, pp. 1 9. enhancing the success of sotl research: a case study using modified problem-based learning in social work education sabrina williamson 1 and valerie chang 2 abstract: this article describes a study which utilizes modified problem-based learning (mpbl) as a teaching method in undergraduate social work practice classes. the authors report both qualitative and quantitative findings of the research. additionally, the authors reflect on the use of the mpbl method and on the lessons learned throughout this research on the scholarship of teaching and learning. keywords: problem-based learning, transfer of learning, social work practice i. introduction. a primary objective of postsecondary education is to prepare students for their professional careers. as such, when students declare a major and begin to pursue studies in a particular area, they begin a trajectory of movement from thinking like a student to thinking like a professional in their chosen discipline. in any given field, thinking like a professional involves discerning meaning from information that is presented to them, organizing knowledge around major principles, responding to changes in context, and accessing and retrieving knowledge smoothly (thompson, licklider, and jungst, 2003). in schools of social work, as with other professional schools, faculty are responsible for developing and delivering curriculum that aids in the student’s transition from novice to professional (koerin, harrigan, and reeves, 1990). in social work educators must identify and teach the step-by-step thinking process used by professional social workers and also be sure that students know how to appropriately apply knowledge and use professional skills as they work with clients. however, despite best efforts students often experience disconnects in their learning when they move from the classroom to professional practice (lager and robbins, 2004). classroom knowledge often is not transferred to the real world experience. consequently, students struggle in their internships as they either don’t use what they learned in class or apply very little of what they learned. in this article we will describe a study in which modified problem-based learning (mpbl) was used in two undergraduate social work practice courses to teach students the process of thinking like professional social workers. we will also discuss the “lessons learned” by the authors during the research process. during phases of data collection and analysis, the authors learned invaluable lessons about teaching and learning and about research in the field of scholarship of teaching and learning. we hope our reflections on this project will help other faculty as they develop their courses and consider engaging in sotl projects. the authors wish to thank indiana university’s mack center for inquiry on teaching and learning for their 1 school of social work, indiana university bloomington, 1127 atwater, bloomington, in 47401, sabawill@indiana.edu 2 school of social work, indiana university purdue university indianapolis, 902 west new york street, indianapolis, in, vchang@iupui.edu williamson, s. and chang, v. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 2 support in this project, both in funding the authors as faculty fellows and in providing guidance and direction throughout the study. ii. literature review. problem-based learning is a learner-centered approach in which learners “conduct research, integrate theory and practice and apply knowledge and skills to develop a viable solution to a defined problem” (savery, 2006, p.12). in social work, the “problem” that students receive is a client scenario that is representative of a real-life situation that an individual, family, group or community might face. students receive parts of the case a section at a time, replicating the process of working with a case in the field. additionally, pbl is pedagogy based on the premise that acquiring process skills is as important as assimilating content (margetson, 1991.) using problem-based learning, students are presented with scenarios faced by social workers in the field and are challenged to find information, apply previously acquired knowledge and work with their colleagues (other students) in planning for assessment and developing interventions. as students work on the problem-based learning cases, faculty can help them learn the process of “thinking like a professional” (middendorf and pace, 2005). pbl is similar to case-based learning with the primary difference between the two being the manner in which information relevant to the case or problem is presented to the students. typically, instructors utilizing case-based learning techniques present an entire case study to student groups rather than in the discrete parts utilized in problem-based learning. course instructors help students analyze the problem, consider solutions, and determine the actions that a professional would take in the situation (herreid, 1994). in addition to its use in social work education, problem-based learning has also been utilized extensively in other professional schools, including business (c.f. saatci, 2008), medicine (c.f. spencer, 2003), and education (c.f. edwards and hammer, 2005). in social work, lam (2004) reports on a bsw program that utilizes pbl in four courses throughout the curriculum. this study measured students’ change in competence, values, and clinical skills before and after pbl instruction and compared these scores with pre-test and post-test scores of students in prior years who had completed the traditional (non-pbl) curriculum. lam found that students who received pbl instruction were more efficient in searching out information and in taking responsibility for their learning. other program faculty, however, rated the performance of pbl students as weaker than non-pbl students in written examinations concerning theories and practice skills. similar to lam’s finding, chan and ng (2004) reported that graduates from pbl training module were found to be “more self directed and creative than their predecessors” (p.316). altshuler and bosch (2003) used pbl in school social work policy and school social work practice classes to prepare students for work in various school settings. at the end of the pbl course work, they sought student feedback concerning perception of content and skills learned. overall, the students thought that the pbl approach was conducive to learning. gelman and mirabito (2005) discuss using pbl techniques with msw students specifically to teach crisis intervention. while not an empirical paper, the authors present the vignettes used to teach students and provide teaching points about how instructors can utilize this method in the classroom. while other authors have written about modifying problem-based learning techniques (c.f. goodnough, 2005 and baldwin, bankston, anderson, echtenkamp, haak, smith and williamson, s. and chang, v. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 3 iatridis, 2002), there does not seem to be uniformity in what modifications are made. a method for using modified problem-based learning (mpbl) in social work education was developed by chang and is described in chang, scott, decker (2009). like pbl, students using mpbl receive a section of a case at a time. this replicates what happens in most disciplines where the student has basic introductory information initially and gradually learns more. also like pbl students work in groups in order to share perceptions and knowledge. unlike pbl the students lead their own groups. with traditional pbl trained teaching assistants work with each student group. the teaching assistants are trained to ask the kind of questions that lead students to explore more deeply and to think like professionals. since we do not have the resources for trained teaching assistants, we modified pbl first by teaching students to be leaders in their groups and second by following each section of a case with the kind of questions that professionals use to guide their thinking. the instructors provided consultation and guidance to the student groups as they used their life experience, knowledge from this course and previous courses, and new material found by doing research to fill information gaps to discuss and answer the case questions. iii. description of study. this study was conducted on two campuses of a midwestern school of social work with undergraduate students. one campus is in a large urban area where students are predominantly commuters. the other campus is in a traditional college town setting where the majority of students live on or adjacent to campus. mpbl was simultaneously used in two undergraduate practice classes, one on each of the campuses. each course has the same course objectives and uses the same textbooks. students take this course in the fall semester of their junior year as their first introduction to practice skills prior to any practice experiences. the research team included the two social work faculty members who taught the practice course at the two research sites, a postdoctoral fellow who has taught social work courses, and a graduate student/research assistant. all of the data was collected by the faculty members. the entire team participated in the analysis of the data. human subjects approval was obtained from the university institutional review board prior to the start of data collection. a. instructional methods. in each course, instructors spent the first four to five weeks of the semester reviewing and introducing theoretical foundations of social work practice. after completing this section of the course, the instructors introduced the concept of modified problem-based learning. students were told that they would work in a small group. each group would have a case that they would work with for the remainder of the fall semester. instructors of the classes indicated that they would serve as consultants to the groups, but that the students were responsible for responding to the questions about the case. in the first week of working on the cases, each student group was given the first section of a case and questions related to that section. the fourteen questions on part one of the case were designed to help students prepare to work with a client by identifying facts and knowledge needed and how they planned to work with and collaborate with their clients. questions related to the first section of the case included: williamson, s. and chang, v. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 4 what are the key facts in the case? what additional information will you need prior to meeting this client? what concerns do you have about working with this client? what are your preliminary impressions related to the case? (chang, scott, decker, 2009, p. 79) following each additional section of the case, students received additional questions to guide their thinking about the case. questions asked about such things as changes in initial impressions, additional information needed, problems and goals identified, treatment plan developed, level of motivation, appropriate action, and eventually ending the work with the client. student groups were given in class time to discuss their case, to share their perceptions and information gained. each group received a different case and turned in a written report answering each question. after they turned in their work related to part one, they received part two of the same case. again, each group was expected to collaborate and to turn in a written document that addressed the questions related to the second part of the case. each group received written feedback from the instructor concerning their responses to the questions. in one class toward the end of the semester, students were given time to share information about their cases and their responses to the questions about the cases. each group engaged in a dialogue with the instructor and other students about their thinking related to their case. in the other class students reported on and discussed their case on the day they turned in each case report. in this class there were several discussions about the cases during the semester. the mpbl work was done in conjunction with more traditional course work of lecture, discussion, and role-plays. parts one and two of the cases were distributed in the fall semester in the first practice class. students received the remaining five parts of their respective cases in their next practice course. b. sample. the sample consists of thirty-five students. this represents 90% of the combined population of the two practice classes. ninety-seven percent of the sample was female, 68% were traditional college aged (20 to 24) and 37% had worked in social service type jobs during the previous year. in terms of ethnicity, the sample consisted of 83% who identified as caucasian, 9% who identified as hispanic, 6% who identified as african-american, and 3% who identified as other. c. data collection and analysis. during the first practice course, each student in the research project completed individual pretests and posttests related to a case that had not been assigned to any of the groups. in the pretest, students answered questions related to parts one and two of the case. this pretest was conducted at the end of the first course module; i.e. after theoretical foundations of practice had been reviewed but before practice skills were introduced and before students began working in their mpbl groups. the post-test was identical to the pre-test and was administered at the end of the semester after students had been working in modified problem-based groups and after they had been exposed to new information and skills in the course. pre-test and post-test data were analyzed both quantitatively and qualitatively. quantitatively, a rubric was developed and used to score each student’s answers on the pre-test and posttest. the rubric, which consisted of answers to the case questions that are consistent with best practices in the social work profession, enabled researchers to evaluate the information williamson, s. and chang, v. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 5 on each test. using spss, t tests were then run to determine if there were significant differences between each student’s pre-test and posttest. additionally, each student’s answers to the pre and post-test were entered into a database and then analyzed in atlas-ti. content analysis (stemler, 2001) was used to determine qualitative differences in the students’ answers that might indicate improvement in the path of “thinking like a professional”. iv. findings. quantitatively, the findings were non-significant. of course with such a small sample it would be very difficult to have significant findings. qualitatively, there is indication that many students showed improvement from pre-test to post-test. student comments on the experience of using modified problem-based learning illustrate many of these areas of improvement. students appreciated the fact that the clients presented in the cases were real, not “made up”, and that the problems faced by the social worker were ones similar to what they would face in their future practice. students’ reports included comments such as: “it was good to apply what we were learning to real people,” “it allowed me to get a real picture of social work,” “it was the next best thing to working in the field.” students also benefited from working with their peers. they said, “it was useful to learn to collaborate in a group and research relevant material” and “being able to discuss situations with other people was great and it helped me prepare to meet with others to professionally and respectfully discuss issues.” students also recognized that the use of cases assisted them in thinking critically about client situations. student comments to support this theme included, “this process taught me to look at a problem from all angles”, “using the cases helped me to think of possible ethical dilemmas and value conflicts” and “this taught me the importance of researching more than the obvious information.” finally, students perceived that the use of cases helped them be more effective and confident in their field placements. they said that “using cases helped me to have the opportunity to experience case management before entering my practicum” and “it helped me become more efficient in my field placement” and “using cases helped me to realize the magnitude of my work, that everything isn’t easy and that there will always be bumps in the road”. in summary, doing mpbl enhanced students’ confidence in their readiness to work with clients, increased their ability to think like professionals, to use professional vocabulary, to understand the need for assessment throughout the life of the case, and to realize the on-going need for more knowledge related to client issues, background and culture, and necessary about resources. while this is gratifying to see, without a control classroom the question remains as to whether these students showed this type of improvement because of mpbl or if they would have improved in similar ways in a traditional practice course. v. lessons learned about teaching and learning. reflecting on this project, we learned ways to improve our teaching. first, it is important to do a pre-test at the beginning of the semester to assess students’ knowledge related to all of course objectives as well as a pre-test related to a case. the pre-test used in this project did not cover all the course objectives. a better pre-test would invite students to evaluate their knowledge and skills related to all the course objectives. if on the pre-test a student identifies that they have knowledge related to a course objective, a follow-up question should ask them to identify how the knowledge was acquired, e. g., by reading, in another course, or on the job. reviewing the williamson, s. and chang, v. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 6 pre-test allows the instructor to tailor the course to the needs of the students. after identifying the strengths that the students bring into the course, the instructor can develop ways to build on these strengths. for example, in the pre-test a number of students identified the importance of building a relationship and establishing rapport in the pre-test. if the instructor had reviewed the pre-test, s/he would have noticed this strength and been able to enhance and build on it. in the post-test some of these students focused on exploring what was wrong with the client and didn’t mention building a relationship with the client. the pre-test also gives information about the gaps in the students’ knowledge. knowing the students’ areas of weakness from the beginning of the course allows instructors to tailor their teaching and assignments to help students’ master key course objectives. using mpbl, instructors can design questions that invite students to explore, discuss and learn information and skills important to their professional growth. another lesson is the importance of allowing time for instructor follow through. if the students had been given the post-test a few weeks before the end of the semester, rather than on the last day of class, instructors could have reviewed the post-test answers and helped the students correct any misunderstandings. for example, some students wrote responses that focused on an aspect of the case that was accurate but less important than other aspects of the case. after realizing this problem, the instructor could have helped the students think through the case again and learn to identify the most important aspects of the case. since applying concepts to social work practice requires higher level learning, students need additional opportunities to practice. therefore, follow through should continue in subsequent practice classes as well as in the field practicum discussion seminar. to enhance student learning, the field practicum seminar should emphasize the concepts learned in the previous course and require students to use the same mpbl case questions as they work with actual clients. the field seminar assignments should require students to demonstrate using the same thinking skills developed by using mbpl in the previous practice classes. a third important lesson is the importance of establishing measureable, achievable course objectives. for this course the objectives were established by a committee that was focusing on accreditation standards rather than what was possible. as educators interested in scholarly teaching we need to continue to take a stand on the establishment of measureable, achievable objectives; otherwise instructors often try to put more content into each class than students can retain and appropriately use in practice. in this course the pressure to cover course objectives that include too much led the instructors to use lecture, role-plays, and mpbl. the instructors were caught between their belief in the value of active collaborative learning and their requirement to cover a great deal of content. this leads to another important lesson. adding a new approach or method of teaching can be positive, but in the case of mpbl it should have been the central focus and main teaching and learning method. adding new approaches takes time. to enhance the value of using mpbl, more class time needs to be allowed for the student groups to discuss, process, work on the cases. each student should have written out all the case answers before the group discussion of the case answers. other course assignments should be structured to emphasize or use the work with the cases. each student group should have regular opportunities to discuss their thinking about each case with the whole class. we know that classroom assessment of learning is very important (angelo and cross, 1993). mpbl is an excellent way to assess learning. since during the semester, the student groups were able to appropriately answer the case questions, the instructors believed the students williamson, s. and chang, v. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 7 had learned to appropriately apply content to cases. however, at the end of the semester, some individual students were not able to replicate best answers to the same questions related to a different case. the students reported that working with the cases helped them feel more confident as they approached working with clients in field placement. if the instructors had spent more time on mpbl, all the students might have not only felt more confident but individually been able to demonstrate greater ability to critically think through case material. the lesson for us is that multiple ways of assessing learning should be used. specifically we should have had individual students answer the case questions. the instructor who could identify thinking problems should evaluate these individual answers. the group could work with all the individual answers as well as do additional research to develop more sophisticated, professional responses to each question. vi. lessons learned about sotl research. besides lessons about teaching, we also learned some important lessons about research on teaching and learning. before starting this research we studied the use of case-based and pbl with graduate students and were excited about the value of using mpbl with undergraduate students. the best course to introduce mbpl was a practice course that is offered only in the fall semester. unfortunately, by the time we had developed our ideas the fall semester was starting. not wanting to wait until the next fall, we jumped into the project too quickly. both authors were teaching this practice course. each course had the same objectives and same textbooks and we thought our plans for using mbpl were the same. looking back we identified that our methods of using mbpl were similar but had several differences. one author spent more class time on mpbl than did the other author. one author had student groups report and discuss their cases more frequently during the semester. in retrospect we are aware that we should have taken the time to write a manual with step-by-step directions for using mbpl. having a written manual would also allow the study to be replicated. also, we could have had one class use mpbl and had the other class serve as the control group. this method would have strengthened our research design. if we had a control class, we would have been able to assess whether the changes that occurred could be attributed to using mpbl. we didn’t use a control class because both of us thought mpbl was a better way to teach this course. our commitment to offering students what we considered the best teaching and learning method meant that, ethically, we could not utilize a control group. (even if the use of a control group had been possible, however, sample size was too small to demonstrate significant findings.) although most students showed good improvement from the pre-test to the post-test some students did not improve and some had less satisfactory answers on the post-test. there could be many reasons for this decline. we believe a significant reason was that the post-test was given on the last day of class when the students were anxious to get finished. some of their answers were very brief and seemed hurried. in the future we will not wait until the final class to do the post-test. finally, with more time we could have followed the students into their field practicum setting to assess whether what they learned using mpbl enhanced their ability to think like professional social workers in actual practice. although this would strengthen the research, it would be very complicated to control because each student is in a different setting with a different field instructor. even with a control group there would be many uncontrollable variables. given all of these complications, we still believe that meeting with the students at the williamson, s. and chang, v. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 8 end of their practicum semester to discuss their perceptions of the value of mpbl to their education would have been valuable. vii. conclusion. as indicated in preceding paragraphs, each author believed that utilizing mpbl in practice classes was the best way to teach social work students how to think like professional social workers. each certainly expected the method to show success. when the preand post-tests were analyzed and the evidence of success was inconclusive, all members of the research team were somewhat disappointed because the level of student improvement was less than what we had hoped. however, the lessons learned about the mpbl method and about conducting sotl research have been invaluable. just as there is a process of learning to “think like a professional,” there is, perhaps, a process of learning to “think like a sotl researcher” as well. it is our hope that other sotl researchers will continue this research using mpbl with students in other disciplines. as researchers in other disciplines have found (lundeberg, 1999; wolfer and scales, 2006), using cases can help students transfer learning from the classroom to the world of work. in mpbl the instructor models for students the questions that professionals use as they approach challenging case situations. working with these questions seems to be a logical way to help students in any discipline learn how to think about challenging case situations. references altshuler, s.j., and bosch, l.a. (2003). problem-based learning in social work education. journal of teaching in social work, 23(1/2), 201-215. angelo, t.a. and cross, k.p. (1993). classroom assessment techniques: a handbook for college teachers. san francisco: jossey-bass. baldwin, w., bankston, p., anderson, w.m., echtenkamp, s., haak, r., smith, p, iatridis, p.g. (2002). can students in a modified pbl curriculum exceed the national mean on usmle part 1? medical education, 36 (8), 791-795. chan, c.l.w. and ng, s.m. (2004). the social work practitioner-researcher-educator: encouraging innovations and empowerment in the 21 st century. international social work, 47(3), 312-320. chang, v.n. ,scott, s.t., and decker, c.l. (2009) developing helping skills: a step by step approach. belmont, ca: brooks/cole. edwards, s. and hammer, m. (2005). laura’s story: using problem-based learning in early childhood and primary teacher education. teaching and teacher education, 22 (4), 465-477. gelman, c.r. and mirabito, d.m. (2005). practicing what we teach: using case studies from 9/11 to teach crisis intervention from a generalist perspective. journal of social work education, 41(3), 479-494 williamson, s. and chang, v. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 9 goodnough, k. (2005). issues in modified problem-based learning: a self-study in pre-service science-teacher education. canadian journal of science, mathematics, and technology education, 5 (3), p289-306. herreid, c.f. (1994). case studies in science education. journal of college science teaching (23) 221–229. koerin, b. b., harrigan, m. p., and reeves, j. w. (1990). facilitating the transition from student to social worker: challenges of the younger student. journal of social work education. 26(2), 199-208. lager, p.b. and robbins, v.c. (2004). field education: exploring the future, expanding the vision. journal of social work education, 40 (1), 3-12. lam, d. (2004). problem-based learning: an integration of theory and field. journal of social work education, 40(3), 371-391. lundeberg, m. a. (1999). discovering teaching and learning through cases. in m. a. lundeberg, b. b. levin, and h. l. harrington (eds.), who learns what from cases and how? the research base for teaching with cases (pp. 3 – 23). mahwah, nj: lawrence erlbaum. margetson, d. (1991). why is problem-based learning a challenge? in d. boud and g. geletti (eds.), the challenge of problem based learning. new york: st. martin's press, pp. 42-50. middendorf, j., and pace, d. (2005). decoding the disciplines: a model for helping students learn disciplinary ways of thinking. new directions for teaching and learning, 98, 1-12. saatci, e. (2008). problem-based learning in an intercultural business communication course. journal of business and technical communication, 22 (2), 237-260. savery, j.r. (2006). overview of problem-based learning: definitions and distinctions. the interdisciplinary journal of problem-based learning, 1(1), 9-20. spencer, j. (2003). abc of learning and teaching in medicine: learning and teaching in the clinical environment. bmj, 326, 591-594. retrieved november 29, 2008 from http://www.bmj.com/cgi/reprint/326/7389/591 stemler, s. (2001). an overview of content analysis. practical assessment, research and evaluation, 7(17). retrieved january 9, 2007 from http://pareonline.net/getvn.asp?v=7andn=17 thompson, j., licklider, b., and jungst, s. (2003). learner-centered teaching: postsecondary strategies that promote ‘thinking like a professional’. theory into practice, 42(2), 133-141. wolfer, t.a and scales, t.l. (eds.) (2006). decision cases for generalist practice: thinking like a social worker. belmont, ca, brooks/cole. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009, pp. 10 – 21. developing the scholarship of pedagogy: pathfinding in adverse settings flávia vieira1 abstract: the paper looks at the feasibility and value of the scholarship of pedagogy (sop) in an institutional context where it is not a common practice. i will draw on my experience with other colleagues at our university concerning the constraints, shortcomings and achievements of sop, and use this as a springboard for reflection on its transitional nature as a “pathfinding route” in adverse settings (shulman, 2004). i will also discuss implications on how we might assess its value, arguing that this assessment should take into account not only its quality as a research-oriented activity, but also its situational relevance. ultimately, there may be no universal answer to the question “what is valuable sop?” keywords: pedagogy, adverse settings, feasibility, value i. introduction. i came across shulman’s (2004a) metaphors of pathfollowing and pathfinding as different choices in one’s academic career when i was writing a report on a collaborative, multidisciplinary project carried out with a focus on transforming pedagogy at university through classroom-based inquiry (vieira, silva, melo, moreira, oliveira, gomes, albuquerque, and sousa, 2004). shulman’s reflection on the risks and challenges of the scholarship of teaching and learning, as a pathfinding route in contexts where research is mostly disciplinary and detached from teaching, resonated with our personal experience in a profound way. we had been assuming the role of teacher-researchers for the first time in our professional history, not knowing exactly what the result of our work would be, yet feeling that it ran counter to dominant academic discourses, practices and values, that is, it went “against the grain” as shulman puts it. in our contacts with other colleagues, we could sometimes sense their suspicion and skepticism toward inquiry that does not follow the path of mainstream discipline-based research, reminding us of the dangers involved in pedagogy-oriented research, especially in terms of how it may affect your credibility as an academic researcher. we have continued working on the scholarship of teaching and learning, here labeled scholarship of pedagogy (sop) since the term “pedagogy” integrates teaching and learning as interrelated activities. we have developed two other projects and are now even more aware of the risks and challenges it involves, but less frightened by them as we became more convinced of the benefits. our work has also become a bit less marginal, not only due to its growth, but also to the increasing preoccupation with teaching quality in portuguese universities, mainly instigated by current reforms resulting from the bologna process. nevertheless, sop is far from being commonplace, and its value is not yet fully understood and acknowledged, as is probably the case in most institutional settings. this is why i decided to take feasibility and value as the main themes of this paper, hoping that my reflection may resonate with the experience of others in 1 department of methodologies of education, university of minho, institute of education and psychology, campus de gualtar, 4710-057-braga, portugal. flaviav@iep.uminho.pt. vieira, f. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 11 similar circumstances. although many institutions around the world have embraced the idea of sop for a long time, this is far from being a worldwide phenomenon2. i believe that we need more accounts of cases where sop is the exception rather than the rule, so that we may enhance practices not yet established and better appreciate their dilemmas, outcomes and shortcomings, as well as the strategies used to find spaces for manoeuvre. i will start with some considerations on what we mean by sop and why it should (not) be developed in our setting, then move on to an overall evaluation of our projects and discuss the issues of sop feasibility and value. my purpose is not to go into the details and results of our work, but rather to highlight some of its constraints, shortcomings and achievements, presenting it as a case of transitional pathfinding in adverse settings, with implications on how we might assess its value. my argument is that the value of sop may not lie exclusively in its quality as a research-oriented activity, but also in its situational relevance. actually, circumstances may reduce its research quality in significant ways, but that does not necessarily mean that it is worthless. ultimately, there may be no universal answer to the question “what is valuable sop?” ii. what is sop and why should it (not) be developed? in order to clarify our understanding of sop, i will tell a personal story that involves not only me and my students, but also the colleagues with whom i first engaged in it. in my rather long professional history as a teacher educator in my university3, i have always advocated an inquiry-based approach to teaching in schools and supported school teachers in using action research to become reflective practitioners in search of context-sensitive, learner-centred pedagogies. i myself have always tried to reflect on my practice and involve my students in evaluating its value and shortcomings. nevertheless, it was not until 2003 that i began to understand more fully what becoming a teacher-researcher means. i was then coordinating (since 2002) the collaborative project referred to above, and we had decided to undertake small-scale case studies of pedagogical innovation in our own classes, which was quite innovative in our context4. my case study involved a group of 13 experienced language teachers in a postgraduate course, and i took the opportunity to enhance and evaluate professional learning through school-based pedagogical inquiry documented in reflective portfolios. at the same time, i wrote a teaching journal that documented the approach i was implementing. that was the first time i inquired into my own pedagogy in a disciplined way, trying to understand its value and shortcomings through analysing data from the teachers’ portfolios and my journal (vieira, 2005, 2007a/b). what struck me most at the time was the fact that, although i had been an educational researcher for a long time and had always advocated the use of pedagogical inquiry in schools, i 2 for example, tight’s (2003) overview of research into higher education in 406 articles by 668 authors from 48 countries, published in 17 journals outside north america in 2000, shows that inquiry into one’s teaching is not a preferred research mode among higher education researchers, even though teaching and learning are prominent themes. in fact, the expression “scholarship of teaching (and learning)” does not even figure in his book’s index. 3 the university of minho is a teaching and research university in the north of portugal with about 16.000 students, offering a wide range of undergraduate and postgraduate programmes, including teacher education in various subject areas. it is organised around schools or institutes (e.g., the institute of education and psychology, where i belong), and these are organised into departments. i have worked in the department of methodologies of education since 1984, mainly on reflective teacher education/ supervision and pedagogy for autonomy in schools. my work on sop has focussed primarily on pedagogies of postgraduate teacher education. 4 in 2001, the research centre at the institute of education and psychology set up an internal contest for projects focussed on pedagogy at university, since this was considered an understudied area. our project was the only proposal presented to the centre, which accepted and funded it from 2002 to 2004. it was the first sop project carried out in the institute. vieira, f. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 12 still had to learn how to investigate my practice. this also meant that i knew much less about being a teacher-researcher than i had previously assumed, which made me question my role as a teacher educator. what i had been missing all along was experiential knowledge, and the case study helped me understand, more clearly than ever before, that improving the education of others depends on improving my own education. this very simple idea lies at the heart of sop, and it resonated with my colleagues’ experience as they conducted their own case studies. we began to realise that we were developing a bottom-up and side-to-side approach to professional growth through inquiry into our students’ and our own education. taking learning seriously, to take shulman’s (2004b) phrase, can not be dissociated from taking teaching seriously, and that means re-examining our pedagogical beliefs and choices. since that first experiment i have wondered about, theorised and improved my practice in a more purposeful and systematic way, and i have become more interested in the concept and practices of sop5. by mid-2004, when the project was coming to an end, the team managed to institutionalise higher education research within the institute of education and psychology, by setting up a larger interdepartmental research group that has integrated projects on sop and other areas. from 2004 to the present date, i have coordinated two more sop projects involving colleagues from various disciplinary fields6. as a result of this work, we have been talking and writing about our teaching experience more extensively than before, moving away from “pedagogical solitude” by making our pedagogical practice “community property” (shulman, 2004c), and encouraging others to do the same. we have also developed a more critical view of academic work and cultures, and how they both hamper and justify sop. in sum, pedagogy became a central professional concern and definitely entered our research agenda, even though pedagogical inquiry is not acknowledged and rewarded. this story illustrates some features of sop as we have understood it so far:  it rests on the assumption that pedagogy is a valuable yet understudied activity, therefore it should become a field of inquiry;  it is, first of all, a self-initiated path to become a better educator, instigated by professional motivations and concerns related to issues of student and teacher development in a given disciplinary field;  it is also a collective enterprise, not only because students and colleagues become partners in pedagogical dialogue and inquiry, but also because it involves making that inquiry public and open to debate so that others can evaluate and build on it, thus seeking to enable change in institutional cultures and contribute to advances in the teaching profession as a whole;  it is not something that academic researchers are necessarily ready to undertake, since it moves away from conventional disciplinary inquiry, especially by blurring the frontiers between research and teaching. of course, developing sop is more easily said than done, and contexts of practice clearly affect its feasibility (see hutchings and shulman, 2004; socket, 2000). in our context, there are various reasons why it should be promoted or avoided, depending on one’s point of view. 5 the website of the carnegie foundation for the advancement of teaching (http://www.carnegie foundation.org) was particularly useful at the beginning. the chapters of shulman’s paper collection edited by pat hutchings (2004) which i refer to in this text were firstly accessed on that website. 6 our work is summarised on the website: http//webs.uminho.pt/tpu (tpu stands for the phrase transforming pedagogy at university, which we have used as the main title of our projects since 2002). vieira, f. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 13 the idea of sop is quite unfamiliar to the portuguese academic community in general. to a large extent, “discussions about teaching and learning tend to be fugitive affairs” (huber, 1999, p. 1), and pedagogy is not yet seen as a worthwhile field of inquiry. academics teach everyday and value their role as educators, and current curriculum changes within the bologna process have led institutional managers to become more committed to innovation and staff development. yet, research is mostly discipline-based and only one aspect of scholarship, the “scholarship of discovery” (boyer, 1990), tends to be given institutional and professional credit. who we are and what we do as teachers gets little attention and merit in our academic career, and the relation between teaching and research tends to be either conflictive or null (cf. hattie and marsh, 1996; gottlieb and keith, 1997; serow, 2000; vidal and quintanilla, 2000). scattered research groups have worked on higher education as an emergent crossdisciplinary area, though very seldom with a focus on self-inquiry. as in many other parts of the world, higher education research in portugal is becoming a specialised territory owned by a few experts, which may explain why its impact on teaching practices and policies is often reduced or null (cf. teichler, 2000). furthermore, we have no specialised journals on higher education, no established in-service or postgraduate staff development programmes, and almost no institutional reward systems to enhance classroom innovation and research. teaching quality is assessed through student feedback and a national system that combines internal and external course evaluation, but all this tends to have little impact on quality improvement, since we lack mechanisms for sustainable professional development. given this scenario, we might argue for the need to develop sop as a strategy that might usefully contribute to changing the present state of affairs. paradoxically though, it is also this scenario that hampers sop and even discourages it. actually, academic work is fraught with conflicting rationalities that make one’s choices problematic. a significant dilemma for any teacher wishing to engage in sop relates to academic merit and success. in the foreword to a collection of sop case studies, shulman (2004a, p. viii) uses a four-fold table to represent (lack of) academic success in terms of “disciplinary and pedagogical virtue”, identifying four kinds of scholars (see table 1). he uses the metaphors of pathfollowers and pathfinders to refer respectively to “those who behave as most of their disciplinary colleagues expect them to, and those who elect to go against the grain” (p. vii). engaging in sop often means going against the grain and becoming a pathfinder, which is not compatible with academic cultures where border crossing among disciplines, peer collaboration and nondisciplinary research tend to be dismissed (see bergquist and pawlak, 2008). and even though successful pathfollowing can be seen as a kind of “specialised ignorance” (santos, 1998), that is, knowing much about little and ignoring everything else, this is exactly what most faculty still cherish and get credit for. table 1. pathfollowing and pathfinding (shulman, 2004a). conforms to disciplinary convention? yes no leads to academic yes successful pathfollowers successful pathfinders advancement? no unsuccessful pathfollowers unsuccessful pathfinders shulman (op. cit.) warns us about the risks and extra demands of becoming successful pathfinders in a world where pathfollowing represents the mainstream culture: “while being vieira, f. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 14 ‘good enough’ may be sufficient for many engaged in traditional research in their discipline, it is probably not going to be sufficient for work in education” (p. ix). he also challenges us to work on a vision of the possible and build a new, more balanced conception of the scholarly career, so that we no longer have to choose between the various facets of academic work. this is certainly an inspiring thought, but very remote from reality in most contexts of practice, where “teaching and research are frequently, even habitually, regarded as rivals: time and status pitting for the ‘learning’ of one against the ‘learning’ of the other” (light, 2003, p. 157). this “pervasive and insidious ‘rivalry of learning’” (p. 162) often turns sop into a marginal, unsystematic, and inconsequential activity. this is something we have been increasingly aware of in our work, feeling that we are swimming against the tide, yet having to swim with it to avoid drowning. we have faced many constraints that affect the feasibility of sop, and we have had to turn our backs on it more often than we would like, mainly because we are divided between pathfollowing and pathfinding, perhaps running the risk of not being successful enough in either one or the other. we may therefore add some more ideas to the list of sop features:  it is difficult to implement in contexts of practice that undervalue pedagogy and pedagogical inquiry, even though it is most needed in those contexts;  it takes self-determination and boldness to face its risks and challenges, and also resistance to historical and structural forces that counteract it in significant ways;  it will most probably entail a tension between conformity to and subversion of mainstream academic practice. in the next section, i will focus on the feasibility and potential value of sop by drawing on constraints, shortcomings and achievements of our work, here taken as an example of transitional pathfinding in adverse settings. iii. sop as a transitional pathfinding process – feasibility and value. so far, our projects have involved around 30 scholars from different subject areas, and our work can be understood as a developmental approach to sop, seeking to explore and consolidate it (very slowly, i must say…), particularly by enlarging its scope in terms of the quantity and diversity of pedagogical experiments, and the number of teachers and disciplinary fields involved. our goals have been: (a) to enhance an inquiry-oriented approach to pedagogy, based on a notion of “quality as transformation”7, where student enhancement and empowerment are valued (harvey and knight 1996; kreber, 2006; vieira, 2002), (b) to develop case studies whereby innovative educational methodologies and resources are explored, evaluated and disseminated, and (c) to encourage the constitution of multidisciplinary teams of educational and non-educational faculty for the construction of educational knowledge and the renewal of educational practices. to a significant extent, our achievements cannot be separated from our shortcomings, since the former relate mostly to how we have tried to face and surpass constraints. although each pedagogical experiment has specific gains for the teacher and students involved, i will focus here on general aspects of our work on sop as a collective undertaking, which may be of interest to others working (or wishing to work) along the same lines in adverse settings. 7 according to this notion of quality, “education is a participative process. students are not products, cusomers, service users or clients – they are participants. education is not a service for a customer (much less a product to be consumed) but an ongoing process of transformation of the participant” (harvey and knight, 1996, p. 7). vieira, f. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 15 table 2 summarizes a possible evaluation of our work, based on the research-oriented standards set up by josotl to evaluate the quality of scholarship in general, including sop: clear goals, adequate preparation, appropriate methods, significant results, effective presentation, and reflective critique8. in using these standards, i draw a distinction between two facets of pathfinding in adverse settings: (a) the first facet refers to the restraining effect (e) of circumstances (c) upon sop development, that is, the shortcomings resulting from constraints (middle column); (b) the second facet refers to strategies used and signals potential achievements (righthand column). from this perspective, pathfinding can be seen as a transitional process where the feasibility of sop – as regards its scope, impact and sustainability – is affected by cultural circumstances. table 2. our work: pathfinding as a transitional process. qualitative standards of scholarship (josotl)** constraints and shortcomings… [circumstances (c) and effects (e)] achievements… [development strategies] clear goals all scholars must be clear about the goals of their scholarship. what is the purpose of the scholarship and are the goals clearly stated? (c) lack of tradition in sop may lead to (e) low sense of direction, difficulties in problem-framing, technical view of educational problems, and fuzziness of sop goals (c) diversity of conceptual/ experiential backgrounds (as regards pedagogy and research) and a tradition of pedagogical solitude may lead to (e) lack of unity/ coherence among different case studies collaboration to find common ground and a shared sense of direction/ purpose (through dialogue, support, feedback, peer observation, joint paper presentation/ writing…) discussion of conceptual/ ethical assumptions and choices as regards pedagogic quality (e.g., learner/centred pedagogical principles ) adequate preparation all scholars have the background knowledge and skills to successfully investigate the problem. does the scholar have the prerequisite skills to thoroughly investigate the problem? (c) lack of (time to invest in) pedagogical knowledge and research skills (especially from non-educational scholars) may lead to (e) communication problems between educational and noneducational experts, over-reliance from the latter on the former for guidance, and low self-confidence/ ability to undertake pedagogical inquiry joint reflective sessions/ seminars on pedagogical issues and research strategies supportive environment and opportunities to share experiences and materials readings on higher education and sop appropriate methods scholarship must be carried out in a competent manner for results to have credibility. did the scholar use the appropriate procedures to investigate the problem? (c) lack of (time to invest in) pedagogical knowledge and research skills (especially from non-educational scholars) may lead to (e) over-reliance on well-established teaching/research practice within different disciplinary fields and inability to design creative teaching/research methodologies that are responsive to the complexity of educational problems joint reflective sessions/ seminars on pedagogical issues and research strategies effort to design pedagogical inquiry that is responsive to relevant educational concerns developing teaching and research skills as an outcome of pedagogical inquiry keeping open to diverse, more and less sophisticated forms of inquiry 8 the editorial board of josotl (the journal of scholarship of teaching and learning) present these standards in the guidelines for reviewers (http://www.iupui.edu/~josotl/review_guide.htm). they are taken from the book scholarship assessed: evaluation of the professoriate, by charles glassick, mary huber, and gene maeroff (1997, san francisco: jossey-bass). i came across the standards when i was searching for information about the journal before submitting this paper, and i decided to apply them to our work. vieira, f. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 16 significant results one of the most critical criteria in judging the quality of scholarship is whether scholarship can be used as the building blocks of knowledge in the field. scholarship may not always result in “significant” results but to have quality the results must inform scholars in the field. does the scholarship help build the knowledge base in the field? (c) conflict between the demands of sop, disciplinary research agendas and teaching/ management workload may lead to (e) research/ writing delays, insufficient exploration of data, and limited conclusions (c) lack of time and opportunities to share research results and undertake comparative analyses of case studies may lead to (e) limited understanding of the nature and impact of sop context-sensitive pedagogical innovation analysis of pedagogical experiments in terms of potential value for teacher and student development focus on the implications of teaching/ research methodologies for future practice (continuity) supportive environment and opportunities to share experiences and results dissemination and supportive peer review/ critique effective presentation to have quality it is essential that scholarship be accessible to the intellectual or professional community. there are many forums that provide opportunities for the review and critique by colleagues with each medium having different criteria for effectiveness. does the scholarship meet the standards or quality for the medium in which it is presented? (c) lack of tradition and forums in sop may lead to (e) low confidence in the credibility of sop, lack of appropriate presentation/ writing abilities, avoidance of public dissemination, production of low quality reports (e.g., too descriptive), and reduced dissemination and internationalisation dissemination in educational conferences, especially through collaborative paper presentation and writing valuing different dissemination media and discourse genres, even though this means not publishing in peer reviewed journals and sacrificing academic prestige reflective critique all scholarship must create an opportunity for collegial critique but it is also essential for the scholar to reflect on the scholarship and learn from the results. insightful reflection is a necessary step in quality scholarship. is there evidence that the scholar has learned from the experience and can apply this knowledge to future problems? (c) lack of tradition and experience in sop, lack of multiple frames of reference to analyse pedagogy, and insufficient involvement in peer review and dissemination of sop may lead to (e) low levels of criticality in terms of interpreting experience and drawing implications for future work on sop joint reflective sessions/ seminars on pedagogical issues and research strategies dissemination, especially through collaborative paper presentation and writing focus on the implications of teaching/ research methodologies for future practice (continuity) readings on higher education and sop [** source: josotl guidelines for reviewers in http://www.iupui.edu/~josotl/review_guide.htm] the circumstances of our work certainly reduce its quality as regards research-oriented standards for assessing scholarship, and we are far from being a community of “successful pathfinders”. however, my analysis also calls into question the universal validity of those criteria by introducing the idea of sop as a transitional process, and therefore a notion of value as situational relevance, which presupposes that the assessment of sop quality must entail an understanding of contexts of practice, since these largely determine its feasibility and potential outcomes. i would then suggest that we need to look at the issue of value by taking into account three questions related to the context where sop takes place: (a) is sop contrary to mainstream academic work as regards both teaching and research, that is, is an inquiry-based approach to pedagogy a form of “going against the grain”?, (b) is sop new to the teacher who engages in it, that is, does it involve a significant personal transformation in her/his teaching and research experience?, and (c) is the impact of sop felt beyond individual practice, that is, does sop help vieira, f. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 17 to build a knowledge base in the field, informing others of relevant educational issues and practices, and enhancing the teaching profession as a whole? if we accept that these questions (the last of which was already suggested by shulman, 2004a) are important to determine the situational relevance of sop, then we would say that its value depends on how it relates to (a) the dominant culture(s) of the academy (value as cultural subversion/ innovation), (b) the teacher’s history (value as professional transformation), and (c) the advancement of sop itself as a field of inquiry (value as the enhancement of the teaching profession, both theoretically and pragmatically). table 3 presents the situational relevance of sop along these three value dimensions, in adverse and favourable institutional settings. table 3. situational relevance of sop. adverse settings favourable settings cultural subversion/ innovation yes no professional transformation yes mostly for novices enhancement of the teaching profession probably not yes from this perspective, unsuccessful pathfinding as defined by shulman is not necessarily a sign of invaluable sop: it may be a sign of situationally valuable sop, whose quality is determined by (and cannot be assessed without reference to) its feasibility. in fact, its value may lie exactly in the struggle to make it possible in contexts that disempower practitioners to pursue it, so that the state of affairs is eventually transformed. contributing to the enhancement of sop in the academy is certainly difficult when pedagogy is not acknowledged as a legitimate field of inquiry. the impact of our work on disciplinary communities depends on whether we manage to disseminate it among our peers and involve them in sop-like initiatives. this is extremely hard to achieve when pedagogy is not a priority of the professoriate. one of our team members points out: “the relevance of peers, particularly from the same scientific area, is very low: i’m not sure about other areas, but i have to admit that, even though some tiny interest in educational issues can be found here and there, in most situations each teacher sees her/himself as someone who ‘has always taught good lessons and will always do so’. discussion of pedagogical matters as well as the participation in pedagogical development sessions are considered to be a waste of time, even more since they do not count in anyone’s cv” (j.a., personal reflection). before teaching is seen as an integral part of inquiry in any disciplinary field, the idea of generalized sop remains an ideal. shulman (2004d) points out several models for campus support of sop through the constitution of what he calls “teaching academies”. he further contends that “it may make perfectly good sense to shape an approach that does not presume to be ‘institutionalized’ in the usual sense of the word but that takes advantage of pockets of interest and potential” (op. cit., p. 212), an approach which he labels “the distributed teaching academy”. i can identify emergent signs of this in our university, where groups of teachers in different schools and departments have carried out interesting research and staff development initiatives9. 9 for example, curriculum innovation projects in the medical and science schools; the implementation of project-led education in engineering courses; the use of action research in pre-service teacher supervision in schools; programmes on study skills training for first year students organised by colleagues in the psychology department; staff development courses on learner-centred methodologies sponsored by the rectory office; materials design projects to support and evaluate innovation on campus; and our own approach to classroom-based inquiry in a multidisciplinary setting. vieira, f. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 18 however, more effort needs to be invested in making these rather scattered initiatives more connected, visible and accessible before we can start talking about a “model”. iii. back to basics – what does pedagogy really entail? as i come to the end of my paper and reflect on my professional experience, my mind keeps wrestling with this disquieting, back-to-basics question: “what does pedagogy really entail?” after all, the value of sop also depends on the nature of pedagogy itself. reflective practitioners know that they often have to sacrifice rigour if they want their action to be situationally relevant, and that teaching often requires artistry to “make new sense of uncertain, unique, or conflicted situations” (schön, 1997, p. 35). this means that professional reflection-in-action is at least as important as the reflection for/on action that is more typical of disciplined inquiry. it further means that an epistemology of practice cannot be reduced to disciplined inquiry and can never be fully captured by it. as van manen (1990) suggests, pedagogy is ineffable, and if we take descriptions and conceptualisations of reality as reality itself, we will probably fail to seek and understand the deep significance of pedagogical encounters that such descriptions and conceptualisations often conceal (p. 149). the ineffability of pedagogy limits our claims to certainty as regards research results: these only tell us part of the story. even if we agree that sop is mostly about pedagogical inquiry, dissemination and public scrutiny, we must also realise that its value lies in ontological, axiological and praxiological aspects of education that are not measurable or even liable to be studied in a disciplined manner. from this perspective, the situational relevance of sop also entails the unexamined experience of teachers and students as they work together to make sense of the pedagogical encounter. therefore, when we emphasise a research-based notion of sop as distinct from scholarly or excellent teaching, we are perhaps dismissing important facets of pedagogy and casting sop into just another measure of research activity, as suggested by bowden (2007; see also kreber, 2006, and silva, 1999). the view of pedagogy as a multifaceted and, to a certain extent, incomprehensible phenomenon may appear to reduce the worth of sop. on the contrary, i believe it turns sop into a moral and political imperative – if pedagogy entails continuous (self-)questioning on what is good education, then it must become a field for continuous inquiry. what this inquiry tells us about pedagogy is another matter: perhaps it does not tell us everything about teachers’ and students’ ways of knowing, acting and being, the three pillars of engagement in higher education (barnett and coate, 2005). v. final remarks. ultimately, there may be no straightforward, universal answer to the question “what is valuable sop?”. its value depends not only on its quality as a research-oriented activity, but also on how it relates to contextual factors, and it entails a consideration of the complexity and ineffability of pedagogy. paradoxically or not, this state of “fuzziness” as regards the value of sop is itself a result from trying to make sense of it. whatever form sop assumes, one thing at least seems to be common to those who commit themselves to it – the hope for a better future in higher education. fulfilling this hope affects our identity as academics and entails learning to deal with uncertainty as we embark on an exploratory journey that builds on our past histories as (mostly pathfollowing) teachers and vieira, f. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 19 researchers, taking us in a more promising (hopefully pathfinding) direction. it is this transitional process that makes the journey purposeful. this suggests the need to investigate local cultures closely and the extent to which scholars reproduce and/ or subvert them, as a potentially fruitful pathunderstanding strategy. reflecting about our work and the work of others on sop, i feel tempted to use shulman’s statement in reference to the carnegie scholars that “one of our central premises is that change is directed toward visions of the valued, the possible, the desirable and the imaginable” (shulman, 2002, p. 6). however, educational visions and agendas in institutional settings are potentially related to multiple frames of reference and directed towards diverse, even competing purposes: “the ethos of the university, therefore, cannot be pure. the university knows, deep down, that its fundamental value structure is flawed” (barnett, 1997, p. 15). as bergquist and pawlak (2008) suggest, we may need to learn to appreciate the different and often conflicting cultures of the academy, and develop an “ironic understanding” of those cultures through engaging critically with their paradoxes, thus developing a capacity for “transformative growth” (p. 228). this may well be one of the greatest challenges sop practitioners face today. acknowledgements the sop projects referred to in this paper have been funded by the centre of research in education at the institute of education and psychology, university of minho. they are part of the research line higher education: images and practices (coordinated by f. vieira). i want to thank my colleague and friend leslie bobb wolff for revising and commenting on a previous version of the paper. references barnett, r. (1997). realizing the university. london: institute of education university of london. barnett, r. and coate, k. (2005). engaging the curriculum in higher education. glasgow: society for research into higher education and open university press. bergquist, w. and pawlak, k. (2008). engaging the six cultures of the academy. san francisco: jossey-bass publishers (revised and expanded edition of the four cultures of the academy, 1st ed. 1992). boyer, e. (1990). scholarship reconsidered: priorities of the professoriate. princeton: the carnegie foundation for the advancement of teaching. bowden, r. (2007). scholarship reconsidered: reconsidered. journal of the scholarship of teaching and learning, 7 (2), 1-21. gottlieb, e. and keith, b. (1997). the academic research-teaching nexus in eight advancedindustrialized countries. higher education, 34, 397-420. vieira, f. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 20 harvey, l. and knight, p. (1996). transforming higher education. buckingham: the society for research into higher education and open university press. hattie, j. and marsh, h. (1996). the relationship between research and teaching: a metaanalysis. review of educational research, 66 (4), 507-542. huber, m. (1999). disciplinary styles in the scholarship of teaching and learning. paper presented at the 7th international improving student learning symposium. retrieved may 8, 2002, from http://www.carnegiefoundation.org. hutchings, p. and shulman, l. (2004; first published in 1999). the scholarship of teaching and learning: new elaborations, new developments. in l. shulman (paper collection ed. by p. hutchings), teaching as community property – essays on higher education (pp. 145-154). san francisco: jossey-bass. kreber, c. (2006). developing the scholarship of teaching through transformative learning. journal of the scholarship of teaching and learning, 6 (1), 88-109. light, g. (2003). realizing academic development: a model for embedding research practice in the practice of teaching. in h. eggins and r. macdonald (eds.), the scholarship of academic development (pp. 152-162). buckingham: srhe and open university press. santos, b. s. (1998, 10th ed.; 1st ed. 1987). um discurso sobre as ciências. porto: edições afrontamento. schön, d. (1987). educating the reflective practitioner. san francisco: jossey-bass publishers. serow, r. (2000). research and teaching at a research university. higher education, 40 (4), 449463. shulman, l. (2002). 97th annual report. the carnegie foundation for the advancement of teaching. retrieved november 25, 2004, from http://www.carnegiefoundation.org. shulman, l. (2004a). four-word: against the grain. in m. t. huber (ed.), balancing acts: the scholarship of teaching and learning in academic careers. retrieved november 25, 2004, from http://www.carnegiefoundation.org. shulman, l. (2004b; first published in 1999). taking learning seriously. in l. shulman (paper collection ed. by p. hutchings), teaching as community property – essays on higher education (pp. 33-47.). san francisco: jossey-bass. shulman, l. (2004c; first published in 1993). teaching as community property: putting an end to pedagogical solitude. in l. shulman (paper collection ed. by p. hutchings), teaching as community property – essays on higher education (pp. 139-144). san francisco: jossey-bass. vieira, f. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 21 shulman, l. (2004d; first published in 1999). visions of the possible: models for campus support of the scholarship of teaching and learning. in l. shulman (paper collection ed. by. p. hutchings), teaching as community property – essays on higher education (pp. 203-217). san francisco: jossey-bass. silva, m. c. (1999). the scholarship of teaching as science and art. inventio, 1 (1). retrieved december 4, 2003, from http://www.doit.gmu.edu/archives/feb98/msilva.htm. socket, h. (2000). creating a culture for a scholarship of teaching. inventio, 2 (1). retrieved december 4, 2003, from http://www.doit.gmu.edu/inventio/past/display_past.asp?pid= spring00&sid=socket. teichler, u. (2000). the relationship between higher education research and higher education policy and practice: the researchers’ perspective. in u. teichler and j. sadlak (eds.), higher education research – its relationship to policy and practice (pp. 3-34). oxford: pergamon. tight, m. (2003). researching higher education. buckingham: the society for research into higher education and open university press. van manen, m. (1990). researching lived experience – human science for an action sensitive pedagogy. london: state university of new york press. vidal, j. and quintanilla, m. (2000). the teaching and research relationship within an institutional evaluation. higher education, 40 (2), 217-229. vieira, f. (2002). pedagogic quality at university: what teachers and students think. quality in higher education, 8 (3), 255-272. vieira, f. (2005). pontes (in)visíveis entre teoria e prática na formação de professores. currículo sem fronteiras, 5 (1), 116-138. accessible on the journal website: http://www.curriculosemfronteiras.org. vieira, f. (2007a). reflective teacher development towards pedagogy for autonomy in schools: promoting and understanding change through pedagogical inquiry. in m. jiménez raya and l. sercu (eds.), challenges in teacher development: learner autonomy and intercultural competence (pp. 147-167). frankfurt am main: peter lang. vieira, f. (2007b). teacher development through inquiry – getting started. in a. barfield, and s. brown (eds.), reconstructing autonomy in language education – inquiry and innovation (pp. 1529). houndmills: palgrave macmillan. vieira, f., silva, j. l., melo, m. c., moreira, m. a., oliveira, l. r., gomes, c., albuquerque, p. b. and sousa, m. (2004). transformar a pedagogia na universidade: experiências de investigação do ensino e da aprendizagem. braga: universidade do minho, centro de investigação em educação. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009, pp. 22 – 35. interteach preparation: a comparison of the effects of answering versus generating study guide questions on quiz scores helen i. cannella-malone 1 , judah b. axe 2 , and edward d. parker 3 abstract: within an interteaching context, an alternating treatments design was used to compare the effects of answering versus writing study guide questions on quiz performance in a 10-week methods course in special education. results indicated quiz performance was not substantially influenced by the type of preparation, but that writing questions led to slightly higher quiz scores. on a social validity questionnaire, most participants reported a preference for answering study guide questions because that condition better prepared them for quizzes. interpretations of the results, the social validity responses, practical implications, and recommendations for future research are discussed. keywords: interteaching, study guide, college teaching i. introduction. research indicates attrition rates of approximately 20% for college freshman (consortium for student retention data exchange, 2003). examining innovations in college experiences, including teaching methods, may reduce this rate. the traditional lecture format of college teaching has been criticized for providing limited in-depth learning, having reduced access to distance learners, lacking the promotion of practical skills, and facilitating insufficient levels of student participation. one alternative to traditional lecture is electronic lectures presented on computers (stephenson, brown, and griffin, 2008; varank, 2006). this has the advantage of being accessible to distance learners and systematically programming multiple levels of bloom’s taxonomy (bloom, 1956; e.g., knowledge, comprehension, synthesis, evaluation). to address the criticism of limited provision of practical skills, researchers have evaluated models of experiential learning in college classes (li, greenberg, and nicholls, 2007). this technique has been shown to promote maintenance of interest and motivation. co-teaching in college courses can provide dual expertise such as, for example, combining a physics professor and a science educator in one classroom to prepare physics teachers (briscoe and prayaga, 2004). finally, a behavioral approach to increasing student participation in a college lecture is using response cards in which students write answers to review questions on dry-erase boards (kellum, carr, and dozier, 2001). this form of active student responding increased both participation and quiz scores. interteaching (boyce and hineline, 2002) is another variation of traditional lecture-based college teaching that utilizes principles and practices from the behavior analytic disciplines of personalized systems of instruction (keller, 1968), precision teaching (lindsley, 1964), and cooperative learning (halpern, 2004). interteaching is comprised of four components: (a) students read assigned course material and answer study guide questions prior to each class 1 the ohio state university, a348 paes building, 305 w 17 th avenue, columbus, oh 43210 2 simmons college, department of education, simmons college, 300 the fenway, room c316c, boston, ma 02115 3 the ohio state university, a315 paes building, 305 w 17 th avenue, columbus, oh 43210 cannella-malone, h., axe, j., and parker, e. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 23 session; (b) the instructor focuses a brief lecture on the concepts identified by students as being difficult from the previous class session during the first 15 min of class; (c) students then spend 30 to 40 min grouped in dyads or triads to discuss their answers to the study guide questions; and (d) finally, students complete an interteach record, which asks about the quality of the discussion and concepts with which they had difficulty. the instructor uses this information to prepare the brief lecture for the following class period. previous researchers have found interteaching to result in higher quiz scores and be preferred by college students when compared with traditional lecture-based teaching (saville, zinn, and elliot, 2005; saville, zinn, neef, van norman, and ferreri, 2006). for example, saville, zinn, and elliot compared interteaching to other traditional teaching methods in a laboratory setting. in this study, eighty-four undergraduates participated in one of four randomly assigned teaching conditions. in the interteach condition, participants were given a study guide for an article and 15 min to read the article and answer the study guide questions. they then spent 15 min discussing the article with a peer, then 15 min asking the instructor questions about the article. in the lecture condition, participants were given a 45 min lecture on the article. in the reading condition, participants were given 45 min to read the article. in each of these conditions, participants returned after one week to take a 10-question, multiple-choice quiz on the article. in the control condition, participants had no exposure to the article and took the quiz on their first, and only, visit to the lab. statistical analyses indicated significant differences between the groups, with participants in the interteach group performing significantly better than each of the other groups. in another study, saville and his colleagues (2006) conducted two experiments to compare interteaching to traditional lecture. thirty-five graduate students participated in the first experiment and 33 undergraduates participated in the second. in both experiments, an alternating treatments design was used to examine the effects of interteaching versus traditional lecture on short answer quizzes. results from both experiments indicated that quiz scores following interteaching sessions were consistently higher than those following lecture sessions. results of social validity questionnaires suggested that the majority of students preferred interteaching to lecture. the authors explained the success of interteaching in terms of active student participation in the discussions, social contingencies from peers, opportunities to earn bonus points, and guidance of the study guides towards important course material that was linked to quiz questions. however, the authors noted that a primary limitation of their analysis was that they did not identify which components of interteaching were necessary for improved quiz performance (p. 59). although the effects of prepared study guides as one component within the interteaching framework have not been examined, the effects of their use in other teaching contexts have been evaluated in a number of previous studies. for example, altus, welsh, miller, and merrill (1993) used points and fines as contingencies to increase university housing cooperative students’ completion of study guides, which improved their scores on mastery tests. flora and logan (1996) found that commercially available, computerized study guides improved the test performance of students in general psychology courses as compared with a control group who had no study guides. these authors also found that 80% of the experimental participants reported enjoying the study guides. finally, dickson, miller, and devoley (2005) suggested that study guides help students focus on critical information. although the available research has found interteaching to be an effective teaching methodology, it would be meaningful to analyze the individual components of interteaching to cannella-malone, h., axe, j., and parker, e. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 24 determine the relative effectiveness of each part of the package (cooper, heron, and heward, 2007). one component that—as an individual independent variable—has been found to have positive effects outside of the interteaching context is the study guide (e.g., altus et al., 1993; dickson et al., 2005; flora and logan, 1996). therefore, the purpose of the current study was to extend the research on interteaching by examining the effects of one component of the package, the study guide, on quiz scores in an undergraduate special education course. ii. method. a. participants, setting, and materials. seven college seniors participated in this study. all were caucasian females, between the ages of 20 and 22, completing a four-year undergraduate teacher licensure program in special education. none of the participants had any previous experience with interteaching, and all students enrolled in the course participated in this study. all participants attended each class session of the quarter. this study was conducted across eight weeks of a 10-week methods course in special education. classes were held in a classroom on the college campus that was equipped with desks, a computer, an lcd projector, a screen, and chalkboards. study guide questions were developed by the instructor for the reading assignments for four class sessions and posted on the course website at the beginning of the quarter. each study guide consisted of approximately 10 questions (i.e., two to three questions per reading) based on the reading assignments for the week. students were instructed to respond to the questions and bring their typed responses to class. weekly reading assignments (consisting of three to four research articles and book chapters) were posted on the course website or provided in a course packet that was available at the beginning of the quarter. students could access all course materials (e.g., readings, study guide questions, assignment guidelines, etc.) at any time during the quarter with no restrictions. interteach records were provided to the students during each class session (see table 1). table 1. interteach record questions date: __________ duration of discussion: ____ sufficient time provided: yes ____ no ____ on a scale of 1 (not at all) to 5 (very), how prepared was (were) your partner(s)? on a scale of 1 (very easy) to 5 (very difficult), how difficult was the material in this unit? list topics that gave you difficulty and the nature of the difficulty. list issues you would like clarified in the class lecture. provide any additional comments or suggestions. b. dependent variable. the primary dependent measure was the mean percent correct on eight weekly quizzes across participants. quizzes were worth 12 points each and had eight questions based on the weekly reading assignments. each quiz was constructed with the following types of questions: (a) two factual multiple-choice questions worth one point each (e.g., which of the following strategies is a reinforcement-based procedure?), (b) two factual fill-in-the-blank questions worth one point each (e.g., the last type of prompt in a least-to-most prompting hierarchy is typically a cannella-malone, h., axe, j., and parker, e. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 25 _________.), (c) two factual short answer questions worth two points each (e.g., describe at least two of the strategies presented in this article.), and (d) two interpretive problem solving short answer questions worth two points each (e.g., describe a strategy you would use to increase the social interactions of a child with autism and explain why you would use that strategy.). for questions worth one point, no partial credit was given. for questions worth two points, students received zero points for incorrect responses, one point for answers that were either partially correct or did not answer all parts of the question, and two points if all parts of the question were answered correctly. quiz scores were measured as the mean score out of 12 across participants. mean percent correct on individual question types—multiple choice, fill-in-theblank, factual short answer, and problem solving short answer—were also reported. c. interobserver agreement. two graders (i.e., two of the authors) independently scored 75% of the quizzes. all three authors prepared and reviewed all answers to the quiz questions. they met prior to grading each quiz to develop a grading rubric, which was used when grading all quizzes. for example, if the answer on the quiz matched the rubric, the answer was scored as correct. for multiple-choice and fill-inthe-blank questions, if the answer on the quiz did not match the rubric, it was graded as incorrect. for both the factual and problem solving short answer questions, if the answer on the quiz only partially matched the rubric, then only one (out of two possible) points was given. data from the two graders were compared for agreements and disagreements. an agreement was scored if the two graders assigned the same score to an answer. any discrepancy between the two graders was counted as a disagreement. interobserver agreement was calculated on a question-by-question basis using the formula: (agreements/(agreements + disagreements)) x 100%. agreement on the different types of questions was variable. interobserver agreement was 100% for the multiplechoice questions, 95% (range 78.5–100%) for the fill-in-the-blank questions, 83% (range 64– 100%) for the factual short answer questions, and 75% (range 57–100%) for the problem solving short answer questions. overall agreement for the quizzes was calculated to be 88% (range 82– 100%). d. experimental design. an alternating treatments design was used to examine the effects of the two preparation formats on quiz performance (kazdin, 1982). the study guide assignments were randomly assigned to class sessions, and each preparation condition was provided four times over the course of the quarter. e. interteaching procedures. each class session occurred once per week and lasted 2 hr and 18 min and was broken into the following schedule: 5 min to address administrative issues, 25 min small group discussions, 5 min to complete the interteach record, 10 min break, 30 min lecture, 20 min quiz, 30 min discussion of practical teaching techniques and discussion of practicum experiences. similar to saville and his colleagues (2006), participants were randomly assigned to either of two dyads or a triad during each class session. they were then given 25 min to discuss either their responses to the provided study guide questions or the questions they had prepared. during the first 10 min, the participants discussed the material independently. for the remaining 15 min, two graduate cannella-malone, h., axe, j., and parker, e. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 26 teaching assistants circulated through the three groups assisting with any questions and helping guide the discussion. participants were given 25, rather than the traditional 35, min to discuss material because this was the amount of time it took them to discuss all of the material for that day. after 25 min, their discussions would shift to other, unrelated topics. at the conclusion of the discussion, participants were given 5 min to complete an interteach record, which provided them with the opportunity to request further clarification on particular topics as well as rate the quality of their discussions (see table 1). participants were then given a 10 min break, during which time the instructor and two graduate teaching assistants reviewed the interteach records to determine what material to cover during the short lecture. following the break, participants were given a 30 min lecture on the material, with an emphasis on those topics that they found difficult. unlike a traditional lecture, the 30 min lecture was structured as a group discussion, with students asking and responding to questions. although each class session was 2 hours 18 min, only 1 hour 20 min were devoted to the material for that week. answering questions. in the answer questions condition, participants were required to download a study guide from the course website and provide answers to each question prior to class. each study guide consisted of approximately 10 interpretive questions that required the participants to think critically about the material and could not be answered by simply skimming the reading (e.g., why are functional skills for daily living important to teach? provide 5 examples of functional skills for daily living and how each benefits a student.). participants answered questions for four randomly assigned class sessions, and were required to bring a hard copy of their responses to class to use during discussion. participants earned points for completed study guides, and all participants completed each assignment. writing questions. in the write questions condition, participants were required to develop their own study guide questions based upon the readings for that week, covering points in the reading that they thought were important or presenting issues that they did not understand. participants were instructed to write interpretive questions, rather than questions that could be answered by simply skimming the reading, and a model was posted to the course website. participants were required to write at least four questions per reading assignment for four randomly assigned class sessions and were required to bring a hard copy of their responses to class to use during discussion. participants earned points for completed study guides, and all participants completed each assignment. f. social validity. on the last day of the class, each participant anonymously completed an eight-item questionnaire consisting of questions with likert scale, open ended, and yes/no questions. the questionnaire examined the participants’ preferred format of interteach preparation. (see table 2 for the social validity questionnaire.) cannella-malone, h., axe, j., and parker, e. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 27 table 2. social validity questionnaire. 1. do you prefer preparing for interteaching by answering provided study guide questions or writing your own questions? please list reasons for your preference. mostly preferred preparing questions somewhat preferred preparing questions preferred both equally somewhat preferred answering prepared questions mostly preferred answering prepared questions 2. please rate the extent to which you feel you learned using the interteach method with answering provided study guide questions and writing your own questions. please explain why you feel this way. what are some advantages and disadvantages that you experienced with each type of preparation (answering prepared questions and writing your own questions)? learned much more with writing questions learned some more with writing questions learned equally with writing & answering questions learned some more with answering prepared questions learned much more with answering prepared questions 3. how would you rate the extent to which you learned using the interteach method in this course as compared to courses that you have taken that did not use the interteach method? learned much more with interteach learned some more with interteach learned equally with and without interteach learned some more without interteach learned much more without interteach 4. what aspects of the interteach method would you judge as useful? (open ended) 5. what aspects of the interteach method would you judge as not useful? how would you change these aspects? (open ended) 6. what other feedback do you have about the structure of this class? (open ended) 7. would you prefer to participate in interteach in another course? (yes/no) 8. would you recommend a class that uses the interteach method to a fellow student? (yes/no) cannella-malone, h., axe, j., and parker, e. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 28 iii. results. a. quizzes. the mean percent correct on quizzes across participants is displayed in figure 1. on quizzes 1, 3, 6, and 7, which occurred in the write questions condition, the class averaged 81% correct (range 62–93%). on quizzes 2, 4, 5, and 8, which occurred in the answer questions condition, the class averaged 82% correct (range 73–87%). the last two mean quiz scores in the write questions condition were slightly higher (i.e., 93% and 90%) than the last two mean quiz scores in the answer questions condition (i.e., 83% and 87%). figure 1. overall mean percent correct across eight weekly quizzes. figures 2 through 5 display the mean percent correct on quizzes in the two conditions for each question type. on multiple-choice questions (figure 2), participants averaged 86% correct (range 57–100%) in the write questions condition and 79% correct (range 57–100%) in the answer questions condition. on fill-in-the-blank questions (figure 3), participants averaged 81% correct (range 79–86%) in the write questions condition and 88% correct (range 71–100%) in the answer questions condition. on factual short answer questions (figure 4), the average was 80% correct (range 57–93%) in the write questions condition and 88% correct (range 71–100%) in the answer questions condition. on problem solving short answer questions (figure 5), the average was 81% correct (range 61–93%) in the write questions condition and 75% correct (range 61–86%) in the answer questions condition. mean quiz scores on problem solving short answer questions were consistently higher in the write questions condition. cannella-malone, h., axe, j., and parker, e. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 29 figure 2. mean percent correct on the multiple-choice questions across eight weekly quizzes. figure 3. mean percent correct on the fill-in-the-blank questions across eight weekly quizzes. cannella-malone, h., axe, j., and parker, e. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 30 figure 4. mean percent correct on the short answer questions across eight weekly quizzes. figure 5. mean percent correct on the problem solving questions across eight weekly quizzes. b. social validity. based on responses to the social validity questionnaire, the majority of the participants preferred answering prepared questions, citing improved understanding of the material and feeling that these prepared questions would be on the quiz as reasons for this preference. the two participants who preferred preparing questions stated that writing questions helped them learn more information because it took less time and therefore they could spend more time reading and analyzing the articles. they also reported that coming to class with prepared questions was better because “everyone wasn’t regurgitating the answers to all the same questions”. the majority of the class reported a non-preference for interteaching, stating that they preferred whole class, rather than small group, discussions. these responses were similar in nature to the feedback provided on the weekly interteach records. while most participants did not like interteaching, three out of seven said that they would both take and recommend another course that used interteaching. cannella-malone, h., axe, j., and parker, e. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 31 iv. discussion. this research extends the literature on interteaching as a college teaching format (saville et al., 2005; 2006). quiz performance was not substantially influenced by the type of interteach preparation—answering study guide questions or writing one’s own questions. the mean percent of correct answers to quiz questions across the two conditions differed by only one percent. however, visual inspection of the data in the alternating treatments design suggests that the write questions condition produced slightly higher quiz scores by the end of the course. examining quiz performance across particular question type in the two conditions suggests some further findings. participants scored slightly higher on multiple-choice questions in the write questions condition and performed substantially higher on fill-in-the-blank questions on two quizzes in the answer questions condition. on factual short answer questions, there were slightly higher scores in the answer questions condition and on the problem solving short answer questions there were consistently higher scores in the write questions condition. differences in quiz scores were examined individually for each student (data not shown) and were not different from aggregate differences. these results are variable and inconsistent and future research is required to substantiate any proposed findings. nevertheless, potential explanations of the data are offered. one interpretation of the data is that the type of interteach preparation—answering study guide questions or generating one’s own questions—is a weak variable in the interteaching package and does not influence quiz performance in a college course that uses interteaching. on the other hand, it may be that having any kind of study guide requirement, either answering provided questions or developing questions, could be too closely related to show significant differences. additionally, it may be the case that the interteach discussion or the interteach record are more influential variables in the package and that their potency as independent variables should be studied. it may also be the case that, especially in a college course with a small number of students, the individual components of interteaching are not influential and only the contingencies for quiz performance (i.e., one’s grade) influence preparation for class and performances on quizzes. another plausible explanation for the insubstantial difference across the conditions is that quiz scores may be an insensitive dependent measure for these independent variables. that is, in evaluating preparation type for interteaching, more sensitive dependent variables might be the quality of interteach discussion, maintenance of the information over months or years, or clinical applications of the concepts from the course. these dependent variables are more labor-intensive to measure but might be considered in future studies. despite the small differences across the two conditions, some plausible explanations for these differences are offered. the most consistent results were that writing questions produced higher scores on short answer problem solving questions, which required participants to synthesize and apply the concepts in the readings to educational problems and might require greater critical thinking. it is possible that providing students with a study guide with questions for students to answer that indicate the important parts of a reading may limit their critical thinking. a student’s critical thinking might be enhanced when they are required to generate their own questions based on what they consider to be important (lampert, 2007). conversely, because study guides indicate to students the critical information in text, they might help students perform well on factual type questions and this was the case in the fill-in-the-blank and short answer questions. however, because writing questions produced slightly higher scores on multiple-choice questions, another type of factual question, this interpretation is weakened. cannella-malone, h., axe, j., and parker, e. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 32 a. limitations. a number of limitations should be considered. first, only seven students participated in this study. the classes in the two experiments conducted by saville et al. (2006) had 33 and 35 students. because interteaching is conceptualized as an alternative to lecture-based courses, which typically have significantly more than seven students, interteaching might be more effective with larger classes. based on the results from the social validity questionnaire, a second limitation of the study is that the participants were generally not in favor of the interteaching format. this non-preference might have been especially strong with these participants, because they had taken two courses with the same instructor in previous quarters in which interteaching was not used. although participants indicated on their interteach records that they did not like the interteach arrangement, deviations would have compromised the internal validity of the study. the added course requirements over previous course requirements (including in-class discussions and more difficult quizzes) also might have influenced the non-preference for interteaching and affected the results of the experimental comparison. a third limitation is that quiz scores might be an insensitive dependent variable to evaluate the effectiveness of the preparation conditions and other possible dependent variables such as quality of discussion and clinical applications of the concepts were not measured. a final limitation is that interobserver agreement for the problem solving questions was low. this may be due to the fact that the students had to discuss how they might use techniques in real-life settings, so there were no discrete correct answers. in future studies, it might be beneficial for the different graders to further discuss how they might answer the questions in order to develop a more comprehensive rubric for this type of question. b. suggestions for future research. to address these limitations, there are at least four avenues for future research. first, this study should be replicated with classes of larger size (e.g., 30 students) at both the undergraduate and graduate levels to continue to evaluate the differences between the two preparation formats. second, future studies should compare the use of study guides with no study guides and no other contingencies for preparation. this would provide a clearer evaluation of the effects of the study guide. third, other dependent variables should be measured. these might include a measure of the quality of class discussion, quizzes with a higher percentage of problem solving and application questions, the clinical application of concepts, the designing of educational goals based on the concepts, and orally explaining concepts and applications. a fourth area for future research is to evaluate the effects of the other components of interteaching such as the in-class discussion or the interteach records. data on the effects of these individual components may provide a more complete account of the construct of interteaching. c. implications for practice. despite the methodological limitations of this study and the need for further research, some implications for practice are suggested. based on this and previous studies, it appears that interteaching is a reasonable alternative to lecture-based formats of college teaching that provides many benefits to both students and instructors as outlined by boyce and hineline (2002) and saville et al. (2006). although there were not substantial differential effects of the two preparation strategies evaluated in this study, one conclusion is that contingencies for any cannella-malone, h., axe, j., and parker, e. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 33 type of class preparation, including reading text and writing about it, help students learn content. further, participants may perform better on quizzes that require critical thinking if they critically examine course reading and identify areas of importance. finally, until there is more component analysis data, interteaching should be implemented as described by boyce and hineline (2002) and saville and colleagues (2005; 2006). d. conclusions. in conclusion, answering study guide questions or writing one’s own questions based on course reading did not substantially differentially affect quiz scores in a college course using interteaching. it is possible that generating one’s own questions produces higher scores on problem solving questions that require critical thinking, but future research is needed to support that conclusion. interteaching can be an effective, behavior analytic alternative to lecture-based teaching and should continue to be researched. appendix appendix 1. sample study guide questions provided by the instructor. 1. what are the differences between discrete trial training and embedded instruction? describe at least three advantages and disadvantages of each. 2. discuss the importance of examining the effectiveness of an intervention that is implemented by the classroom teacher and his/her paraprofessionals? 3. how was errorless learning used in this study? would you use this strategy in your classroom? why or why not? appendix 2. sample study guide questions written by the participants. 1. how was the functional analysis conducted and what were the results? the article does not go into very much detail of this aspect of the experiment. it is not as important as what was done or the results, but it would have been nice to mention. 2. what is an analogue session? i have never heard of this term before. 3. how/why was each of the six assessments chosen? what was each trying to show? 4. describe the graph, which depicts the results and what it means for the experiment? references altus, d. e., welsh, t. m., miller, l. k., and merrill, m. h. (1993). efficacy and maintenance of an education program for a consumer cooperative. journal of applied behavior analysis, 26, 403–404. bloom, b. s. (1956). taxonomy of educational objectives. book 1. cognitive domain. new york: longman. boyce, t. e., and hineline, p. n. (2002). interteaching: a strategy for enhancing the user friendliness of behavioral arrangements in the college classroom. the behavior analyst, 25, 215– 225. cannella-malone, h., axe, j., and parker, e. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 34 briscoe, c., and prayaga, c. s. (2004). teaching future k-8 teachers the language of newton: a case study of collaboration and change in university physics teaching. science education, 88, 947-969. consortium for student retention data exchange (2003). 2002–2003 consortium for student retention data exchange (csrde) report for 1995–2001. black issues in higher education, 20, 30–31. cooper, j. o., heron, t. e., and heward, w. l. (2007). applied behavior analysis (2 nd ed). upper saddle river, nj: merrill/prentice hall. dickson, k. l., miller, m. d., and devoley, m. s. (2005). effect of textbook study guides on student performance in introductory psychology. teaching of psychology, 32, 34–39. flora, s. r., and logan, r. e. (1996). using computerized study guides to increase performance on general psychology examinations: an experimental analysis. psychological reports, 79, 235– 241. halpern, d. f. (2004). creating cooperative learning environments. in b. perlman, l. i. mccann, and s. h. mcfadden (eds.), lessons learned: practical advice for the teaching of psychology (vol. 2, pp.149–155). washington, dc: american psychological association. kazdin, a. e. (1982). single-case research designs: methods for clinical and applied settings. new york, n.y.: oxford university press. keller, f. s. (1968). good-bye teacher… journal of applied behavior analysis, 1, 79–89. kellum, k. k., carr, j. e., and dozier, c. l. (2001). response-card instruction and student learning a college classroom. teaching of psychology, 28, 101-104. lampert, n. (2007). critical thinking dispositions as an outcome of undergraduate education. journal of general education, 56, 17–33. li, t., greenberg, b. a., and nicholls, j. a. f. (2007). teaching experiential learning: adoption of an innovative course in an mba marketing curriculum. journal of marketing education, 29, 25-33. lindsley, o. r. (1964). direct measurement and prosthesis of retarded behavior. journal of education, 147, 62–81. saville, b. k., zinn, t. e., and elliot, m. p. (2005). interteaching versus traditional methods of instruction: a preliminary analysis. teaching of psychology, 32, 161–163. saville, b. k., zinn, t. e., neef, n. a., van norman, r., and ferreri, s. j. (2006). a comparison of interteaching and lecture in a college classroom. journal of applied behavior analysis, 39, 49–61. cannella-malone, h., axe, j., and parker, e. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 35 stephenson, j. e., brown, c., and griffin, d. k. (2008). electronic delivery of lectures in the university environment: an empirical comparison of three delivery styles. computers & education, 50, 640-651. varank, i. (2006). a comparison of a computer-based and lecture-based computer literacy course: a turkish case. eurasia journal of mathematics, science, and technology education, 2, 112-123. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009, pp. 36 – 41. experiential learning in the scholarship of teaching and learning ruth benander 1 abstract: experiential learning is making meaning from direct experience. experiential learning is used in workplace training (silberman, 2007), and is the theoretical foundation for all practicum and co-op program learning. supported by findings on expert/novice learning styles (boshuizen, bromme, and gruber, h., 2004), this essay reflects on the practice of experiential learning as a mode of inquiry in the scholarship of teaching and learning. when expert teachers experiment with becoming novice learners, or when professors become students, they can come to personal, enduring insights about the experience of teaching and learning. keywords: experiential learning, ethnography, faculty development. for a faculty member who has been teaching for many years, taking a class as a novice learner can be a valuable form of reflection on the teaching and learning interaction. the scholarship of teaching and learning promotes inquiry into the learning and teaching process, and different disciplines offer alternative ways of making this inquiry. one such alternative is experiential learning. experiential learning, as comprehensively outlined by kolb (1984), emphasizes reflection on direct, concrete experience. in the case of teaching and learning, the critical reflection involves the expert teacher being a novice student. experts negotiate the learning experience differently from novices (boshuizen, bromme, and gruber, 2004; daly, 1999). a teacher who is an expert in his or her discipline can gain teaching insights from revisiting the novice learner experience. much scholarship of teaching and learning employs observation of learning and critical reflection on teaching by those engaged in teaching (martin, benjamin, prosser, and trigwell, 1999). another method of inquiry that underlies this scholarship is the experimental method or adaptations of this method to the classroom environment. experiential learning lends another method of inquiry to observation and experimentation, one that extends self-reflection on teaching and learning from the point of view of the teacher to that of the learner. the shift of perspective allows the observer to see the process of learning through a new community of practice, that of the novice learner. experiential learning involves participation and critical reflection on that participation (kolb, 1984; shon, 1983). experiential learning is most often invoked in workplace adult learning, but is also branching out into more formal education venues (fenwick, 2000; silberman, 2007). in general terms, experiential learning conceives of the adult learner as participating in an activity, then reflecting on the activity to make generalizations that he or she can then apply in new situations. critics have also extended the concept to include perception of and participation in power relations, personal motivation, and social processes (michelson, 1996; ellsworth, 1997). one of the specific values of experiential learning is the immersion of the participant in the social constructions and cultural expectations specific to the experience (lave and wegner, 1991). when a professor becomes a student, and enters into the classroom in that 1 department of english and communication, university of cincinnati raymond walters college, 9555 plainfield rd., cincinnati, oh 45236, ruth.benander@uc.edu. benander, r. journal of the scholarship of teaching and learning, vol. 9, no. , june 2009. 37 role, the experience will be different from the professor entering the classroom as a researcher to observe. part of the qualitative shift of perspective for a professor becoming a student as a way to gain insight into teaching and learning involves the shift from being an expert to being a novice. as experts in their fields, professors forget the confusion of novelty that students experience in classes. it is well documented that novices and experts have different strategies for negotiating problems, and it is insightful to be a novice and re-visit the novice strategies that an expert may have forgotten. boshuizen, bromme and gruber (2004) document these differences at length. they observe, “experts do not just know more than novices, they also have a different way of structuring their domain-specific knowledge” (p. 6). experts have a different orientation not only to their subject matter, but also learning about their subject matter. daley (1999) observes that novices need direction, rely on roles, are overwhelmed and anxious, and their learning is impeded by refusals to provide directions and clear roles. thus, a professor might be irritated by an anxious student’s question, “what do you want in this assignment,” while from a novice’s point of view, this is an essential question. daley outlines that experts are self-creative, collaborative, and able to sift out important issues. the expert teacher would like to guide his or her students to this kind of orientation to learning but needs to build up to it. reminding oneself of what it feels like to be a novice can provide important insight to help create the structured experiences required to help students move out of the novice state. additionally, it can help professors avoid attributing student behavior to disinterest or incompetence. a difficulty with reading published accounts of experiential learning is that these accounts can be perceived as idiosyncratic, subjective, and informal. at worst, these accounts can be seen as unethical. an example of a controversial episode of a professor becoming a student is the experience of cathy small, who wrote under the pseudonym rebekah nathan (2005). using the participant/observation methodology of anthropology, cathy small provides an account of returning to the classroom as a student after many years as a faculty member in her book, my freshman year: what a professor learned by becoming a student. when the book was published, it was met with both praise and stiff criticism. the book was praised as an insightful and realistic appraisal of the freshman experience (donovan, 2008). it was also criticized as ethically suspect. a reviewer for the american ethnological society observes that “…she never achieved a student identity and … the revelation of her “secret” always produced difficulties” (ethnoadmin, 2006, ¶2). the criticism of small “deceiving” the academic community by shifting roles raises a serious consideration. it is possible that a professor becoming a student can be viewed as suspect by colleagues and students on the home campus. if the goal of the professor is to better understand the experience of being a student, in whatever guise that is possible given the professor’s “dual citizenship”, then deception is not involved. in fact, it is not necessary that a professor become a student on his or her home campus. a person can become a student in many contexts where the potentially conflicting roles of professor and student do not compete. the important issue is for a person who is an expert in his or her field to become involved in a class where he or she is a novice learner and then reflect on that learning process. small (nathan, 2005) reports a number of observations about student life including the pressures of non-academic activities, the collegial code of silence where student do not speak in class so as not to embarrass their friends, and the pragmatic approach to read on a need-to-read basis. however, small comments that one of the main lessons she learned was increased compassion for her students. small asserts that she knew the facts of the situations she became benander, r. journal of the scholarship of teaching and learning, vol. 9, no. , june 2009. 38 involved in from her experiences as a faculty member, but not the contextualized reasons for them, not the emotional backdrop of the decisions students made. for example, small discusses one epiphany she experienced about student reading. she recalls chuckling as a professor put a web address up on the board for a reading assignment that, as a student, she didn’t even copy down because she knew she would not be doing that reading as it was not going to be discussed in class. small reports that she has changed how she assigns readings in her own classes, as a result of her time as a student, by explicitly linking all reading to specific purposes of class discussion and assessments. moving from professor to student is a significant change in relationships of social power, but the change from expert to novice is more personally and emotionally affecting. a group of faculty at a mid-western university came together in a faculty learning community to reflect on how becoming beginning music students affected their teaching (benander and pettit, 2007). these faculty members commented on the challenge of feeling incompetent, the importance of praise, and the need for structured clarity in exercises to gain the facility to improve. these faculty commented that by reflecting on their experiences and needs as novices in learning a new skill, it helped them change the structure of their introductory courses to make the structure of assignments and activities clearer; to create assignments that helped students move from being dependent on rules to slowly begin to learn to apply principles; and to create activities that would provide experiential background knowledge to be applied in new situations. of course these are points that have been made in the pedagogical literature, but as mcglinn (2003) observes, “these [changes in] behaviors don’t come when someone else tells the student teacher what is needed; rather they come from the struggle to understand teaching and students’ learning. they come from reflection”(p.147). mcglinn may be referring to student teachers in her article, but the observation holds for experienced faculty members as well. reading an article about structuring assignments for clarity may not result in teaching change as effectively as personally experiencing the novice’s confusion when faced with a complicated assignment. for a professor to reflect on the process of teaching and learning, it may be more practical to find a learning experience outside the academy where the role of “professor as student” is not pertinent. however, it is also possible to engage in experiential learning on one’s home campus. if one has a collegial relationship with another instructor, it is possible to join a class as a student. it is not necessary to hide that one is a professor in another department. i have attended foreign language and fine arts courses taught by my colleagues. the experience of taking these classes and becoming part of the group of students trying to understand what is going on has profoundly changed how i structure the classes i teach, how i give feedback on student work, how i design assignments, and how i deal with silence during class discussions. as a novice student in a foreign language course, i often did not understand how different topics were related to each other. it was very helpful when the professor would explicitly explain why we moved from one topic to the next. i try to make the structure of my own classes very clear with explicit explanations of how the assignments and topics we cover build on each other. equally explicit feedback was also helpful. as a novice artist, the teacher would quickly indicate what was wrong with my drawing and move on. the instructor’s feedback confused me because now i knew what was wrong, but i would have also liked to know how i was supposed to fix it. as i became a more experienced artist, i came to know how to fix problems when the professor identified them. in my own first year classes, i try to use feedback methods that allow for detailed suggestions to help novice students acquire the tools they need to make corrections for benander, r. journal of the scholarship of teaching and learning, vol. 9, no. , june 2009. 39 themselves. as a way to continue to guide novices to self-sufficiency, i try to design assignments that build on each other and which allow for continued practice of new skills. finally, i am very much aware of how silence in a classroom can result from novices not knowing how to answer the question, or not knowing enough to even form a question of their own, but also from a desire not to betray each other by participating. i try to create participation scenarios that allow students to help each other participate, have enough time to answer, or to consult with each other before answering. the previous examples of professors learning by becoming students demonstrate how experiential learning is not just for students but can be a valuable tool to reflect on teaching and learning. the attempt to shift roles poignantly emphasizes the social power that affects teaching and learning. participation in the experience of the classroom highlights the emotional backdrop of how students make decisions about their learning. professors who critically reflect on their own learning experiences may be able to take advantage of the idiosyncratic, subjective, and informal nature of this style of learning to understand how his or her personal style of learning responds to different kinds of teaching. as mcglinn (2003) points out, adult learners do not like to be told what to do, and boshuizen, bromme, and gruber (2004) suggest that experts prefer to use resources and previous experience to inform their practice. the challenge with experiential learning for professors is that it is a calculated risk to give up the capability of the expert and the power of the professor. the implication of experiential learning as a form of critical reflection for the scholarship of teaching and learning is that is adds another mode of inquiry. more traditional inquiry in teaching and learning involves observing from the position of the professor or as a nonparticipant. alternatively, researchers institute interventions and controls to assess the effectiveness of a learning activity. becoming a participant in the experience of learning, and engaging reflective practice in as a participant in the community of learners, offers insights not available to the other two modes of inquiry. in addition, being a student in the early 21 st century is radically different from what it meant to be a student for many current midor late-career faculty members. it is neither necessary nor advisable to duplicate the experience of cathy small to use experiential learning to reflect on one’s practice of teaching and learning. the experience of being a learner can be found in many contexts. the greatest change as a result of changing roles often seems to be change of emotional orientation. faculty who have participated in this role reversal comment that the greatest change was increased compassion for their students (e.g. nathan, 2005; benander and pettit, 2007). mid-career faculty have long experience with the many reasons why a person can be late for class or miss an assignment. the resulting “compassion fatigue” can sometimes lead to strained relationships with students. renewing empathy for the novice learning experience can help create more positive interactions with the recurring difficulties instructors must negotiate with their students. it is also refreshing to have a new perspective on learning, assignments, and technology. for example, many instructors only use the instructor view of software like web-ct or blackboard. negotiating assignments and quizzes through the student view of the electronic interface can help one anticipate student challenges. instructors who have returned to being students in one context or another, whether it is attending a course in another discipline or learning a new musical instrument, seem to report lasting and deep changes in their teaching as result of their critical reflections on being a learner. silberman (2007) comments that experiential learning is a “sticky” learning: “when it is done benander, r. journal of the scholarship of teaching and learning, vol. 9, no. , june 2009. 40 well, it adheres to you. participants will usually forget a great presentation, but they often remember a great experience” (p. 4). when experts revisit the experience of being a novice, they may gain insights on how to help their novice students problem solve more like experts. in addition, the recommendations from research in the scholarship of teaching and learning may “stick” and be more readily applied in the classroom, resulting in enduring and real change in individual practice. references benander, r. and pettit, r. (2007). when teachers are learners: case studies in learning outside our discipline from the overthe-hill-band, association of university regional campuses of ohio annual conference, bowling green state university, firelands college, huron, ohio. boshuizen, h., bromme, r., and gruber, h. (2004). introduction: on the long way from novice to expert and how traveling changes the traveler. in h. boshuizen, r. bromme and h. gruber (eds.), professional learning: gaps and transitions on the way from novice to expert (pp. 3-8). dordrecht, de: kluwer academic publishers. daley, b. (1999). novice to expert: an exploration of how professionals learn. adult education quarterly, 49 (4), 133-147. donovan, n. (2008). review of my freshman year: what a professor learned by becoming a student. education book review, retrieved dec 29, 2008, from http://staff.lib.msu.edu/corby/reviews/posted/nathan.htm ellsworth, e. (1997). teaching positions: difference, pedagogy, and the power of address. new york: teachers college press. ethnoadmin (1006) review of my freshman year: what a professor learned by becoming a student. the american ethnologist online, 33, retrieved dec 29, 2008, from http://aesonline.org/3626 fenwick, t. (2000). expanding conceptions of experiential learning: a review of the five contemporary perspectives on cognition. adult education quarterly, 50 (4), 243-272. kolb, d. (1984). experiential learning: experience as the source of learning and development. englwood cliffs, nj: prentice hall. lave, j. and wegner, e. (1991). situated learning: legitimate peripheral participation. new york: cambridge university press. martin, e., benjamin, j., prosser, m., and trigwell, k. (1999). scholarship of teaching: a study of the approaches of academic staff. in c. rust (ed.), improving student learning: improving student learning outcomes (pp. 326-331). oxford: oxford centre for staff learning and development, oxford brookes university. benander, r. journal of the scholarship of teaching and learning, vol. 9, no. , june 2009. 41 michelson, e. (1996). usual suspects: experience, reflection, and the (en)gendering of knowledge. international journal of lifelong education, 15 (6), 438-454. nathan, r. (2005). my freshman year: what a professor learned by becoming a student. ithaca, ny: cornell university press. schon, d. (1983). the reflective practitioner. new york: basic books. silberman, m. (2007). the handbook of experiential learning. san francisco, ca: john wiley and sons. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009, pp. 42 – 56. a single conversation with a wise man is better than ten years of study: a model for testing methodologies for pedagogy or andragogy bryan taylor1 and michael kroth2 abstract: this article creates the teaching methodology instrument (tmi) to help determine the level of adult learning principles being used by a particular teaching methodology in a classroom. the instrument incorporates the principles and assumptions set forth by malcolm knowles of what makes a good adult learning environment. the socratic method as used in law schools is applied to the instrument to determine whether it displays more pedagogical or more andragogical principles. keywords: andragogy, pedagogy, adult learning, adult education, socratic method. i. introduction. in his seminal treatise “the common law,” oliver wendell holmes stated that the most important thing in the life of the law is “experience” (holmes, 1881). usually the first opportunity for students to gain legal experience is in law school. for many law schools, students are hoping to gain experience which they will be able to apply to their real world experience. law programs tend to focus on the education of adults. law students usually have already obtained undergraduate degrees and may have real world experience already. once they arrive on the doorsteps of a law program they often want to gain additional experiences and knowledge to further their careers and lives. eduard lindeman, a seminal figure in the field of adult education, expressed adamantly that adults need to learn through experience (knowles, 1980, 1984a; lindeman, 1926). “too much of learning consists of vicarious substitution of someone else’s experience and knowledge” (lindeman, 1926, p. 6). lindeman (1926) further went on to write “[p]sychology teaches us that we learn what we do…experience is the adult learner’s living textbook” (pp. 6-7). a question emerges: do the teaching styles used in law schools focus on providing students with experience that will adequately prepare them for their professions? to answer this question teaching methodologies must be examined and evaluated according to the principles of adult learning. over the past century, professors of law programs have used a variety of delivery tools of instruction. although many methods of instruction exist, some focusing more on pedagogical principles (teacher-focused education) and others more on andragogical principles (learnerfocused education), there is usually some of each in each methodology (conner, 2004; knowles, 1984a, 1984b; knowles, holton, and swanson, 1998). the amount of what type of techniques is usually determined by the instructor. 1 department of adult and organizational learning, university of idaho 322 e. front 1115 albany street, caldwell, id 83605 bryantaylor@vandals.uidaho.edu. 2 department of adult and organizational development, university of idaho 322 e. front street, boise, id 83702. mkroth@uidaho.edu. taylor, b. and kroth, m. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 43 the purpose of this article is to create an instrument to test a teaching methodology and determine whether it adheres more to pedagogical principles or andragogical principles. the instrument will be based on malcolm knowles’ checklist of whether education and learning is more information based (pedagogical model) or experience based (andragogical model) (saunders, 1991). an instrument is needed to establish empirical evidence related to the andragogical theory. once the instrument is created a particular teaching methodology, the socratic method in law schools, will be used to test the model. ii. why use a law school methodology? law school education has become a very controversial subject in the legal community (maccrate, 2004). the american bar association released what is now known as the maccrate report in 1992. the report’s primary conclusion was that there remains a continuing disconnection between what law school teaches and what “students need to learn for initial competence in practice” (elson, 1994, p. 363). in recent years, law schools have been making several adjustments in an effort to improve the quality of education their students receive. these adjustments include restructuring curriculums (bodie, 2006; kozinski, 1997), emphasizing scholarship that would be beneficial to the legal community (edwards, 1992), and identifying and naming the skills and traits necessary for the successful practice of law so that they may be taught to students (chanen, 2007; denney, 2007). trying to decipher the literature on reforming legal education is difficult because there are so many different philosophies of legal education for different fields of law. the literature is very limited in empirical evidence and without evidence, law schools are reluctant to incorporate proposals such as the maccrate report (elson, 1994). since law students are adults, the failure of gaining “experience” may be a reason for their struggles upon entrance into the practice of law. if so, determining whether “experience” is gained through current teaching methodologies for instructing adults in law school becomes a very important issue. over the past century, law professors have used the socratic method as the predominant delivery tool of instruction. although many methods of instruction exist, the socratic method appears to have gained the most attention in the academic literature (kerr, 1999). one lens to look through to see if this method is providing the “experience” needed is the theory of andragogy. a common theme found in the literature is the incorporation of “adult learning” principles in order to create an “experience” that is valuable to students. currently the “pedagogical model employed by legal educators is designed to give students a set of experiences that will prepare them to adequately and effectively identify, classify and address legal problems once they pass the bar examination and enter the professional practice” (butleritchie, 2003, p. 43). the expectations of the legal profession and the practice of law are completely distinct, however, from what is learned in this law school “experience” (elkins, 1996). a common doctrinal position of what legal education is intended to do was clearly stated by one law school professor: “legal education…is the place where you find out what law is, the place you begin to distinguish from what barstool pundits profess” (elkins, 1996, p. 518). this paper will be divided into four sections. the first section will examine what the socratic method is and its application in legal education. the second section will compare andragogical and pedagogical types of learning and the model to be used for andragogical application. the third section will be the application of the andragogical model to the socratic taylor, b. and kroth, m. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 44 method. finally, the fourth section will analyze the findings of whether or not the socratic method incorporates adult learning principles. iii. let’s have a dialogue: the usage of the socratic method in law school. when one thinks of the socratic method, a vision of professor kingsfield from the movie the paper chase comes to their mind. professor kingsfield exemplified the socratic method’s use in law school, he would call upon a student to brief an assigned case, but would interrupt the student with questions until the student became so helplessly confused that the student’s selfesteem was completely demoralized (warkentine, 2000). most doctrinal law professors believe that the socratic method of instruction helps students to “think like a lawyer,” with the goal of the process to provide students skills to analyze a case on their own (niedwiecki, 2006). the socratic method has been an accepted fixture in law schools since christopher columbus langdell began instructing law students in the 1870s (friedland, 1996). the name comes from plato’s socratic dialogues (1987), in which a dialectic method of inquiry is established between two individuals asking questions surrounding a central issue. plato contends that socrates was convinced that the human mind in its normal condition discovers certain truths through its own energies, provided one knows how to lead it and stimulate it (compayré and payne, 1970). the socratic method in adult education “involves the use of systematic questions, inductive thinking, and the formulation of general definitions” (paraskevas and wickens, 2003, p. 6). the legal community lacks a commonly accepted definition as to what constitutes a socratic style. different professors apply their own versions of the socratic method, since most of them have never been trained in the actual instructional method that the socratic dialogues sets forth. for the purposes of this paper, the following definition will be adopted: the socratic method is a: “pedagogical technique in which a teacher does not give information directly but instead asks a series of questions, with the result that the student comes either to the desired knowledge by answering the questions or to a deeper awareness of the limits of knowledge” ("socratic method," n.d.). the socratic method incorporates other preferred methods of instructions such as the “case study” or “case law” method, lecture, and small groups. of these, the case study method is linked to the socratic method, compared to other methodologies. its design enables law professors to instruct and train a large class of students; and the discussion demonstrates the “logic, language and context of legal decision-making” (butleritchie, 2003). in a socratic method-based class, students perform in a “variety of thought-demanding ways to explain, muster evidence, generalize, apply concepts, analogize, and represent in a new way” (paraskevas and wickens, 2003, p. 6). in law school students are given an appellate case or a series of appellate cases to read. after they read these cases, they return to the classroom to participate in a discussion with the professor. the professor calls upon a student to engage in a dialogue with the professor what she has learned, why the courts came to decisions that they did, and how the legal rules adopted in the cases are important to the greater whole of that body of law (i.e. contracts, torts, property, etc.). the professor may either focus on one student by asking the series of questions, or may randomly jump from student to student with the questions. this process is known as the socratic method. in using this method two particular strategies help the professor engage the students and taylor, b. and kroth, m. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 45 later enable them to get rid of their cognitive egocentrism: the elenchus and inspiring aporia (paraskevas and wickens, 2003). the mechanism of the elenchus works by probing each response from a student, and examining whether the entire set of beliefs held by that student is mutually consistent. the elenctic questioning aims at preparing the students not simply to replace passively their existing knowledge with new, but to actively pursue a new learning experience which is vital in the legal education process (paraskevas and wickens, 2003). as stated in vlastos, the natural outcome of the mechanism of the elenchus is aporia, or confusion, because the socratic inquiry never reaches the absolute truth (vlastos, 1980). gregory vlastos (1983), in his article the socratic elenchus, articulates four steps that should be taken in the application of the socratic method. these steps, in the hands of a skilled instructor, would occur in the following manner: step 1: a student presents a thesis which is elicited by the professor a professor calls upon a student to identify the holding of a case (the primary rule laid forth by the court). the student asserts a thesis: for example, “the defendant did not possess mens rea (criminal intent) because he caught his wife in bed with another man before he stabbed him.” the professor considers the thesis false and targets it for refutation by playing devil’s advocate. step 2: the professor sets up the student the professor secures his student’s agreement to further premises: for example, “is one’s criminal intent negated because of his passion? if that is the case, then passion is an affirmative defense and everyone would get off.” step 3: the professor dialogues with the student to think of the opposite position the professor then presents a string of questions that will lead the students in the desired direction. each and every answer provided by the student goes through the elenchus process (tested, cross-examined, and possibly refuted by counterexamples). the professor must be cautious to not dismiss any answer; since it is the students who will have to refute inappropriate answers even if that means refuting the student’s original thesis. in the example, this would be: “a defendant can still possess the criminal intent to commit a crime in the heat of passion, it just might be negated.” step 4: the professor provides the legal principle to be learned when a consensus is reached or in some cases not reached, the professor summarizes by presenting the students’ solutions to the scenario. the professor should emphasize that this is the result of their knowledge and experience that was achieved through the continuous questioning. (paraskevas and wickens, 2003; vlastos, 1983). these four steps will continue to cycle as long as the professor is trying to guide the students to a desired outcome. one of the cornerstones of the socratic method is that there is usually no correct answer, rather, the objective is to see how one can argue his or her belief of the truth. this method is an effective way to assist students in learning how to “think like a lawyer.” this style is based on humility, irony, and fun: humility on the part of the student, irony on the part of the student observers, and fun on the part of the professor (stein, 1991). iv. pedagogy vs. andragogy: who should we be focusing on in law schools? much literature has been written about the differences between pedagogy and andragogy (cartor, taylor, b. and kroth, m. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 46 1990; gehring, 2000; knowles, 1970, 1980, 1984a, 1984b; marshak, 1983; ozuah, 2005; saunders, 1991). those reviews raise some central ideas concerning how adult college students learn best (birzer, 2004). however, very little has been written on whether andragogy applies to law schools. most law students, after all, are adults. in order to further this discussion it is important to first clarify the difference between pedagogy and andragogy. the publication of lindeman’s the meaning of adult education (1926) began the mainstream discussion of adult learning. malcolm knowles, the “father of adult learning,” began to synthesize the concept and brought to popular use the term “andragogy” which had previously been coined in europe as the parallel to pedagogy (saunders, 1991). over the course of knowles’s work, he would eventually summarize six key assumptions about adult learners, which he said are the foundation of adult learning. 1. self-concept: as a person matures, his/her self-concept moves from one of being a dependent personality towards one of being a self-directed human being. adults tend to resist situations in which they feel that others are imposing their wills on them. 2. experience: as a person matures, he/she accumulates a growing reservoir of experience that becomes a resource for learning. adults tend to come into adult education with a vast amount of prior experiences compared to that of children. if those prior experiences can be used they become the richest resource available. 3. readiness to learn: as a person matures, his/her readiness to learn becomes oriented to the development task of his/her social roles. readiness to learn is dependent on an appreciation of the relevancy of the topic to the student. 4. orientation to learn: as a person matures, his/her time perspective changes from one of postponed application of knowledge to immediacy of application, and accordingly his/her orientation towards learning shifts from one of subjectcenteredness to one of problem-centeredness. adults are motivated to learn to the extent in which they perceive that the knowledge which they are acquiring, will help them perform a task or solve a problem that they may experience, or are actually facing in real life. 5. motivation to learn: as a person matures, the motivation to learn is internal. although adults feel the pressure of external motivators, they are most driven by internal motivation and the desire for self-esteem and goal attainment 6. the need to know: adults need to know the reason for learning something. in adult learning the first task of the teacher was to help the learner become aware of the need to know. when adults undertake learning that they deem valuable, they will invest a considerable amount of resources (time, energy, etc.) (forrest iii and peterson, 2006; kidd, 1973; knowles, 1984a, 1984b; knowles et al., 1998; lindeman, 1926; ozuah, 2005; thompson and deis, 2004; tough, 1979; yoshimoto, inenaga, and yamada, 2007, p. 81). knowles argued that the learning process of adults is drastically different than a child’s learning process (birzer, 2004). these six concepts encompass the concept of a self-directed learner. according to mezirow (1981), “[a]ndragogy is an organized and sustained effort to assist adults to learn in a way that enhances their capacity to function as self-directed learners” (p. 21). these six assumptions become a personal interactive agreement between the leaner and the learning endeavor -the “experience” (birzer, 2004). malcolm knowles distinguished these assumptions of andragogy with those of pedagogy taylor, b. and kroth, m. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 47 table 1. comparison of the assumptions of pedagogy and andragogy. regarding pedagogy andragogy concept of the learner role of the learner is a dependant one. the role of the learner is more selfdirected, but the movement from dependency to self-directedness occurs at different rates for different persons. role of the teacher the teacher is expected to take full responsibility for determining what is to be learned, when it is to be learned, how it is to be learned, and if it has been learned. the teacher has a responsibility to encourage and nurture this movement towards selfdirectedness. role of learner’s experience the experience learners bring to a learning situation is of little worth. the experience from which learners will gain the most is that of the teacher, the textbook writer, the audiovisual aid producer, and other experts. as people grow and develop they accumulate an increasing reservoir of experience that becomes an increasingly rich resource for learning. people attach more meaning to learnings they gain from experience than those they acquire passively. primary technique of delivery transmittal techniques – lecture, assigned reading, av presentations. experiential techniques – laboratory experiments, discussion, problem-solving cases, simulation exercises, field experience, and the like. readiness to learn people are ready to learn whatever society says they ought to learn. most people of the same age are ready to learn the same things. people become ready to learn something when they experience a need to learn it in order to cope more satisfyingly with real-life tasks or problems. how learning should be organized learning should be organized into a fairly standardized curriculum, with a uniform step-by-step progression for all learners. learning should be organized around life-application categories and sequenced according to the learners’ readiness to learn. orientation of learning learners see education as a process of acquiring subject-matter content, most of which they understand will be useful only at a later time in life. learners see education as a process of developing increased competence to achieve their full potential in life. learners want to be able to apply whatever knowledge and skill they gain today to living more effectively tomorrow. people are performance-centered in their orientation to learning. organization of curriculum organized into subject matter units which follow the logic of the subject from simple to complex. should be organized around competency/development categories. taylor, b. and kroth, m. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 48 in an exhibit in his book the modern practice of adult education (1980, pp. 43-44). table 1 is an adaptation of that exhibit. to summarize table 1, individuals mature as their self-concept moves from one of being a dependent personality toward being a self-directed adult. individuals also accumulate a growing reservoir of experience that becomes an increasingly rich resource for learning. an individual’s readiness to learn becomes oriented increasingly to the developmental tasks of their social roles. finally, an individual’s time perspective changes from one of postponed application of knowledge to immediacy of application. accordingly, their orientation toward learning shifts from one of subject-centeredness to one of performance-centeredness (knowles, 1980, pp. 4445). the comparison of the assumptions of andragogy and pedagogy provides a foundation for developing the teaching methodology instrument (tmi) to examine whether a teaching methodology is using more andragogical principles or more pedagogical principles. the tmi can provide different levels of orientation. the instrument will help determine whether certain teaching styles are mostly learner-centered or teacher oriented. an instrument assesses the application of andragogical principles to a teaching methodology set forth by malcolm knowles. figure 1 (below) is the tmi to test any teaching methodology to determine whether it is more pedagogical or andragogical. the model works in the following way: 1. a particular teaching methodology is chosen, for example, the socratic method. it is inserted into the first box as the identified methodology to be tested towards adult learning principles. 2. eight categories are examined as they apply to the method to be tested. the layer referred to as “concepts of regard” are the eight characteristics that distinguish pedagogy and andragogy according to knowles. 3. the researcher would then answer the questions listed below each of the eight concepts. these questions are found in the next layer, the “questions of assumptions” level. 4. below each question is the fourth level labeled as the “box score.” for each box on the box score level the researcher is provided with three potential possibilities to choose from: “yes” – meaning he or she overwhelming agrees with the statement, “no” – meaning that he or she overwhelming does not agree with the statement, “somewhat” – meaning that he or she somewhat agrees with the statement because it depends on the situation. 5. the researcher then scores 0, 1, or 2 for each box on the box score level as indicated by the corresponding answer for that box. the scores will differ in each box. 6. after scoring each box, the researcher totals the 16 boxes. the sum is divided by 16; the resulting number is the score. the results may range from 0 to 32, with a median of 16. the closer to 32 on the scale, the stronger the presence of andragogy. the closer to 0 indicates stronger pedagogical principles. the researcher subsequently moves from top to bottom, and left to right throughout the model. taylor, b. and kroth, m. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 49 figure 1. teaching methodology instrument. teaching methodology concept of the learner role of the teacher role of the learner’s experience primary technique of delivery readiness to learn how learning should be organized orientation of learning organization of curriculum learner is dependent learner is self directed teacher takes full responsibility for determining what is to be learned responsibility to encourage and nurture movement towards selfdirectedness experience learners bring is of little worth attach more meaning to learning they gain from experience than those they acquire passively transmittal techniques – lecture, assigned readings experiential techniques, discussion, problemsolving cases, simulation exercises learn what ever society says they ought to learn ready to learn something when they experience a need to learn it standardize curriculum, with a uniform step-by-step progression for all learners organized around life application categories and sequenced to learners’ readiness to learn education is a process of acquiring subject matter content education is a process of developing increased competence to achieve their full potential organized into subject matter units which follow the logic of the subject from simple to complex organized around competencydevelopment categories if yes then score “0” if no then score “2” if somewhat then score “1” ______ if yes then score “0” if no then score “2” if somewhat then score “1” ______ if yes then score “0” if no then score “2” if somewhat then score “1” ______ if yes then score “0” if no then score “2” if somewhat then score “1” ______ if yes then score “0” if no then score “2” if somewhat then score “1” ______ if yes then score “0” if no then score “2” if somewhat then score “1” ______ if yes then score “0” if no then score “2” if somewhat then score “1” ______ if yes then score “0” if no then score “2” if somewhat then score “1” ______ if yes then score “2” if no then score “0” if somewhat then score “1” _____ if yes then score “2” if no then score “0” if somewhat then score “1” _____ if yes then score “2” if no then score “0” if somewhat then score “1” _____ if yes then score “2” if no then score “0” if somewhat then score “1” _____ if yes then score “2” if no then score “0” if somewhat then score “1” _____ if yes then score “2” if no then score “0” if somewhat then score “1” _____ if yes then score “2” if no then score “0” if somewhat then score “1” _____ if yes then score “2” if no then score “0” if somewhat then score “1” _____ taylor, b. and kroth, m. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 50 v. applying the andragogical model to the socratic method. in exploring whether andragogical principles are present, knowles’ assumptions can be examined in three areas: (1) does law school instruction emphasize the skills of analysis and decision making through a series of job-related cases or problems? (2) does the law school instruction establish a learning approach rather than a teaching approach by a series of planned structured activities enabling the learner to acquire the appropriate knowledge? (3) is the law school instruction a practical, job-based approach keeping the learners constantly aware of the value of the training program to them and their work? in applying the socratic method to the tmi a researcher must provide a sixteen question questionnaire to the desired respondents. after a collection of a statistically significant number of responses the scores could be averaged and the mean score calculated. once the score is calculated there would be a better understanding of whether the socratic method is more andragogical or pedagogical. table 2 provides an example of a questionnaire that may be applied. simulated answers are created for the purposes of demonstrating how the model works. a simulated answer is italicized. table 2. sample questionnaire for the socratic method. questionnaire of the socratic method question answer 1. in using the socratic method is the learner dependent? a. yes b. no c. somewhat 2. in using the socratic method is the learner self-directed? a. yes b. no c. somewhat 3. does the teacher take full responsible to determine what is to be learned through the socratic method? a. yes b. no c. somewhat 4. does the teacher have a responsibility to encourage and nurture moment towards self-directedness through the socratic method? a. yes b. no c. somewhat 5. is the experience that the learner brings to the learning environment of little worth through the socratic method? a. yes b. no c. somewhat 6. do learners attach more meaning to learning they gain from experience than those they acquire passively? a. yes b. no c. somewhat 7. does a teacher use transmittal techniques, such as lecture, dialogue, assigned readings, etc. in the usage of the socratic method? a. yes b. no c. somewhat 8. does a teacher use experiential techniques, such as discussions, problem solving cases, and simulations exercises when using the socratic method? a. yes b. no c. somewhat 9. do students learn whatever society says they ought to learn? a. yes b. no c. somewhat taylor, b. and kroth, m. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 51 10. is the student ready to learn something when they experience a need to learn it? a. yes b. no c. somewhat 11. is there a standardized curriculum with a uniform step-by-step progression for all learners through the socratic method? a. yes b. no c. somewhat 12. is the socratic method organized around life application categories and sequenced to learner’s readiness to learn? a. yes b. no c. somewhat 13. is the primary objective of the learning process of the socratic method focused on the acquiring subject matter content? a. yes b. no c. somewhat 14. is the primary objective of the learning process of the socratic method focused on developing increased competence to achieve a student’s full potential? a. yes b. no c. somewhat 15. is the curriculum organized into subject matter units which follow the logic of the subject from simple to complex? a. yes b. no c. somewhat 16. is the curriculum organized around competency and the development of categories? a. yes b. no c. somewhat an example of the results from a simulated questionnaire is displayed in figure 2. a total score of “7” resulted from the exercise presented. this would indicate that the socratic method tends to apply more pedagogical principles than adult learning principles. the socratic method from the score would indicate that it is more teacher-centered than learner-centered. the professor establishes the curriculum, identifies the principles to be learned, decides which questions to ask, and the method by which the students are to be instructed. this is a very pedagogical line of thinking. the theory of pedagogy indicates that the value of practical experience is more minimized. if a survey was conducted and the results were as indicated in table 2, conclusions could be drawn that the socratic method does not provide practical experience. from a pedagogical approach students tend to obtain the didactic concepts of the law, without receiving more of the physical and psychological skills which would be cornerstones to experience. it is important that this instrument be field tested. the sample analysis conducted in this paper was based on the first author’s experience with the subject matter. although it was not intended as a rigorous research test, it does lay the groundwork for future tests and confirmation of the instruments validity. the instrument develops a questionnaire consisting of sixteen questions, and a scoring system. using a significant volunteer sample, a statistical analysis of the scores would determine if the teaching methodologies used are incorporating andragogical principles or pedagogical principles. taylor, b. and kroth, m. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 52 figure 2. teaching methodology model: application to the socratic method. total score: 7 teaching methodology concept of the learner role of the teacher role of the learner’s experience primary technique of delivery readiness to learn how learning should be organized orientation of learning organization of curriculum learner is dependent learner is self directed teacher takes full responsibility for determining what is to be learned responsibility to encourage and nurture movement towards selfdirectedness experience learners bring is of little worth attach more meaning to learning they gain from experience than those they acquire passively transmittal techniques – lecture, assigned readings experiential techniques, discussion, problemsolving cases, simulation exercises learn what ever society says they ought to learn ready to learn something when they experience a need to learn it standardize curriculum, with a uniform step-by-step progression for all learners organized around life application categories and sequenced to learners’ readiness to learn education is a process of acquiring subject matter content education is a process of developing increased competence to achieve their full potential organized into subject matter units which follow the logic of the subject from simple to complex organized around competencydevelopment categories if yes then score “0” if no then score “2” if somewhat then score “1” ___0___ if yes then score “0” if no then score “2” if somewhat then score “1” ___0___ if yes then score “0” if no then score “2” if somewhat then score “1” __ _0___ if yes then score “0” if no then score “2” if somewhat then score “1” ___0___ if yes then score “0” if no then score “2” if somewhat then score “1” ___0___ if yes then score “0” if no then score “2” if somewhat then score “1” ___0___ if yes then score “0” if no then score “2” if somewhat then score “1” ___0___ if yes then score “0” if no then score “2” if somewhat then score “1” ___1___ if yes then score “2” if no then score “0” if somewhat then score “1” __ 1___ if yes then score “2” if no then score “0” if somewhat then score “1” __ 0___ if yes then score “2” if no then score “0” if somewhat then score “1” __1___ if yes then score “2” if no then score “0” if somewhat then score “1” __1__ if yes then score “2” if no then score “0” if somewhat then score “1” __ 1___ if yes then score “2” if no then score “0” if somewhat then score “1” __ 0___ if yes then score “2” if no then score “0” if somewhat then score “1” ___1__ if yes then score “2” if no then score “0” if somewhat then score “1” ___1 __ taylor, b. and kroth, m. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 53 vi. conclusion and recommendations for further research. the teaching methodology model is a model to help determine the level of adult learning principles being used by a particular teaching style in a classroom. the model incorporates those principles set forth by malcolm knowles of what makes a good adult learning environment. when applied to a particular teaching method, in this case the socratic method, a clear delineation of whether the teaching style is more pedagogical or andragogical in its approach is made. although this instrument has preliminary been tested with a law school methodology, its desired usage would be for a wide array of disciplines. if this instrument truly can predict whether or not a methodology is pedagogic or andragogic, it will allow for instructors to reflect more heavily on their teaching styles. as the literature indicates adults learn differently than children. incorporating the andragogical assumptions into their teaching methodologies may provide for the improvement for adult education. when we explored the socratic method, it was discovered that it tends to focus more on pedagogical techniques. it avoids incorporating practical “experience” into the curriculum. how many other disciplines do this as well? students learn more of practical experience from on the job performance and conversing with more experienced attorneys than they do from law school. the chinese proverb holds true that “a single conversation with a wise man is better than ten years of study.” the following are recommendations for further research: 1. the instrument should be tested with the socratic method as well as other teaching methodologies to improve its ability to discriminate between pedagogical and andragogical teaching methodologies. 2. although this instrument is intended to help assess how andragogical a particular teaching approach and methodology might be, it cannot assess the appropriateness or effectiveness of using one approach or another. further research should be conducted to determine which approaches most effectively do prepare law students for the practice of law. references birzer, m. l. (2004). andragogy: student centered classrooms in criminal justice programs. journal of criminal justice education, 15(2), 393-411. bodie, m. t. (2006). open access in law teaching: a new approach to legal education. lewis & clark law review, 10(4), 885-898. butleritchie, d. t. (2003). situating "thinking like a lawyer" within legal pedagogy. cleveland state law review, 50, 29-56. cartor, r. a. (1990). a comparison of andragogy and pedagogy: assessing the relationship between individual personality differences, learning styles, and training types. unpublished dissertation, the university of tennessee. taylor, b. and kroth, m. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 54 chanen, j. s. (2007). re-engineering the j.d.: schools across the country are teaching less about the law and more about lawyering [electronic version]. aba journal. retrieved september 3rd, 2007 from http://www.abajournal.com/magazine/re_engineering_the_jd/. compayré, g., and payne, w. h. (1970). the history of pedagogy (4th ed.). st. clair shores, mich.,: scholarly press. conner, m. l. (2004). andragogy and pedagogy. retrieved july 29, 2007, from http://agelesslearner.com/intros/andragogy.html denney, r. w. (2007). preparing for the real world: more law schools add practical courses [electronic version]. law practice magazine, 33, 10-11. retrieved september 3rd, 2007 from http://www.abanet.org/lpm/magazine/articles/v33/is4/an13.shtml. edwards, h. t. (1992). the growing disjunction between legal education and the legal profession. michigan law review, 91, 34-78. elkins, j. r. (1996). thinking like a lawyer: second thoughts. mercer law review, 47, 511-541. elson, j. s. (1994). the regulation of legal education: the potential for implementing the maccrate report's recommendations for curricular reform. clinical law review, 1, 363-387. forrest iii, s. p., and peterson, t. (2006). it's called andragogy. academy of management learning & education, 5(1), 113-122. friedland, s. i. (1996). how we teach: a survey of teaching techniques in american law schools. seattle university law review, 20, 1-44. gehring, t. (2000). a compendium of material on the pedagogy-andragogy issue. journal of correctional education, 51(1), 151-163. holmes, o. w. (1881). the common law. boston: little, brown, and co. kerr, o. s. (1999). the decline of the socratic method at harvard. nebraska law review, 78, 113-134. kidd, j. r. (1973). how adults learn (rev., updated, completely rewritten. ed.). new york: association press. knowles, m. (1970). the modern practice of adult education: andragogy versus pedagogy. new york: association press. knowles, m. (1980). the modern practice of adult education: from pedagogy to andragogy (rev. and updated ed.). englewoods cliff, nj: cambridge adult education. taylor, b. and kroth, m. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 55 knowles, m. (1984a). the adult learner: a neglected species (3rd ed.). houston, tx: gulf publishing. knowles, m. (1984b). andragogy in action. san francisco: jossey-bass. knowles, m., holton, e. f., and swanson, r. a. (1998). the adult learner: the definitive classic in adult education and human resource development (5th ed.). houston, tx: gulf publishing company. kozinski, a. (1997). in praise of moot court -not! the columbia law review, 97, 178-197. lindeman, e. c. (1926). the meaning of adult education (redistributed edition 1989 ed.). new york: new republic. maccrate, r. (2004). yesterday, today and tomorrow: building the continuum of legal education and professional development. clinical law review, 10, 805-832. marshak, r. j. (1983). what's between pedagogy and andragogy? training & development journal, 37(10), 80. mezirow, j. (1981). a critical theory of adult learning and education. adult education quarterly, 32(1), 3-24. niedwiecki, a. s. (2006). lawyers and learning: a metacognitive approach to legal education. widener law review, 13, 33-71. ozuah, p. o. (2005). first, there was pedagogy and then came andragogy. einstein journal of biology & medicine, 21(2), 83-87. paraskevas, a., and wickens, e. (2003). andragogy and the socratic method: the adult learner perspective. journal of hospitality, leisure, sport & tourism education, 2(2), 4-14. plato, and saunders, t. j. (1987). early socratic dialogues. harmondsworth, middlesex, england ; new york, n.y., u.s.a.: penguin books. saunders, c. e. (1991). pedagogy vs. andragogy: are we treating our students like children? military intelligence professional bulletin, 17(1), 42. socratic method. (n.d.). the american heritage dictionary of the english language fourth edition. retrieved october 1st, 2007, from www.answers.com/topic/socratic-method stein, h. t. (1991). adler and socrates: similarities and differences. individual psychology: the journal of adlerian theory, research & practice, 47(2), 241. thompson, m. a., and deis, m. h. (2004). andragogy for adult learners in higher education. academy of educational leadership journal, 8(3), 77-89. taylor, b. and kroth, m. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 56 tough, a. m. (1979). the adult's learning projects: a fresh approach to theory and practice in adult learning (2d ed.). austin, tex.: learning concepts. vlastos, g. (1980). the philosophy of socrates: a collection of critical essays. notre dame, ind.: university of notre dame press. vlastos, g. (1983). the socratic elenchus. oxford studies in ancient philosophy, 1, 27-58. warkentine, e. r. (2000). kinsfield doesn’t teach my contracts class: using contracts to teach contracts. journal of legal education, 50, 112-133. yoshimoto, k., inenaga, y., and yamada, h. (2007). pedagogy and andragogy in higher education a comparison between germany, the uk and japan. european journal of education, 42(1), 75-98. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009, pp. 57 – 69. students’ perceptions of their connectedness in the community college basic public speaking course hollis f. glaser 1 and shereen bingham 2 abstract: this study explores what classroom behaviors and activities in the basic speech course contribute to student connectedness. the results indicate that student encouragement, humor, honesty, interactive exercises and individual speeches, can help student bonding and motivation, and impacts their overall college experience. keywords: communication, student relationships, classroom activities, student behavior, instructor behavior. i. introduction. the purpose of this study is to understand what classroom behaviors, communication, exercises, and assignments in the basic public-speaking course at an urban community college increase student connectedness. it has been well-documented that students who feel a sense of community in the classroom report greater academic motivation, affinity for school, and enjoyment of class (battistich, et al., 1995; battistich et al., 1991; schaps et al., 1997). conversely, a lack of social support has been related to difficulty in adjusting to school, a propensity to drop out of college, and negative academic performance (cutrona et al., 1994; mcgrath et al., 2000). rovai (2002) developed a “classroom community scale,” which researchers have subsequently used to parse this construct (graff, 2003). a number of scholars have explored which classroom behaviors and pedagogical activities create a sense of community in the classroom (allen (2000), rovai and whiting (2005), david and capraro (2001), summers and svinicki (2007). much of the research on classroom relationships has focused on k-12 (i.e. doveston and keenaghan, 2006; wittse, 2006), internet classrooms (i.e. rovai, 2002), and residential campuses (zhao and kuh, 2004). expanding this line of research into the urban community college speech class is important for a number of reasons. first, for the same reason it is useful to study student connectedness in internet courses, the students at community colleges are also often highly disconnected. although they are in face-to-face classrooms, they live in big cities, come to campus only for class, and it is difficult to establish a campus community. the classroom, therefore, is often the only space where the students have an opportunity to experience a sense of community on campus. second, because it is a basic course, there is a cross-section of students in the class, so techniques that work in this class have a great chance of working in other classes, those that are similarly heterogeneous and those that are more homogenous. third, the course encourages active learning by presenting speeches, a pedagogical activity known to facilitate learning communities (zhao and kuh, 2004). finally, because the basic speech course is required at colleges and universities across the country, the lessons from studying it may be applicable nationwide. 1 department of speech, communication, and theatre arts, borough of manhattan community college, 199 chambers st., new york, ny 10007, hglaser@bmcc.cuny.edu. 2 school of communication, university of nebraska at omaha, 6001 dodge st., omaha, ne 68102, sbingham@unomaha.edu. glaser, h. and bingham, s. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 58 this study is a collaborative project between the authors: glaser, an associate professor of communication at an east coast community college, and bingham, a professor of communication at a four-year metropolitan university in the midwest. glaser noticed that many of her publicspeaking students seem to create very strong bonds during the semester. she perceived that they create stronger relationships than the ones she has seen students develop in her classes at other universities. there may be many reasons she has noticed this—her own biased perceptions, regional or cultural differences in the areas where she has taught, and differences in her teaching across time. the way the students relate to each other colors every moment of the class and glaser has become increasingly interested in this aspect of the public-speaking course, becoming more conscious of making sure the tone of the class is respectful and the students come to know each other. however, when trying to find scholarly work on student relationships, glaser discovered that the great majority of research about classroom communication focuses on instructor-student interaction, i.e. instructor behaviors that influence student learning (hays, 1970; myers, 1995; stuart and rosenfeld, 1994). this void in the literature also has been noticed by bingham, who is part of a research group of communication scholars who have studied specific classroom behaviors and student relationships in the basic public speaking course. while the classroom community scale probes students’ sense of community, it does not ask about specific student behaviors. the research group developed a self-report instrument, the connected classroom climate inventory (ccci) (dwyer et al., 2004) (appendix i), to understand how student connectedness relates to student learning (prisbell et al, 2009) and communication apprehension (carlson et al., 2006). the ccci was developed based on students’ individual comments and group discussions in speech communication courses at a four-year university in the midwest. the questionnaire items were based on commonalities in the students’ responses. studies using the ccci have found a positive correlation between student connectedness and student learning, and a negative correlation between student connectedness and communication apprehension. glaser wanted to understand her community college students’ experiences in the classroom and how they perceive their relationships. at the same time, bingham wanted to know if community college students would find the ccci to be valid and comprehensive. so we decided to build on the quantitative ccci research in this current study in two ways: first, we used qualitative methods to more fully understand students’ perceptions of their classroom experience and how student behavior and communication helps them develop relationships with one another. a qualitative approach to inquiry is appropriate when the researcher seeks a rich and detailed understanding of human experience from the perspective of the people being studied. the participants are asked to share their perceptions and experiences unencumbered by the researcher’s imposition of frameworks or expectations from the literature (creswell, 2007, p. 40; polkinghorne, 2005). second, we asked community college students to complete the ccci and then to answer some questions about its validity. two primary research questions guided this study: “how do urban community college students perceive their relationships in the basic public speaking course?” “do community college students think the items on the ccci do an adequate job of capturing their own sense of connectedness in the public-speaking class?” secondary questions were: “what classroom behaviors do the students believe influence these relationships?” “what classroom activities do they believe influence these relationships?” glaser, h. and bingham, s. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 59 ii. methodology. in order to answer these questions, we gave 62 students in three of glaser’s public-speaking classes two questionnaires, one quantitative, the communication classroom connectedness index (ccci) and one qualitative (see appendix ii). the quantitative questionnaire asks the students to rate the degree to which they feel connected to one another, on a scale from 1-5, 5 being the strongest. the qualitative questionnaire asks eleven open-ended questions about how connected students feel to one another and what behaviors and activities encouraged their relationships. the qualitative portion was administered first so that the students would not be influenced by the activities and behaviors mentioned in the quantitative portion. purposive sampling (polkinghorne, 2005) was used to select three sections of the public speaking course to participate in this study. glaser teaches five sections of the public speaking course during the academic year, and she chose three sections in particular because the students in them seemed to have a high degree of camaraderie; it appeared to glaser that they had a good time in class and were creating friendships. one section was enrolled in a special program where they took four block classes together in the fall of 2007, so that the public-speaking class was one of those four classes. most of these students were traditional first year students, immediately out of high school. the other two sections were evening classes in which the students were a little older and had no other classes together. one section met during the fall semester of 2007 and the other during the spring semester of 2008. the community college where the data were collected enrolled 18,000 students (63 percent women, 37 percent men) with a median age of 23.8. its students come from over 100 countries and according to their self-descriptions, 38% are black, 30 % are hispanic, 12% asian, 10% white, and 10% are other. the basic public-speaking course is required of almost all of the majors in the college and is one of the few courses open to students enrolled in the developmental skills (or remedial) classes. therefore, the students in the public-speaking class represent a cross-section of the entire student population of the college. glaser gave the students the questionnaires the last day of the semester after the final presentations. she explained that the students did not have to complete the questionnaires and stepped out of the room so she would not know who chose not to participate. while not everyone answered every open-ended question, 54 out of 62 students completed the quantitative portion and answered at least some of the qualitative questions. it took the students about 15 minutes to complete. after glaser calculated and turned in their final semester grades, she labeled the questionnaires according to the students’ class section, typed all the answers under each question, then read individual answers to the questions. she then engaged in what lindlof and taylor (2002) refer to as “coding and categorization” analysis; she looked for patterns regarding common perceptions of relationships and consensus on the activities and classroom behaviors that influenced those relationships. she read the answers numerous times, labeled each answer according to a more general category, listed the categories, then looked for similarities among them. she synthesized the categories into a few over-arching concepts that included all of the answers. finally, she interviewed two students during the following semester where she explained the results and asked for their reactions. the students confirmed her findings. bingham entered the numerical data from students’ responses to the ccci into an spss file and analyzed the data statistically. she calculated frequencies and group means for each class, and used one-way analysis of variance to compare the means among the three classes. she glaser, h. and bingham, s. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 60 also read the students’ responses to the qualitative portion of the questionnaire and offered thoughts and feedback to glaser on the interpretation of the data and identification of themes. iii. results. a. students feel connected. the students’ responses to the questions 1, 3, and 10 of the qualitative portion clearly indicated that the students felt connected to each other. when they were asked how many of the students in class they felt connected to, 26 out of 54 responded “all,” and the others ranged from “3” to “almost everyone.” all of the students answered, “yes” in response to “do you feel you are friendlier with students in this class than with students in other classes?” and in response to “have you made friends in this class?” 14 of 15 students in the block section, 15 out of 17 students in the fall evening section, and all of the students in the spring evening section said they had. when asked if the ccci did a good job measuring how connected they felt to their classmates, the students all said “yes.” it is therefore not surprising that the students’ scores on the ccci reinforced the qualitative portion of the questionnaire, indicating that the students, on average, perceived a high level of connectedness with one another (m = 85.98, sd = 7.90). the scores were slightly higher for students in the block section (m = (86.19, sd=5.38) and the first night section (m = 87.29, sd = 5.01) than in the second night section (m = 84.85, sd = 11.09), but a one way analysis of variance showed that the scores for the three sections did not significantly differ from each other [f = 0.377 (2, 50), p = 0.69]. by contrast, previous research using the ccci reports lower ccci scores. the average scores per section for students enrolled in 30 sections of a basic public speaking course at a four-year university ranged from 66.40 to 80.41 (overall m = 70.97, sd=9.91) (dwyer et al., 2004). these results suggest that the level of connectedness among students in the three sections of public speaking at the community college was unusually high. b. certain classroom activities increase the sense of connectedness. the activity that the students most often mentioned as one that increased their sense of connectedness was the speeches they presented in class. the students had to present three major speeches (two informative, one persuasive) and three mini-speeches (two about personal past experiences, and one about their futures). most simply, the speeches were a way that the students got to know each other. one typical response was: “you get to hear everybody’s stories and you get to know everybody better.” however, those of us who teach public speaking know that to speak alone in front of others is a powerful, complex, and risky endeavor. one student captured the complexity and depth of this assignment and how it influences the relationships among the students: “people have to get up in front of their peers and recite a speech that can make them very uncomfortable emotionally and physically. other students respect this and can bond on a deeper level to other individuals based on their emotional and physical needs.” besides the speeches, any activity that required the students to interact helped them get to know each other and seemed to have deepened their sense of connectedness. every interactive exercise glaser assigned was mentioned multiple times as one that helped the students bond. the peer groups, which were formed for every speech round so the students could help each other glaser, h. and bingham, s. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 61 with preparation and feedback, were frequently mentioned, as were more general activities such as “working together” and “talking.” the block section had the opportunity to go to a museum and a few of the students claimed this was a major bonding activity. the evening section had time for a spontaneous debate one day where four teams argued about the legalization of marijuana. glaser was a bit surprised that this came up as an activity that students believed increased their sense of connectedness because it was a fairly contentious and lively event. however, a number of students found it to be a very powerful bonding experience. it is also noteworthy that a handful of students mentioned “feedback” as one of the activities that made them feel closer. all of the students were required to offer written and verbal responses to the presentations, which included both positive and negative comments. critical feedback can be a very sensitive communication act that can easily alienate the receiver. however, a number of students found that it helped them create stronger relationships. as one student wrote in response to “do you feel you are friendlier in this class than in other classes? “: “yes. we can joke and be [ourselves] without being judged, and when you are criticized, it is constructive.” in sum, all of the activities that required the students to interact—the speeches, the working groups, peer feedback, the class discussions and debates—were mentioned as activities that helped them feel a sense of connection with one another. when asked to explain why these activities worked in such a way, the students indicated that the power of these activities was in the opportunities they provided for exhibiting the behaviors the students valued in one another, behaviors that made them feel connected. c. certain student behaviors increase the sense of connectedness. one kind of behavior that the students frequently mentioned can be labeled “friendliness”. this includes a variety of actions that are commonly considered to be friendly: being out-going and talkative, smiling, laughing, saying hello. a number of students in response to question #6 (“what behaviors from the students in this class make you feel connected to them?”) answered with some form of “friendly behavior”. as one student most directly stated, “everyone is just friendly.” a number of the students from the spring evening section appreciated that people come in early to class in order to talk to one another. it is significant that one of the friendly behaviors often mentioned was some type of humor. many of the students mentioned laughter, joking, and people being funny. one student wrote (in response to #6), “they laugh at my comments.” another wrote, “many of them have great senses of humor. i love to laugh and make others laugh.” and a third wrote, “”their personality, how they can just be themselves, funny and lovable.” clearly, common acts of friendliness were recognized and valued by the students as those that increased their sense of connectedness. a second behavior that the students highly valued in one another was what they perceived to be as “honesty”. in response to question #2 (“what happens in this class that helps people feel connected?”), #5 (“what was it about those activities that helped people feel connected?”), and #6 (“what behaviors from the students in this class make you feel connected to them?”), the students repeatedly wrote about “openness,” “honesty,” people “being real,” “emotional,” and sharing personal experiences. one student wrote, “the speech shows a lot what people think and believe. and each one spoke what they had in their heart, and it [made us] appreciate each one.” in response to #5 one student wrote, “the mini-speech allows you the freedom to talk about personal experiences that others can connect with….” in response to #6 glaser, h. and bingham, s. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 62 one student wrote, “everyone seems incredibly open and accessible.” another wrote, “i like the feeling of sincerity and honesty that i got when everybody gave their speeches.” as in other relationships, the opportunity and ability to be honest is a fundamental way to strengthen the connection. the third kind of behavior that students perceived strengthened their bonds was “supportive” behavior. in response to questions #2, #5 and #6, students wrote that others listen, encourage, respect, and help each other. perhaps significantly, when writing about this kind of behavior, supportiveness was the behavior that elicited the most complete sentences from students. many of the responses were quite touching and it is worth quoting a number of them to give a deeper sense of the students’ experience. in response to question 2 (what happens in this class that helps people feel connected?): • “people learn to help and appreciate each other. “ • “everybody listens and makes you feel comfortable. respect, sympathy, friendly people.” • “we are all peers and we all help each other to feel important and worthy of listening to.” in response to question #3 (do you feel you are friendlier with students in this class than with students in other classes? why or why not?): • “you break down social barriers when a group of people all individually have to do the same trying task in front of their group. people typically respect this and can open to console their fellow classmates which typically turns into a strong friendship bond.” • “i think people are closer in this class because we are all vulnerable, and because of that we treat each other with warmth and respect.” in response to question 5 (what was it about those activities that helped people feel connected?): • “the camaraderie you receive when you feel other people good willingly are trying to help you and their group get positive goals accomplished.” in response to question 6 (what behaviors from the students in this class make you feel connected to them?): • “everyone was here for a purpose and that was to learn and we all did it together.” in sum, students perceived that friendly, honest, and supportive interactions help them connect with one another. specific supportive behavior--listening, encouraging, helping, and being respectful--seems to have been profoundly important for the students’ sense of connectedness. many of them appear to recognize their common feelings of vulnerability in the public-speaking course and respond compassionately and gently toward one another. d. certain instructor behaviors increase the sense of connectedness. the instructor activities that helped students feel connected to each other broke down into three kinds: structuring activities, the manner of her speech, and the content of her speech. first, the students recognized the import of structuring activities so that they had the chance to interact with each other. indeed, the vast majority of the responses to question 7 (does your instructor engage in behaviors that make you feel connected to the students? if yes, briefly describe those behaviors.”) discussed getting into groups, assigning speech topics that let students get to know each other, or structured “activities where we have to share our feelings.” as already discussed, interactive student activities gave students opportunities to engage in the behaviors they valued in one another—friendly, honest, and supportive talk—and the students clearly saw the way that glaser structured the class as her most important contribution. glaser, h. and bingham, s. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 63 a second instructor behavior the students attributed to their own sense of connectedness was the relaxed manner in which she spoke to them. one student wrote that class was “laid back and easy-going.” another wrote “she speaks to us on our level.” and a third student wrote, “she joined in when appropriate with our collective insane humor.” the least-mentioned instructor behavior was what she said to them. three students pointed out something about the content of her speech. one student wrote that pointing out similarities among the students was important for their relationships, and two felt that glaser’s sample speeches for each of the assignments helped them connect to each other. (glaser’s sample speeches talked about her trip to china, her dog, a life lesson she learned from a friend, her hopes for the future, and areas of knowledge, such as information about the genographic project, and an analysis of the university’s budget.) one important note is that early in the semester, glaser gave a few lectures about the importance of supportive behavior in the classroom, how listening effects relationships, and some basics of interpersonal communication. evidently, these lectures were pretty well forgotten by the end of the semester. in sum, when thinking about what instructor behaviors promoted a sense of connectedness, it appears that the students most valued when their instructor gave them a chance to talk to each other. secondarily, they valued that she talked to them to in a relaxed manner, pointed out similarities among them, and shared her experiences with them. e. student connectedness impacts the college experience. finally, we wanted to know if students’ relationships in the class impacted their wider college experience. as discussed above, a number of studies have claimed that students who feel a sense of community in the classroom are more motivated and enjoy school more (battistich et al., 1995; battistich et al., 1991; schaps et al., 1997). those who do not have social support have a harder time academically and are more likely to drop out of college (cutrona et al., 1994; mcgrath et al., 2000). we asked two related questions: question 9 “has this class made you feel more comfortable at [this community college]? why or why not? and question 11: “how does feeling connected to other students in a class affect your learning in the class?” in response to question 9, 51 of the 54 students answered “yes.” the reason most often stated for their increased comfort at the college was that they know more people, made friends, feel there are people they can relate to at the school, and feel like they belong. a number of students from the block section even grew close enough to their classmates to claim that they felt like family. while the students clearly felt that their friendships made them feel more comfortable at the college, the class also helped some students feel more confident about speaking to strangers. one student said, “this class has made me see that you do not have to be silent.” another wrote, “i don’t feel as shy or embarrassed to speak in public and i also feel more relaxed at meeting new friends.” “it’s public speaking so you learn to open up more,” wrote a third student. it seems that some students felt that the process of making friends in the basic speech course helped them make friends in other courses also. wrote one student, “i feel there are people who i can relate to, just like people in this class.” so there may be a certain ripple effect where strong student connections in one class increase the connections throughout the college. finally, two students felt that the basic speech course was a generally motivating experience. one wrote the class “makes me confident in my learning ability,” and another that, “it gives us the motivation to get on with life.” it is unclear whether these attributions are related glaser, h. and bingham, s. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 64 directly to the unique aspects of the basic speech course or if they are related to a successful experience in any course, so that feeling motivated might also occur in other courses in which the students did well. the responses to question 11, however, relate specifically to the students’ sense of connectedness in the public speaking course. fifty-one out of 54 students answered positively to the question, “how does feeling connected to other students in a class affect your learning in the class?” the students perceived that their relationships helped their learning in a number of ways: socially, motivationally, and cognitively. in terms of the social aspects of the course, quite a few students said the class was easier, more comfortable, and more fun because of their friendships. one answer was typical: “it makes it easier and fun.” having friends also added a supportive element to the class so that they could call on each other for help. according to one student, “i know if i have trouble with anything i didn’t understand, i can consult with my classmates.” this dimension of comfort and support in the classroom wasn’t simply task-oriented so that students had help with the material, but according to one student, also stretched into the emotional dimension: “i feel more comfortable like if i make an effort, no one is going to judge me.” many students also found their classroom friendships to be motivating. some felt that their friendships made them want to come to class and to do well in it. wrote one, “it’s a drive to want to do the work and attend class. it’s that extra push.” other students said it made them want to learn, work harder, and “give that information to them clearly.” it seems, then, that classroom friendships helped some students push themselves to get to class and work harder on their assignments. interestingly, two students felt that they benefited cognitively from their classroom friendships. one student wrote, “connecting with others makes me think better.” another wrote, that it “improves my ability.” while only two students mentioned an effect on their cognitive processes, it is certainly one directly relevant to our responsibilities as educators and deserves some serious exploration in the future. in sum, students seemed to form strong connections in the basic speech course and highly valued those connections. they attributed interactive class activities (including the speeches) to helping them form those connections, as well as very specific student behavior, including joking, listening, talking honestly, and encouragement. in addition, having strong student connections in the classroom appears to have had some profound and wide effects for the students. they enjoyed the classroom experience more, found others to help them with assignments, were more motivated to attend class and complete their assignments, and consistent with previous research (prisbell et al., in press), may even have learned better. iv. conclusion. this research suggests a number of important findings. first, it supports the validity of the ccci, suggesting that it captures students’ sense of connectedness. however, our qualitative findings also suggest ways the ccci may need to be expanded to make it more comprehensive, for east coast community college students if not for college students in other regions and at other kinds of institutions. the ccci appears to omit or underemphasize some key kinds of communication behaviors that help students develop friendly relationships and a sense of connection with each other. these include shared humor, openness, honesty, genuineness, vulnerability, helping each other, and compassion. glaser, h. and bingham, s. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 65 the ccci may also be an important complement to rovai’s (2001) classroom community scale (ccs). while the ccs explores a more general sense of community in the classroom, the ccci focuses on specific behaviors and interpersonal connectedness within the classroom. in other words, the ccci allows researchers to study student communication patterns that, as a whole, build a larger sense of community. by using both instruments, future research could explore this relationship more directly. second, student behavior and communication needs to be studied as much as instructor behavior is. as stated previously, the vast majority of the communication education literature focuses on teacher-student relationships and perceptions, and rarely on student-student relationships. yet, this study clearly shows that the connections students make with one another have a profound impact on their college experience. currently, education scholars (rovai, 2001, dawson, 2006; gould et al., 2000) are focusing on this important aspect of classroom interaction; we hope our research encourages more scholars in the communication field and other disciplines to do the same. third, this research has clear implications for teachers of all disciplines who are using collaborative and interactive learning techniques in the classroom (bean, 2001; barley et al., 2005). as instructors are creating more opportunities for students to work and think critically together, this study focuses instructors on ensuring that the students behave in a supportive, nonjudgmental, honest, and friendly manner while in those groups. inversely, instructors may want to structure their classes in ways that discourage students from behaviors that undermine their sense of being connected. inconsiderate and harassing “misbehaviors” by students, for example, are associated with diminished student connectedness, as measured by the ccci (bingham, carlson, dwyer, and prisbell, 2009). fourth, on a more practical and administrative note: students found the course to be an important bonding experience. in support of previous studies (battistich, et al., 1995; battistich et al., 1991; schaps et al., 1997), students reported that the relationships they built in the basic course helped them stay motivated, enjoy the class, and maintain their attendance. many found the speech presentations to be very powerful experiences that helped them connect to one another. the presentations allowed students to get to know each other, gave them opportunities to support and encourage one another, and the ability to be vulnerable in front of each other. certainly, the data indicates that any interactive task or exercise helped the students bond. yet, the presentations, the singular task of each student speaking alone in front of the group, goes a long way in encouraging student connections. many students reported that the class helped them talk to students in other classes and situations. the course introduced them to fellow students who they related to. this sense of familiarity then gave them a feeling of belonging in the wider college. it may be that the basic speech course is one way to help students build the community they need in order to succeed in college. to the extent that colleges are concerned about student persistence, perhaps this research supports current efforts in encouraging students to take the basic speech course their first semester, in order to make important connections with other students. finally, this study suggests the need for more research. we looked at three sections of one course in one institution. because it is so important for students to have strong relationships with one another, it is also important for us to have a deeper and wider understanding of how students form these relationships in our classrooms. future research could expand into other kinds of classes, educational institutions, go deeper into the student experience through focus groups and interviews, and probe instructors’ attempts to influence student connectedness. this study glaser, h. and bingham, s. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 66 contributes to the exploration of student relationships, their significance for learning in the classroom, and the import of those relationships for the students’ wider college experience. appendix appendix 1. connected classroom climate inventory students respond to the following statements with a 5-point likert scale ranging from strongly disagree (1) to strongly agree (5). 1. i feel a sense of security in my class. 2. i have common ground with my classmates. 3. i feel a strong bond with my classmates. 4. the students in my class share stories and experiences with one another. 5. the students in my class are friendly with one another. 6. the students in my class respect one another. 7. i feel included in class discussions in my class. 8. the students in my class are courteous with one another. 9. the students in my class praise one another. 10. the students in my class are concerned about one another. 11. the students in my class smile at one another. 12. the students in my class engage in small talk with one another. 13. the students in my class are non-judgmental with one another. 14. the students in my class laugh with one another. 15. the students in my class are supportive of one another. 16. the students in my class show interest in what one another is saying. 17. the students in my class cooperate with one another. 18. the students in my class feel comfortable with one another. glaser, h. and bingham, s. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 67 appendix 2. interview protocol—student connectedness i am studying student relationships and what helps build friendships in the classroom. i am especially interested in the basic public-speaking course. please answer the following questions as thoroughly as you can. 1. how many of the students in this class do you feel connected to? 2. what happens in this class that helps people feel connected? 3. do you feel you are friendlier with students in this class than with students in other classes? why or why not? 4. what classroom activities do you think make people feel connected to each other? 5. what was it about those activities that helped people feel connected? 6. what behaviors from the students in this class make you feel connected to them? 7. does your instructor engage in behaviors that make you feel connected to the students? (yes/no/not sure) if yes, briefly describe those behaviors. 8. was there a particular incident or incidents that made you feel more connected to the class? if so, please describe it. 9. has this class made you feel more comfortable at bmcc? why or why not? 10. have you made friends in this class? 11. how does feeling connected to other students in a class affect your learning in the class? references allen, t. (2000). creating community in your classroom, education digest, 65. barkley, e. f., cross, k. p. and major, c. h. (2005). collaborative learning techniques: a handbook for college faculty. san francisco: jossey-bass. battistich, v., solomon, d., kim, d., watson, m., and schaps, e. (1995). schools as communities, poverty levels of student populations, and students' attitudes, motives, and performance: a multilevel analysis. american educational research journal, 32,627-658. battistich, v., watson, m., solomon, d., schaps, e., and solomon, j. (1991). the child development project: a comprehensive program for the development of prosocial character. in w.m. kurtines and j.l. gerwirtz (eds.), handbook of moral behavior and development: vol. 3. application. new york: erlbaum. glaser, h. and bingham, s. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 68 bean, j. c. (2001). engaging ideas: the professor’s guide to integrating writing, critical thinking, and active learning in the classroom. san francisco: jossey-bass. bingham, s.g., carlson, r.e., dwyer, k.k., and prisbell, m. (2009). student misbehaviors, instructor responses, and connected classroom climate: relationships and classroom implications. basic communication course annual, 21, 30-68. carlson, r., dwyer, k., bingham, s., cruz, a., prisbell, m. and fus, d. (2006). connected classroom climate and communication apprehension: correlations and implications for the basic course. basic communication course annual, 18, 1-27. creswell, j. w. (2007). qualitative inquiry and research design: choosing among five approaches. thousand oaks, ca: sage publications cutrona, c.e., cole, v., colangelo, n., assouline, s.g., and russel, d. (1994). perceived parental social support and acadeic achievement: an attachment theory perspective. journal of personality and social psychology, 66, 369-378. david, h.l. and capraro, r.m., (2001). strategies for teaching in heterogeneous environments while building a classroom community. education, 22, 1. dawson, s. (2006). a study of the relationship between student communication interaction and sense of community. internet and higher education, 9, 153-162. doveston, m. and keeenaghan , m. (2006). improving classroom dynamics to support students learning and social inclusion: a collaborative approach. support for learning, 21, 5-11. dwyer, k.k., bingham, s.g., carlson, r.e., prisbell, m., cruz, a.m., and fus, d.a. (2004). communication and connectedness in the classroom: development of the connected classroom climate inventory. communication research reports, 21, 264-272. gould, c.a., grossman, s., and williston, j. (2000). building classroom communities at the university level. childhood education, 77, 104-106. graff, m. (2003). individual differences in sense of classroom community in a blended learning environment. journal of educational media, 28, 203-210. hays, e.r. (1970). ego-threatening classroom communication: a factor analysis of student perceptions. speech teacher, 19, 43-48. lindlof, t.r., and taylor, b.c. (2002). qualitative communication research methods (2 nd ed.). thousand oaks, ca: sage publications. mcgrath, p.b., gutierrez, p.m., and valadez, i.m. (2000). introduction of the college student social support scale: factor structure and reliability assessment. journal of college student development, 41, 415-426. glaser, h. and bingham, s. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 69 myers, s.a. (1995). student perceptions of teacher affinity-seeking and classroom climate. communication research reports, 12, 192-199. polkinghorne, d.e. (2005). language and meaning: data collection in qualitative research. journal of counseling psychology, 52, 137-145. prisbell, m., dwyer, k.k., carlson, r.e., bingham, s.g., and cruz, a.m. (2009). connected classroom climate and communication in the basic course: associations with learning. basic communication course annual, 21, 151-172. rovai, a.p. (2002). development of an instrument to measure classroom community. internet & higher education, 5, 197-212. rovai, a. p., wighting, m.j., (2005). feelings of alienation and community among higher education students in a virtual classroom. internet & higher education, 8, 97-110. schaps, e, and lewis, c., (1997). “building classroom communities.” thrust for educational leadership, 27:1. stuart, w. d., and rosenfeld, l.b. (1994). student perceptions of teacher humor and classroom climate. communication research reports, 11, 87-97. summers j.j. and svinicki, m. (2007). investigating classroom community in higher education. learning & individual differences, 17, 55-67. wittse, l. (2006). “like pulling teeth”: oral discourse practices in a culturally diverse language arts classroom. the canadian modern language review, 63, 199-223. zhao, c-m. and kuh, g.d., (2004). adding value: learning communities and student engagement. research in higher education, 45, 115-138. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009, pp. 70 – 81. teacher as trickster on the learner’s journey kenneth w. davis 1 and scott r. weeden 2 abstract: for tens of thousands of years, teachers have used stories to promote learning. today’s teachers can do the same. in particular, we can employ joseph campbell’s “monomyth”—with its stages of separation, initiation, and return—as a model for structuring learning experiences. within the monomyth, one tempting role for teachers is the sage, but we should resist this temptation. instead we should acknowledge, and benefit from, our role as tricksters. to do so is to accept and illuminate the dual responsibility of the teacher as both supporter and challenger. keywords: myth, initiation, trickster, metaphor, course design. for tens of thousands of years, teachers have used stories to promote learning. we can imagine an early classroom, made up of would-be hunters gathered around a fire, listening to a tale of a successful or unsuccessful hunt. by hearing a story, the apprentice hunters learn what works and what doesn’t. storytelling comes naturally to us. as therapist and teacher joseph gold (2002) states in the title of a recent book, we humans are “the story species.” today’s teachers are drawing on the power of story in a variety of ways. for example, in the health professions the field of “narrative pedagogy” is flourishing. student doctors, nurses, and other health-care professionals are learning from stories how to see their patients, not as “presenting” conditions but as whole persons, with their own life stories. one especially powerful use of stories in teaching involves joseph campbell’s “monomyth.” we teachers can employ the monomyth— with its stages of separation, initiation, and return—as a model for structuring learning experiences. however, we must be careful. we must resist the temptation to take on the role of sage, a frequent figure on the monomythic journey. instead we should acknowledge, and benefit from, our role as tricksters. to do so is to accept and illuminate the dual responsibility of the teacher as both supporter and challenger. i. the monomyth joseph campbell (1949), in the hero with a thousand faces, argues that many traditional stories are variations on a single story. campbell, adopting a word coined by james joyce, calls this story “monomyth.” campbell describes the monomyth as having three stages: • separation • initiation • return he summarizes the monomyth this way: “a hero ventures forth from the world of common day into a region of supernatural wonder: fabulous forces are there encountered and a decisive 1 department of english, indiana university-purdue university indianapolis, 425 university blvd., indianapolis, in, 46202, kdavis@iupui.edu 2 sweeden@iupui.edu. davis, k. and weeden, s. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 71 victory is won: the hero comes back from this mysterious adventure with the power to bestow boons on his fellow man” (30). think of jack (of beanstalk fame), snow white, dorothy (of kansas and oz), luke skywalker. some critics have held that campbell’s monomyth is too specific, too detailed, to describe a large number of stories. for example, donald e. polkinghorne (1988) writes, “attempts to uncover a single deep plot in all the world’s stories are no longer held in esteem in narrative theory. in the end, there is no way to tell where these investigations are right or wrong, because finding similarities among the surface diversity seems to depend on the imaginative function of the person identifying them” (78). others have found campbell’s account of the monomyth too male-centered, not taking into account differences between the male and female life journeys—the male more outer, the female more inner. but even if we are uneasy with the specifics of campbell’s monomyth as a descriptor of traditional tales, we can still find value in it—in general terms—as a model for structuring learning experiences. we can employ the three stages of the monomyth as a kind of map, guiding us and our students through a rich learning experience by showing us where we are, and what we have to do to get somewhere else. educational theorist jerome bruner (1986) makes a case for a generalized form of the monomyth: narrative deals with the vicissitudes of human intentions. and since there are myriad intentions and endless ways for them to run into trouble—or so it would seem—there should be endless kinds of stories. but, surprisingly, this seems not to be the case. one view has it that lifelike narratives start with a canonical or “legitimate” steady state, which is breached, resulting in a crisis, which is terminated by a redress, with recurrence of the cycle as an open possibility. (16) bruner’s “breach” is, of course roughly equivalent to campbell’s “separation”; his “crisis,” campbell’s “initiation”; and his “redress,” campbell’s “return.” another educational theorist, kieran egan (1989), generalizes, and simplifies, even further. “there is, then,” he writes, “at the simplest level a rhythm in stories. they set up an expectation at the beginning, this is elaborated or complicated in the middle, and is satisfied at the end. stories are tied beginning to end by their satisfying the expectation set up at the beginning” (24). what egan may be seeing is the influence of our own tacit cultural knowledge of the monomyth. beginning with the first stories we hear, we learn to expect that the wrongs at the beginning will be redressed at the end. however specific the description of the monomythic pattern, what is important to us as teachers is that this pattern is always about learning. we repeat, always about learning. as screenwriter christopher vogler (1998) says, “in any good story the hero grows and changes, making a journey from one way of being to the next: from despair to hope, weakness to strength, folly to wisdom, love to hate, and back again” (13, emphasis ours). this characteristic of the monomyth makes it an ideal pattern for the design of learning experiences, whether lessons, units, or entire courses. the monomyth is a model of what happens in all learning. to learn, we must leave the comfort of the familiar, of what we think we know, and enter an unknown territory, a territory that can be frightening. there we are confronted with challenges, even dangers. if we overcome these challenges, we can return “home” with boons, in the form of new knowledge and skills, and apply them to our lives. to explore the monomyth as a model for teaching and learning, let us look at each of campbell’s three stages in turn. davis, k. and weeden, s. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 72 a. separation. for campbell, the monomyth begins with a “call,” a message that “destiny has summoned the hero and transferred his spiritual center of gravity from within the pale of his society to a zone unknown” (53). for the hero, “the familiar life horizon has been outgrown; the old concepts, ideals, and emotional patterns no longer fit” (47). such a realization is necessary for learning to occur. to be truly motivated to learn, students must be somehow dissatisfied with their present knowledge or skills. as english teacher dan lindley (1993) writes, successful teaching begins when “a student is puzzled, even upset, by a discrepancy, a painful occurrence in a story, a concept not understood” (126). maxine greene (1995) concurs: “the difficult task for the teacher is to devise situations in which the young will move from the habitual and the ordinary and consciously undertake a search” (24). john seely brown and paul duguid (2002) express this principle by labeling learning as “demand driven.” they write, “people learn in response to need. when people cannot see the need for what’s being taught, they ignore it, reject it, or fail to assimilate it in any meaningful way. conversely, when they have a need, then, if the resources for learning are available, people learn effectively and quickly” (136). one way of expressing the teacher’s role at this stage is helping learners move from “unconscious incompetence” to “conscious incompetence” (haines, 1998, 95), though this may not be the original source for these terms). one helpful model for understanding the separation stage of the monomyth comes from systems theory. consider the activity of any natural system, from atom to earth: the system is in a status quo, a steady state in which internal and external forces are balanced. when, from time to time, new external forces disrupt the system, it quickly “corrects” and settles back into the status quo. sometimes, however, greater external forces—forces ultimately directed toward disorganization and death—threaten the very existence of the system. facing this stronger threat, the system is unable to “correct” itself, and so, at first, succumbs to those forces. but, in doing so, it uses them as a means toward reorganization. if this reorganization is successful, the system emerges into a new steady state, one more organized and resistant. and we’ve just heard campbell’s monomyth. if the monomyth is a jungian archetype, a pattern inherited as part of one’s collective unconsciousness, it’s a small wonder. humanity is the result (although, it is to be hoped, not the final result) of the precise process the monomyth recounts, over and over since the universe began. if any story is structured into our unconsciousness, it must surely be this one. the evolution story is truly the only story there is. in these systems terms, learners begin their hero’s journey with a comfortably ordered psyche, in a state of relative equilibrium. to learn, that equilibrium must be disturbed, thrown into disorder. in that relatively disordered state, far from equilibrium, a new order, a new equilibrium, can emerge, with a higher degree of complexity. as michael roemer (1995) writes, “we think of heroes as eager to act but only a few, like don quixote or emma bovary, seek great deeds or adventure. most of them do not go looking for trouble but do what they must to return life to equilibrium. odysseus, like hansel and gretel, is simply trying to get home” (6). the ideal setting for learning occurs when the learner himself or herself realizes that old knowledge, old ways of thinking, no longer suffice. one of the joys the two of us find in teaching at a “commuter” campus is the opportunity to work with older, nontraditional students who have come to this realization by themselves, who have—on their own—heard the call to adventure, the call to further learning. but more often, we as teachers have to facilitate this davis, k. and weeden, s. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 73 realization. jacqeline grennon brooks and martin g. brooks (1999), in their discussion of “constructivist” teaching, claim that “students’ fundamental quest is discrepancy resolution” (28). they elaborate: cognitive growth occurs when an individual revisits and reformulates a current perspective. therefore, constructivist teachers engage students in experiences that might engender contradictions to students” current hypotheses. . . . contradictions are constructed by learners. teachers cannot know what will be perceived as a contradiction by students; this is an internal process. but teachers can and must challenge students” present conceptions, knowing that the challenge only exists if the students perceive it as a contradiction. teachers must, therefore, use information about the students’ present conceptions, or points of view, to help them understand which notions students may accept or reject as contradictory” (112-13). a friend and colleague of ours, the late tony sherrill, sometimes wore a jacket and tie to the first meeting of his introductory religious studies course, and asked that his students dress similarly at future class meetings. at the second class meeting, when at least some students had complied with tony’s request, tony himself would arrive in t-shirt, shorts, and sandals. the subsequent discussion, of why some students had complied and some had not, led to a discussion of authority, and of how we accept or reject it. the stage was thus set for the rest of the course, a course in which students inevitably struggle with issues of religious authority, the decision to follow, or not, the religious paths of their family and friends. but the call does not have to emphasize the inadequacy of present knowledge or skills. it does not, and should not, have to make students feel incapable or unworthy of the proposed adventure. rather, the call can be a positive experience, growing out of built-in desire for the beyond. jonathan culler (2000) points out that “the pleasure of narrative is linked to desire. plots tell of desire and what befalls it, but the movement of narrative itself is driven by desire in the form of ‘epistemophilia’, a desire to know: we want to discover secrets, to know the end, to find the truth” (91). ultimately, the separation stage requires, of both teacher and learner, trust and imagination—the ability to see beyond the present situation. as maxine greene (1995) writes, “to learn and to teach, one must have an awareness of leaving something behind while reaching toward something new, and this kind of awareness must be linked to imagination” (20). b. initiation. the second stage of the monomyth is the most difficult, for student and teacher. leaving the safe and familiar can trigger fear and mourning. as stephen g. haines (1998) writes, “people experiencing change typically feel a deep sense of loss. they are heading toward new territory, with old, familiar ways—always so comfortable, and often valued—falling behind them” (175). haines elaborates: when we go through change, whether personal or professional, we don’t move on a straight line of productivity from a to b. our thoughts, feelings, and experiences fluctuate between highs and lows; we feel as if we are on a rollercoaster. (175) as much as anyone is the past century, gandhi gave up everything safe and familiar in his quest for justice. he is widely reported to have said, providing us yet another instance of campbell’s monomyth, “every worthwhile accomplishment, big or little, has its stages of drudgery and triumph: a beginning, a struggle and a victory.” one of the most important actions davis, k. and weeden, s. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 74 a teacher can take at this stage is simply naming it, defining the stage as an essential step in learning. haines (1998) continues: often, just knowing about the rollercoaster of change helps people who are undergoing change. they see it is only natural to experience difficulties at such times. the key is ‘hanging in there,’ in developing persistence.” (174) one of us—ken—teaches an advanced copyediting course that asks students, usually for the first time in their college careers, to follow an exacting set of mechanical rules, as preparation for jobs in publishing. for many students, the moment of truth comes about a third of the way into the course, when the first exam is returned and some students learn that they have received their first-ever d or f. at this point, ken explicitly invokes the monomyth, telling these students that they may have been jerked out of their comfort zones, that they are in unfamiliar territory, with dragons to slay, but that they have been given what they need to conquer these threatening forces and return to their familiar lives with new, and valuable, skills. in some courses the teacher also needs to assure students that mistakes are an important part of the initiation stage. scott takes students through this kind of process in his first-year composition classes, requiring students to develop a drafting and revising process, submit their papers for initial feedback from both peers and the instructor, and make productive decisions about their writing based on instruction and the feedback they are receiving. inevitably, most students find the process of dealing with error unsettling because they have learned to see error as an indicator of the degree to which they have strayed from being right. but as many theorists and researchers point out, mistakes are often signs that risks are being taken. haines quotes management theorist rosabeth moss kanter, in her book men and women in the corporation: “a basic truth of management—if not of life—is that nearly everything looks like a failure in the middle” (haines [1998], 175). brooks and brooks (1999), as well, emphasize the necessity of error: on most tests and homework assignments, students aren’t asked to reveal and elaborate on their points of view. they are asked instead to be “right.” being “right” often diverts energy away from the generation of new views. we must remember that the ptolemaic view of the solar system was a conceptual stop on a path that led to the copernican views presently held by most astronomers. we think today that ptolemy was not “right,” but his point of view certainly counted. (68) in this context, brooks and brooks point out that a key word for the hero’s journey is errant—as in “knight errant”—a word closely related to the word error. by making and correcting errors, the learner moves from “conscious incompetence” to “conscious competence.” c. return. a friend and colleague from the boston area has given us a piece of urban folklore for the return stage of the monomyth. we report it here merely as a folklorists, with no wish to slander the good people of lynn, massachusetts: lynn, lynn, city of sin, you never come out the way you went in. in traditional stories, heroes are changed, for the better, by their adventures; otherwise they’re not heroes. campbell (1949) writes, “the hero comes back from this mysterious adventure with the power to bestow boons on his fellow man” (28). the learner, as well, needs to move from the davis, k. and weeden, s. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 75 initiation stage to a new integration, a new comfort, a new equilibrium. dan lindley (1993) writes that at this stage, the student owns the new idea. at that moment it ceases to be new. in fact, if the new idea is inherently true to the student’s nature, true to human nature in general, then the newness will disappear insensibly into familiarity. at this point the student may very well say, with a certain wonderment: “i knew that. i knew that, all along.” (126-27) bobbi deporter, mark reardon, and sarah singer-nourie (1999) use the concept “theirs to ours, ours to theirs” in describing the necessary cycle for teaching and learning. they begin by asserting that the teachers must begin the cycle with their own the separation stage, by entering the learners’ world: theirs to ours, ours to theirs reminds us of the importance of entering the students’ world first. in order for you to earn the right to teach, you must first build authentic bridges into your students’ lives. at the initiation stage, deporter et al are a bit didactic for our tastes: “once this link has been established,” they continue, “then you can bring them into your world, and give them your understanding of the content.” but we agree fully with their description of the return stage: “with this expanded understanding and greater mastery, the students can take what they’ve learned into their world and apply it to new situations” (6-7). at this return stage, the learner moves from “conscious competence” to “unconsciousness competence,” as equilibrium is restored. but that equilibrium is, of course, never final. the hero’s journey must be undertaken again and again. though campbell represents the monomyth as a circle, the monomythic hero is inevitably changed by his or her journey. therefore we might represent that change by having campbell’s circle gradually rise into a third dimension, like a thread of dna. the hero returns to the place he or she left from, but “higher.” indeed our life can be represented as a helix, as a climb up a circular staircase. as alida gersie and nancy king (1990) write, using epic terms themselves: every return is born of hope and expectation. repetition offers us a second chance, a new future. through repetition we enhance our experience, knowledge and skills. we demonstrate our mastery and control, our ability to make the unpredictable predictable. thus we extend the past and defeat the transitory quality of time. (262) ii. the teacher’s role. within the monomyth, one tempting role for teachers is the sage. many embodiments of the monomyth include in their casts a wise elder woman or man, guiding and protecting the hero along the journey. the temptation for a teacher to play a sage’s role is a strong one: many of us have been inspired to enter our profession by a sage, in literature or film, or in real life. who wouldn’t want to be a mr. chips, an anne sullivan, a mr. holland, a barbara jordan? but there’s great danger, to one’s students or oneself, in consciously taking on that role. to attempt to be a sage can lead a teacher into smugness and arrogance, trying to rely on unearned power. allan combs and mark holland (2001) write of the archetype of …the wise old man, the embodiment of deep and ancient wisdom personified in literary and film characters such as merlin the magician, gandalf the gray, and obi-wan kenobi of star wars. each wields magic powers that derive from his mastery of ancient, all-butlost knowledge. other examples of less mysterious and more beneficent wise old men, davis, k. and weeden, s. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 76 such as the wise men from the east, touch upon another archetype, that of the god-man, or manna man to use jung’s term. this is the ideal of a human embodiment of the essence of the divine. projecting this image onto someone else is to give that person great emotional power over yourself. needless to say, this can be very dangerous unless that person is a remarkably worthy individual. to identify personally with this archetype is a major obstacle to inner growth, for it virtually guarantees an absence of humility. it is fine for others to refer to mohandas gandhi as mahatma, “the great soul,” but beware of those who confer such titles upon themselves. (70) however tempting the role of sage is, teachers should resist the temptation. but the monomyth includes another role they can play: the role of trickster. to consciously adopt this role is to accept and illuminate the dual responsibility of the teacher as both supporter and challenger. the difference between sage and trickster is illustrated by the wizard of oz. in the film, the wizard himself is first imagined by the hero, dorothy, and her three friends as the wise sage, the magus who can solve all their problems, remedy all their deficiencies, if they can only get to the emerald city. but in fact, their expectations are met only when the wizard abdicates his magus role and plays a trickster role, setting tasks or obstacles for the four. by overcoming these obstacles, dorothy and her friends find within themselves what they desire: a heart, a brain, courage, and something all human beings may unconsciously desire: the means of going home. the trickster is the figure, seen in myths and legends across the world, who acts as fool, but who also initiates wisdom and insight, if not for other characters in a story, then for the story’s listeners or readers. in some stories, the hero himself can be a trickster. for example, the wandering odysseus is a trickster figure, engineering the trojan horse and fooling the cyclops, among other tricks. so also is prometheus, who steals fire from the gods and brings it to the earth. in his extensive study of the trickster, lewis hyde (1998) notes that in myths and legends, tricksters can be thieves who bring boons to humankind. however, and more importantly, tricksters also help to create culture. two examples illustrate. ridie wilson ghezzi (1998) reminds us that among the ojibwa of the upper midwest and southern ontario, the trickster nanabush was said to have created the present world and taught people the cultural arts. and, as julie cruikshank and angela sidney (1998) comment, for the peoples of the yukon, the trickster “crow creates the world, brings, light and fire and fresh water. he creates human beings and teaches them the principles of culture” (140). thus, tricksters can be said to promote culture and to see to it that it is carried on. one way they do so, hyde (1998) tells us, is by both creating and crossing boundaries and bringing previously hidden distinctions into view. hyde writes: we constantly distinguish—right and wrong, sacred and profane, clean and dirty, male and female, young and old, living and dead—and in every case trickster will cross the line and confuse the distinction. trickster is the mythic embodiment of ambiguity and ambivalence, doubleness and duplicity, contradiction and paradox. (7) and nancy hathaway (2001) notes: on the one hand, tricksters are slippery, selfish, and occasionally evil. they lie, cheat, do stupid things, and cause trouble for one and all. on the other hand, they perform the essential task of bringing culture to humanity. they show us how to hunt, cook, and make musical instruments, they force us to work, and, like the african-american trickster br'er rabbit, they teach us to tell stories. (42-43) davis, k. and weeden, s. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 77 it is in this role as initiator of culture that tricksters and teachers can be said to have similar roles. but the similarity can go even further. tricksters are not only the creators of culture and its distinctive boundaries; they are also disrupters of culture. julie cruikshank and angela sidney (1998), for instance, tell us that not only does crow bring culture to the yukon peoples, but he also marries fish mother so that he can “eat without doing any work, and then he treats her with disrespect” (140). in other words, rather than taking part in the important activities that sustain people, crow refuses to accept his part of the necessary work, and in the process, he insults an important figure. for their part, not only do the ojibwa recognize the important contribution nanabush has made to the world, they also recognize that he is “a fool, a witch, a manipulator, and an example of behavior to avoid” (ghezzi, 1998, 444-45). however, this boundary-crossing role also proves beneficial, for as hyde (1998) remarks, the boundary crosser or even destroyer brings with it benefits. tricksters, he asserts, help to cut the ties that bind us in social and spiritual life, leading to what anthropologist george foster and even plato refer to as a lucky find. a lucky find, hyde explains, reveals a larger view and helps us to realize that our conceptions of things are in our mind rather than out there. this process of cutting ties is important, hyde adds, because cultures have webs of signification built around opposites that members can take as both natural and immutable. tricksters help members of a culture disturb these webs, revealing the fallibility of the immutable ideas. one example of the trickster as boundary-crosser is the first gravedigger in hamlet. he works literally on the boundary between life and death, between above-ground and underground, and (as he asserts) between kings and commoners. his jokes, like those of a court jester, speak truth to power and provide hamlet with a new perspective on his way home from his sea voyage. it is significant that the skull hamlet discovers (and which unmistakably identifies him in a picture) is that of the late jester in the danish court. think about that: when hamlet’s father occupied the throne, he employed a jester to mock him, and to tell him the truth when others would not. with his passing, the court of denmark has no such professional fool. hamlet’s usurping uncle has no one to challenge him, no one to tell him the truth about himself. the selfdeception that pervades claudius’s court ultimately leads to its destruction. note here the echo with brooks and brooks (1999): like good teachers, tricksters point out what is contradictory, and in this way they lead us to think about and even talk about what we assume to be true or known. when we quit talking about what we understand, and accept things as they are, we also quit thinking and live by convention. as hyde (1998) reminds us, things become blocked and go stale when we accept convention without questioning it, accept what we have been told without scrutinizing it, or take truth to be a given. when our expectations are crossed, we often ask, “really?” or we exclaim, “no, that can’t be true.” but each response is the opening for a discussion about what we know and what we find to be true. teachers know that in the classroom, discussion is an important source of learning, for discussion leads students to test both their ideas and their learning. teachers who test their students’ ideas know at heart that they must do so to keep language active and evolving, for as hyde (1998) remarks when thinking about the trickster, “language goes dead [when] cultural practice has hedged it in, and some shameless double-dealer is needed to get outside the rules and set tongues wagging . . .” (76). the trickster, says hyde, creates “lively talk where there has been silence, or were speech has been prohibited. trickster talks freshly where language has been blocked, gone dead, or lost its charm” (76). humor can be one effective catalyst to get discussion going. allan chinen (1993) provides an example of this point when he recounts that among the hopi, tricksters work to keep davis, k. and weeden, s. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 78 people from taking life too seriously. “they usually appear,” he comments, “by suddenly jumping down from high buildings, and then parody tribal priests and officials”; they also “poke fun at marriage and funerals and make light of love and death to prevent people from taking religious dogmas—or life itself—too seriously” (72). echoing chinen in this regard, paul radin (1972) tells us that while tricksters dupe others around them, they can also be duped. thus, reactions to a trickster can be complex, marked by laughter on the one hand and awe on the other. here then is the significance of tony sherrill’s classroom move. when he told students he expected a certain prescribed dress, he counted on their unquestioning acceptance of his authority. when he arrived in clothing that contradicted his own authoritative proclamation, a conversation easily began about what it means to accept without questioning, setting up later critical exchanges in his classroom. in addition, his “tricky” behavior reminded them that the voice of authority is situated rather than immutable, conferred as much as it is imposed, and in learning the lesson, they found themselves in the company of a teacher who represented culture and its boundaries, but who also disrupted accepted cultural training. for many, thinking of teachers as tricksters may seem unnatural. what happens in the classroom is a serious matter, for the development of our students is at stake. some teachers may have trouble imagining that they themselves could stoop to duping their students or doing anything except to complete the serious business of training the next generation. for these teachers, it is important to consider that whether we acknowledge it or not, the classroom is a place where life adventures can, and do, both begin and continue. in the television series joseph campbell and the power of myth, interviewer bill moyers asks campbell why there are so many stories about heroes in the cultures of the world. campbell (1988) responds that the hero’s adventure is a fundamental part of being human. any birth, and later any initiation, that helps to take us from one stage of life to another has the potential to affect us like a hero’s journey. as he remarks, we are in childhood in a condition of dependency under someone’s protection and supervision for some fourteen to twenty-one years—and if you’re going on for your ph.d., this may continue to perhaps thirty-five. you are in no way a self-responsible, free agent, but an obedient dependent, expecting and receiving punishments and rewards. to evolve out of this position of psychological immaturity to the course of self-responsibility and assurance requires a death and resurrection. that’s the basic motif of the universal hero’s journey—leaving one condition and finding the source of life to bring you forth into a richer or mature condition. (124) in your reading of this article, you yourself are undergoing a hero’s journey. you began on familiar ground, with a discussion of storytelling. you’ve entered what may be unfamiliar territory, with our discussion of the monomyth in general and the trickster in particular. we’re about to help you come home to your classroom, bringing what we hope is a boon. every classroom holds the potential to begin a new adventure for the student, or to enhance the one already undertaken. by acting as a trickster, the teacher not only calls the student further upon the journey, but also does so in a way that would appear to reinforce convention. however, if the teacher is acting with insight, with wisdom—that is, craftily—then the teacher also calls convention into question in order to promote students’ growth. it seems to us that the best teachers are those who acknowledge their role as people who will challenge their students to move beyond what the students know and help facilitate their hero’s journey through davis, k. and weeden, s. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 79 the classroom. humor often accompanies these teachers, even if they are not the natural comics that we know among our friends and relatives. can playing the trickster have negative consequences? yes, if the function of trickster-aschallenger comes to dominate over the function of trickster-as-supporter. if students perceive the teacher solely as trick-player, they may well just give up in frustration, believing that the educational deck will always be stacked against them. trickster teachers have to remember that as carl jung (1969) argues, trickster figures can have a therapeutic effect, reminding people of the progress they have made as they gain insight about themselves and the world they live in. jung even goes so far to say that in trickster stories the transforming presence of the savior is suggested, for the savior “brings liberation from the imprisonment in , unconsciousness, and is therefore a bringer of light as well as healing” (272). hyde (1998) builds upon the understanding of tricksters as transformers with his comment that tricksters can be thought of as “the spirit of the doorway leading out” to new insight (6). thus, for scholars like these, tricksters help to transform what is thought meaningless into what is meaningful, and often in ways that echo what we expect of saints. we sometimes think of our own best teachers in this way, as figure who led us to insights we didn’t think possible, even when we thought of them as fools or even unknowledgeable. by playing this dual role—the fool to laugh at but the bringer of wisdom and boons—teachers can use the trickster role to help students reach new levels of understanding, finally, we want to claim that teachers do not really have the choice whether to be tricksters or not. it is hard to avoid being a trickster in the students’ eyes: we are boon givers to them, but we ask them to earn the boons in ways they may find frustrating. thus, as they begin the journeys prompted by what they experience in our classes, our students may find us fools, first for challenging orthodoxy, and second for presenting new ideas so preposterous that they cannot possibly be right. as they descend into this realm of uncertainty, they may question whether we can truly be trusted. this, we think, is at the heart of the emotional response to teaching that challenges. by acknowledging our inevitable trickster role, we (1) accept that we cause frustration, we (2) take advantage of that frustration, and we (3) gain the awareness that we can accomplish what tricksters do in stories: changing the world, making it a better place for humankind. we bring about that change by recognizing that students’ time in our classes as a journey from what is familiar to what is unfamiliar. like the heroes of myths, our students feel separation from what they know, an initiation into a new world of expertise, and—with our intentions, skills, and (yes) luck—a return with boons for themselves and even their worlds. john g. parks (1996), in his article "the teacher as bag lady," discusses the trickster as one of three metaphors for the teacher. he writes: the college years can be thought of as liminal space and time, to use victor turner’s concept. it is a space and time between or at the margins or boundaries of normal historical time. it is the opportunity to explore; experiment; test ideas, identities, and beliefs. liminality offers freedom but also risks, obstacles, and tests, possibly involving pain and suffering. the teacher as trickster can be a guide during this experience. such a teacher is open to improvising, to risking disorder, to threatening boundaries. (135) as we have seen, the role of wise sage is not one should consciously take on. we think instead that we can be like the trickster coyote, who, in the words of richard erdoes and alfonso ortiz (1998), “teaches humans how to live” (xiv). davis, k. and weeden, s. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 80 references brooks, j. g. and brooks, m. g. (1999). in search of understanding: the case for constructivist classrooms. upper saddle river, nj: merrill education/prentice hall. brown, j. s. and duguid, p. (2002). the social life of information. boston: harvard business school press. bruner, j. (1986). actual minds, possible worlds. cambridge, ma: harvard university press. campbell, j. (1949). the hero with a thousand faces. new york: mjf books. campbell, j. with moyers, b. (1988). the power of myth. new york: doubleday. chinen, a. b. (1993). beyond the hero: classic stories of men in search of soul. new york: tarcher/putnam. combs, a. and holland, m. (2001). synchronicity. new york: marlowe and company. cruikshank, j. and sidney, a. (1998). how the world began. coming to light: contemporary translations of the native literatures of north america. ed. swann, b. new york: random house, 138-40. culler, j. (2000). literary theory: a very short introduction. oxford: oxford university press. deporter, b., reardon, m., and singer-nourie, s. (1999). quantum teaching: orchestrating student success. boston: allyn and bacon. kieran, e. (1989). teaching as storytelling: an alternative approach to teaching and curriculum in the elementary school. chicago: university of chicago press. erdoes, richard, and alfonso ortiz, eds. (1998). american indian trickster tales. new york: viking. gersie, a. and king, n. (1990). storymaking in education and therapy. london: jessica kingsley publishers. ghezzi, r. w. (1998). “nanabush stories from the ojibwe.” coming to light: contemporary translations of the native literatures of north america. ed. swann, b. new york: random house, 443-47. gold, j. (2002). the story species: our life-literature connection. markham, on: fitzhenry and whiteside. greene, m. (1995). releasing the imagination: essays on education, the arts, and social change. san francisco: jossey-bass. davis, k. and weeden, s. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 81 haines, s. g. (1998). the manager’s pocket guide to systems thinking and learning. amherst, ma: hrd press. hathaway, n. (2001). the friendly guide to mythology. new york: viking press. hyde, l. (1998). trickster makes this world. new york: farrar, strauss, and giroux. jung, c. g. (1969). the archetypes and the collective unconscious, 2 nd ed. princeton: princeton university press. lindley, d. a. (1993). this rough magic: a life of teaching. westport, ct: bergin and garvey. parks, j. g. (1996). the teacher as bag lady. college teaching, 44 (4), 132-136. polkinghorne, d. e. (1988). narrative knowing and the human sciences. albany: state university of new york press. radin, p. (1972). the trickster: a study in american indian mythology. new york: schocken books. roemer, m. (1995). telling stories: postmodernism and the invalidation of traditional narrative. lanham, md: rowman and littlefield. vogler, c. (1998). the writer’s journey: mythic structures for writers, 2 nd ed. studio city, ca: macail wiese productions. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009, pp. 82 – 97. a typology of relational turning point events in college teacherstudent relationships tony docan-morgan 1 abstract: the purpose of this brief report was to develop a typology of relational turning point events between college teachers and students. participants were 640 undergraduates who were asked whether they could recall a turning point with a teacher, and if so, to report the turning point event in detail. analysis of the data yielded 6 large categories of turning point events, including instrumental (discussion of grade; discussion of course assignment, course content, course more generally; discussion of college, major, independent study, and/or internships; discussion of course policy/rule), personal (discussion of coursework and personal information; discussion of common interest; compliment; invitation; name used), rhetorical (lecture topic or claim; teaching style), ridicule/discipline, locational, and other person. as the first of a series of studies on relational turning point events between teachers and students, the current study presents new insight into the literature on relational turning points and the larger body of literature on teacher-effectiveness. the current study also offers teachers an understanding of how to establish and build meaningful relationships with students. keywords: teacher-student relationship, relational turning point events, undergraduate students i. introduction and literature review. research spanning the field of teacher effectiveness has contributed a wealth of knowledge about the significance of instructor behavior (frymier, 1994; frymier and wanzer, 2006), student motivation (christophel, 1990; docan, 2006), classroom climate (dwyer, bingham, carlson, prisbell, cruz, and fus, 2004), and teaching methods (chesebro, 2003; docan-morgan, 2007; wulff and wulff, 2004) on outcomes such as student learning and motivation. one key element that cuts across each of these areas, albeit rarely recognized explicitly, is that of the relationship between teacher and student. teven (2001) has argued that “in order to maximize learning, it is essential for teachers to develop a good relationship with their students, because the rapport established between teachers and students, in part, determines the interest and performance level of students” (teven, 2001, p. 159). the professor-student relationship has been conceptualized along a continuum of relational development (devito, 1986), which asserts that: (1) teaching can be described as a relational process from initial contact, intimacy or closeness, and dissolution; (2) teacher-student interaction that assists teaching and learning depends in part on the development of an interpersonal relationship; (3) the development of a relationship between student and teacher will 1 communication studies department, university of wisconsin-la crosse, 319 center for the arts, 1725 state street, la crosse, wi 54601, docan.anth@uwlax.edu. docan-morgan, t. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 83 lead to greater satisfaction and more effective learning; and (4) and a failure in teaching can be attributed to the ineffectiveness of the relational development process. devito illustrates that a good teacher-student relationship is not the only goal of teaching; “rather, the development of the interpersonal relationship is viewed as the means by which more effective, efficient, and satisfying teaching and learning may take place” (p. 53). similar to devito (1986), dobransky and frymier (2004) also conceptualize the teacherstudent dynamic as an interpersonal relationship; however, in their analysis, the teacher-student relationship is marked by control, trust, and intimacy as conceived by millar and rogers (1976). millar and rogers posit that the ways control is shared, the level of intimacy, and the amount of trust help define the relationship. in their study, dobransky and frymier found that “students who perceived their teachers as exhibiting higher levels of shared control, trust, and intimacy reported greater learning” (p. 211). not only do relationships exist between teachers and students, but they develop through distinct stages (devito, 1986), involve shared control, trust, and intimacy (dobransky and frymier, 2004), include relational elements such as control and affection (lowman, 1984, 1994), entail relational communication behaviors (graham, west, and schaller, 1992), and are goal-based like other types of relationships (frymier and houser, 2000). one of the most significant factors identified in relationships is that of relational turning points. a relational turning point is “any event or occurrence that is associated with change in a relationship” (baxter and bullis, 1986, p. 470). turning points affect outcomes such as closeness and relational satisfaction (e.g., golish, 2000; surra, 1987), are deemed important by those involved (baxter and bullis, 1986), and often have a powerful effect on relational development (masheter and harris, 1986). only a few studies have examined relational turning points in the academic arena. for example, bullis and bach (1989a, 1989b) examined turning points in the graduate student-professor relationship and discovered nine types of turning points: academic recognition, perceived similarity, mutual confirmation, advising, personal bonding, relational clashes, relational evolution, relational decline, and miscellaneous. to date, only one published study has analyzed turning points in faculty-undergraduate student relationships. o’neill and todd-mancillas (1992) asked 52 college seniors to “recall turning points pertinent to an out-of-the-ordinary (either very positive or very negative) and recent relationship with an instructor” within the classroom environment (p. 282). the results indicate two macro-categories of turning points divided into six sub-categories. the first category was labeled perception of instructional communication competence and character (including subcategories perceived competence and character) and the second was labeled perception of instructor’s management style (including learning climate, course administration style, rhetorical sensitivity, and feedback). although in many ways a useful investigation, o’neill and todd-mancillas’s sample consisted of only 52 students, all of whom were seniors, and only classroom interaction was examined, whereas out-of-classroom encounters may have contributed to relational turning points. o’neill and todd-mancillas’s study does, however, call for further investigation into the relational turning points experienced by students and teachers. in their call, o’neill and todd-mancillas remark that “ultimately, the undertaking of such research can only add further to our understanding the ‘how,’ ‘why,’ and ‘so what’ of studentfaculty relational dynamics” (p. 290). the following study is the first of a series of studies aimed at unraveling the communication events that change the teacher-student relationship. the goal of the current study is to develop a typology of relational turning point events as perceived by students. the following research question was asked: docan-morgan, t. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 84 what specific types of events do students report as relational turning points in teacherstudent relationships? ii. method. a. participants. the sample for this study consisted of 640 students at a large northwestern university recruited from communication courses meeting general-education university requirements and enroll students from all undergraduate majors. students were asked to participate in a study involving teacher-student interaction and accessed a questionnaire on the internet, which allowed students ample time to decide whether to participate in the study, ask the researcher questions, and ensure confidentiality. the questionnaire took approximately 10-20 minutes to complete. participants were offered extra credit and their participation was voluntary. students (n = 17) who did not want to take part in the study or who were enrolled in multiple classes that were solicited for participation were given the option of completing an alternative extra-credit assignment. of those who did participate, six reported on a turning point with a k-12 teacher, thus their cases were deleted from the data set. participants who were able to recall a turning point (n = 394, 62%) subsequently completed open-ended questions about the turning point. the majority of participants able to recall an event (n = 280) were female. their ages ranged from 18 to 49 (m = 20.49, sd = 2.57). two hundred fifty four reported as being white, 84 as asian, 20 as hispanic or latino, 14 as mixed race, 9 as african american, 8 as native hawaiian or pacific islander, 2 as american indian or alaska natives, 2 as middle eastern, and 1 did not report his or her race/ethnicity. four people did not report their academic status; 42 said they were in their first year, 137 were sophomores, 130 were juniors, 79 were seniors, and 2 were graduate students. b. procedure and data analysis. research demonstrates that the recall of specific events is less subject to distortion than summary judgments of events occurring frequently (podsakoff and organ, 1986; schwartz, 1999); therefore, this study utilized the critical incident technique (cit) (flanagan, 1954). the cit, involves asking a number of respondents to identify events or experiences that were ‘critical’ for some purpose. these incidents are then pooled together for analysis, and generalizations about the event or activity are drawn from the commonalities of the incidents. (kain, 2004, p. 71) the cit is appropriate for analyzing relational turning points as it allows participants to provide accounts of their firsthand experiences. given that relational turning points are experienced by relationship partners themselves (e.g., baxter and bullis, 1986; johnson et al., 2003), an analysis of turning points requires that data capture their experiences. kain (2004) remarks that the “appeal of the critical incident technique of research lies largely in this systematic approach to inquiry—into what significance others place on given events” (p. 72). specifically, the cit asks respondents to tell a story and explain why it is significant or important for a given context (kain, 2004). by focusing on a specific event, the contextual and case-specific nature of the phenomena under investigation is captured. the questionnaire defined relational turning point and asked students to report the turning point in detail. docan-morgan, t. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 85 the current study utilized flanagan (1954), woolsey (1986), and kain’s (2004) approach for analyzing data. the cit, according to woolsey (1986), involves “an analysis of thematic content, arrived at by inductive reasoning” with the objective of providing “a detailed, comprehensive and valid description of the activity studied” (p. 248). to begin, i selected a small sample of incidents and sorted them into piles and tentative categories. using the constant comparative approach, i compared examples for similarities and differences (strauss and corbin, 1998). after establishing these tentative categories, i developed definitions for each category, and added additional incidents to each category. throughout this process, i added, redefined, combined, and revised categories until the categories did not need further modification. during the analysis process, i subdivided the larger categories into smaller groups and placed together the incidents that described very nearly the same type of behavior. i then re-examined the definitions for all the categories and major headings in terms of the actual incidents classified under each category and subcategory. iii. results. analysis of the data yielded 6 larger categories of turning point events, some of which were subsequently divided into more specific sub-categories. the six larger categories of relational turning point events were labeled as instrumental, personal, rhetorical, ridicule/ discipline, locational, and other person. each of these categories, as well as their subcategories, are explicated below. the appendix summarizes the types and frequency of turning point events reported. a. instrumental. instrumental relational turning points (n = 170, 43.1%) consisted of events based largely on reports of the teacher helping or assisting a student with class or college-related issues or concerns. four subcategories emerged from the data, including discussion of grade; discussion of course assignment, course content, course more generally; discussion of college, major, independent study, and/or internships; and discussion of course policy/rule. discussion of grade. turning point events that were reported to occur when the student and teacher discussed a student’s grade on an exam, assignment, or a final course grade were categorized as discussion of grade. other applicable turning points included in this category consisted of a teacher grading a student’s work in his or her presence and discussing it, discussing consequences of a grade, and how to make up a bad grade (n = 21, 5.3%). for example, one student wrote the following: i went to my professor to ask about a question i got wrong on a test to see if i could possibly get some points back. i showed him the question and explained why i think i should get it right in clear detail. he took very little time to hear what i had to say and did not give me any points or an explanation of why not. as can be seen, the focus was on talking about the grade. another instrumental, grade-related turning point reported by a student entailed the following: docan-morgan, t. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 86 i scheduled a meeting to discuss a possible error in exam scores. i knocked on the professor's door to find that there was a miscommunication with the meeting time. i had come earlier than expected, but was welcomed in anyway. we looked over the scores to find that i was actually given a higher score when i had thought i was graded lower. the professor gave me the higher grade and assured me that the low test grade would not affect my overall grade and that i had nothing to worry about. again, the primary focus of the event described was the grade discussion, be it a positive or negative experience from the students’ perspective. discussion of course assignment, course content, course more generally. turning point events that occurred when a student and teacher discussed an assignment (e.g., paper, upcoming or previous exam), course concept or topic, a student’s progress or standing in course, or the course more generally comprised the current category (n = 111, 28.2%). for example, one student’s discussion with her/his teacher about a class presentation served as a turning point: in my class, we have to write a marketing plan that has to be 15 pages long, and we have to give a presentation by the end of the quarter. my team members and i were not sure how the presentation should be since the teacher did not give specific instructions. i decided to ask my instructor after the class is over. another student responded: my ta explained to me for two hours the entire overview of political science … international relations and didn't laugh no matter how obvious the answer was to him. what i realized was that he actually cared about my learning and that this wasn't such a cold university. discussion of college, major, independent study, and/or internships. this subcategory comprised turning points that occurred when the main topic that a teacher and student discussed was college, a specific major or majors, what classes to take, independent study possibilities, job opportunities, job offers, future goals, studying abroad, and/or internship possibilities, and letters of recommendation (n = 26, 6.6%). some students reported a turning point occurring when they approached their teacher for career and graduate school advice: i went to her office to speak with her about my decision of where to go to school. she was a communication teacher. she came from a smaller, intimate college for her undergraduate work. i wanted her opinion. in another response, a student’s instrumental request focused on a school-related opportunity: there were many other people in the cafe other then myself and my professor. the event was when i asked him to write a recommendation for me. i am planning on studying abroad this summer and i needed a letter of recommendation from him. this meeting was when i was able to give him more information about myself. docan-morgan, t. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 87 discussion of course policy/rule. turning point events that occurred when a student and teacher discussed a course policy or rule (e.g., turning in an assignment late, extending a deadline, time to complete a test) or a teacher allowing students to have extra time on an assignment or test were categorized as discussion of course policy/rule (n = 12, 3.0%). for example, one students reported the following turning point: it happened in the morning (approx. 9:20am), at the end of class. in a classroom. i approached the teacher. a female grad student who was teaching the technical communication class. i asked why she had announced a pop quiz the day before, after i and several other students left the class early. i had notified her prior to this that i had a class far away right afterwards, and that some days i would be leaving early to make it to tests, quizzes, etc. on time. in a similar discussion, one student perceived a change in her/his relationship with the instructor based on a conversation about making up an exam: i missed my midterm earlier that day, and after i emailed my professor and explained the situation to her, she was kind enough to let me take the midterm later that day with no consequence to my grade. she was extremely kind and considerate about my situation. b. personal. turning point events based largely on the sharing of private, personal information, or if there was a specific, approach/affinity seeking behavior or statement intended for one person (e.g., compliment, invitation, name used) were labeled as personal (n = 104, 26.4%). five subcategories emerged from the data. discussion of coursework and personal information. turning point events occurring when the teacher and student discussed coursework (e.g., paper, grade, participation, late work, course/topic/subject) or were in the context of the class (e.g., during class activity, after class) and discussed personal information (e.g., includes discussion of events that happened in life (e.g., life experiences, recent tragedies/sickness, family, background, personal life, weekends) (n = 46, 11.7%) were labeled as discussion of coursework and personal information. the following example illustrates one relational turning point event between a teacher and student coded in the current category: i had to talk to [my teacher] about my final project ideas. i arrived about 10 minutes early and my professor wasn't in yet. he had posters of "no war on iraq" on his door, and on his window there were various signs of his beliefs and interest as well as student work (for example gay and lesbian rights). this explained that he is very interested in his students' interests, ideas, and works. during my session with him, he was first interested in who i am as a person. he asked about my ethnic background, my hobbies, my beliefs, and many other questions. then we went into detail about my ideas. this was used in a way to bring out what i might want to do about for project. for the first time, i felt that a professor was engaged with what i had to say. it was the professor feeding me more and more information but we were working together to help me. docan-morgan, t. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 88 discussion of common interest. turning points occurring when a teacher and student discussed a common interest or experiences, either course related (e.g., chemistry research) or not course related (e.g., favorite car) were categorized as discussion of common interest (n = 15, 3.8%). for example, one student remarked, “…after class…we began to talk about authors and found a common interest in faulkner. i was just introduced to the author and he was writing his doctoral thesis on it.” another student reported that a turning point occurred when: the class was asked if anyone knew the make/model/year of an automotive figure on an overhead projector in a course during my freshman year. the model car was a 1963 vw type 1 beetle. i raised my hand and nailed it dead on. i then went on to explain details which the model lacked, such as the standard production vent windows and a few other features. the teacher then mentioned that he was a vw beetle enthusiast and owned a superbeetle. i stated that i also was a large vw enthusiast and had restored/customized a 1967 beetle. we then talked for about ten more minutes during lecture about our vehicles. compliment. turning points occurring when a teacher provided a student with an expression of praise, commendation, or admiration were categorized as compliments (n = 21, 5.3%). responses indicating that a teacher thanked or disclosed appreciation for a student’s participation in class, complimented student on their behavior in the classroom, or offered verbal or written praise on work returned to a student were labeled compliments. one student perceived a change in her/his relationship with the teacher in this event: it was outside of the regular class time and was just between the teacher and i. i sought out my accounting teacher for extra help. he was able to explain things much more clearly than in class and also displayed a strong desire to make sure that i truly understood the material. he also let me know how much he appreciated my class participation and eagerness to learn in his class. invitation. turning points occurring when a teacher or student requested the other’s presence or participation of the other, or a request to participate or be present or take part in something were categorized as invitation (n = 8, 2.0%). invitations were made for attendance at a meeting, going out to eat or for a walk, and to participate in a group or activity. for example, one student responded “my ta asked me to go for a walk outside and asked me on a date.” another student reported: after enrolling in this class, i promptly found out it was not a regular class. we did not have regular meeting times, but rather we arranged times to meet with the professor personally. additionally, the professor is very personable and invited a few of his students to attend a mit alumni meeting that google was hosting. i jumped at the invitation. name used. turning points reported by students that focused on the teacher using or failing to use the student’s name were labeled within this category (n = 14, 3.6%). for example, one student reported a turning point with her or his teacher at a basketball game: “i saw my docan-morgan, t. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 89 professor and he remembered my name. out of 120 students in the class he remembered me!” within the classroom context, another student responded: my professor had a seating chart the first day of class and the next day knew everyone’s names. this helped me get engaged in the class and the material. i felt like she new me when she called me by my first name. this makes a huge difference and feel like i had a part in the class. c. rhetorical. turning points based largely on a teacher-directed behavior or statement intended for the entire class were labeled rhetorical (n = 57, 14.5%). two subcategories include lecture topic or claim and teaching style. lecture topic or claim. turning points that occurred when a teacher discussed a topic, made a claim, provided an example, or discussed experiences related to a topic, the course, or college during lecture were labeled lecture topic or claim (n = 31, 07.9%). the following response is a turning point reported by one student: the event occurred in…the afternoon during a 2 hour lecture. a discussion was raised regarding the general acceptance of illegal immigrants in certain regions of the country and whether or not this was acceptable. in a similar response, another student reported: rather than being taught from an objective perspective, this particular professor immediately used this class as a platform to instill his own beliefs onto us. we were talking about race matters in america when he concluded that we should all be active in our fight against racism through any means, including violence and destruction of property. teaching style. rhetorical turning points also concerned the teacher’s style (n = 26, 6.6%) or the manner by which the teacher engaged or failed to engage the students (e.g., informal, held everyone’s attention, interacted in funny manner; used humor). such turning points also included the manner by which the teacher led discussion and a teacher’s overt discussion of her or his teaching style. for example, one student reported the following: [f]irst day of class i was just very impressed with the teacher's personality and teaching style. she had a great ability to catch and hold the attention of everyone in this huge lecture hall. one teacher’s discussion of her teaching style emerged as a turning point for a student when she explained to the class that part of her teaching style was not to shy away from controversial or potentially touchy subject matter during class discussions. she believed that encouraging lively debate about real-world issues would improve the quality of the essays we were to write for the class. i remember that this was a turning point for me because it represented one my first experiences outside of the high school environment docan-morgan, t. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 90 where everything (especially subject matter) is so much more controlled and sterilized. i remember this event as being very refreshing, worthwhile, and effective. d. ridicule/discipline. another larger category of turning points reported by students included being ridiculed or threatened by the teacher (n = 31, 7.9%). responses mentioning ridicule entailed the teacher using language or behavior in a mocking or humiliating manner. responses mentioning being disciplined entailed the teacher bringing the student to a state of order and obedience or punishing the student. the following event was reported as a turning point by one student: the event took place in a classroom during class in front of all other students enrolled. i had recently missed a day of class because i had to attend my grandfather's funeral. my professor asked me to get a note written by someone important having to do with the funeral in order for me to not lose credit for missing class. my mom wrote a note for me to give to him the next day. he took one look at the note and said that his name was not correct. at first i thought it was not correct because it was written in english and he was a spanish professor. then he proceeded to tell me that it was still not the reason he said it was wrong. so i again said another option of his name and he still said it was wrong. so the day after my grandfather's funeral in front of our entire spanish class, i had to stand there and be ridiculed by him. e. locational. turning point events based on being in a different location or environment than normal with a teacher were categorized as locational (n = 25, 6.3%). students’ responses focused on seeing their teacher outside of the usual academic environment. the following example illustrates the location-based nature of this type of turning point: the location was at a barbeque at our professor’s house during the day celebrating my professors birthday. the setting was very informal where we talked and ate and changed the dynamic between us lab students and our professor. similarly, students reported locational turning points off campus, including the following: i was at the train station and went into the bathroom to wash my hands. next to me was one of my professors from the previous quarter. i had to say hello just because it was so weird seeing her outside of school. i told her i had been in her class and how i enjoyed it. then we talked about where we were traveling. f. other person. the final category of turning points, labeled other person, includes turning points that occurred not because of the teacher’s behavior, but instead as a result of someone else’s behavior (n = 7, 1.8%). the majority of these turning points occurred because of a third party’s interaction with the student. the following examples illustrate the other person category: docan-morgan, t. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 91 the professor was sent out of the room and a lady from some sort of office came in and we discussed the good and bad teaching techniques used by our professor. the lady created a memo for the professor telling her what we thought she could improve on for the rest of the quarter. another reported, early in the quarter, probably the 2nd week, another student came up to me during our 10-minute break in class. she asked me my name and then told me she would need to be absent from class one day the following week. she said our teacher had recommended for her to get the notes for that day from me. iv. discussion. the purpose of this study was to identify relational turning point events that college students perceive occurring with their teachers. previous research posits that the teacher-student relationship develops through distinct stages (devito, 1986), includes relational elements such as control and affection (lowman, 1984, 1994), entails relational communication behaviors (graham et al., 1992), and is defined by control, trust, and intimacy (dobransky and frymier, 2004). however, one of the most salient features of relationships—that of relational turning points—merits explication in the teacher-student context. the present study attempted to establish a typology of relational turning point events as perceived by students from which the teacher-student relationship could be more readily understood. the events identified in the current study provide a richer conceptualization not only of the teacher-student relationship, but helps us better understand one-time events that shape students’ perceptions of their relationships with their teachers. in their analysis of relational turning point events, o’neill and todd-mancillas (1992) advance two macro-categories and six sub-categories of turning point events: perception of instructional communication competence and character (perceived competence and character) and perception of instructor’s management style (learning climate, course administration style, rhetorical sensitivity, and feedback). the current study extends their work by providing six macro-categories, some of which are divided into sub-categories: instrumental (discussion of grade; discussion of course assignment, course content, course more generally; discussion of college, major, independent study, and/or internships; discussion of course policy or rule), personal (discussion of coursework and personal information; discussion of common interest; compliment; invitation; name used), rhetorical (lecture topic or claim; teaching style), ridicule/discipline, locational, and other person. upon a closer examination, there are notable similarities and differences between these two typologies of turning point events. both studies address relational turning point events based on the use of a student’s name, a course policy or course administration style, grading, and the location at which the turning point event occurred. when comparing these studies, three types of turning points appear to align more closely, however. in particular, being ridiculed or disciplined by a teacher was a common theme in both studies. although they do not provide a separate category for this type of turning point event, o’neill and todd-mancillas (1992) refer to issues of ridicule in their sub-categories of character (e.g., students commented that their professor was “condescending to students,” “yelled at the students,” and “made derogatory remarks”) and rhetorical sensitivity (e.g., one student reported docan-morgan, t. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 92 that a teacher stated, “didn’t you listen to me during lecture when i assigned the project?”). similar types of turning point events were found in the current study and subsequently labeled ridicule/discipline. for some students, then, being ridiculed by their teacher appears to change their relationship. seeing that students identify being ridiculed by their teacher as a significant moment in their education, teachers should be aware of the effects of their disciplinary strategies. another relational turning point event common in both studies concerned the teacher’s style. o’neill and todd-mancillas’s (1992) sub-categories of perceived competence and learning climate, as well as the current study’s subcategory teaching style, address the manner by which teachers engage or fail to engage students. for example, both studies discuss relational turning point events based on the teacher using humor, being enthusiastic about their teaching, and presenting lectures that were difficult to follow. a teacher’s style, then, also appears to facilitate relational turning points for some students. finally, both studies found that one-on-one interactions in which the student and teacher engaged in what students described as a more “personal interaction” facilitated a relational turning point. o’neill and todd-mancillas’s (1992) sub-category rhetorical sensitivity, in-part, addressed getting to know teachers on a more personal level, as well as one-on-one interactions whereby teachers were described as understanding, helpful, receptive, and friendly. in the current study, the macro-category, personal, entailed the sharing of private, personal information, or the presence of a specific, approach/affinity seeking behavior or statement (e.g., compliment, invitation, name used). one-on-one interactions that involved sharing personal information and/or when the teacher is perceived as receptive appear to facilitate relational turning points for students. the commonalities between o’neill and todd-mancillas’s and the current study demonstrate that a diverse range of events work to facilitate relational turning points for students. more importantly, perhaps, students in both studies illuminated the importance of teachers acting in ways that are understanding, helpful, receptive, personal, and friendly. although previous research, specifically o’neill and todd-mancillas’s 1992 study, and the current investigation have important crossover, the differences between the two also help advance our understanding of relational turning points in college teacher-student relationships. perhaps the most salient difference is that o’neill and todd-mancillas’s typology of turning point events focuses on students’ judgments of the professor (e.g., perception of instructional communication competence and character, perception of instructor’s management style), whereas the typology advanced in the current study focuses on perceptions of events (e.g., discussion of common interests). in previous research, turning points events have been conceptualized broadly as intrapersonal/normative (individual evaluates him or herself, the partner, or relationship against an ideal or standard), dyadic (occurs in interaction with another), social network (individual from either or both partner’s social network affects the relationship), or circumstantial (an event beyond the dyad’s control affects the relationship) (surra and huston, 1987). similarly, the typology advanced in the current study offers a description of turning point events that involve dyadic (e.g., labeled instrumental, personal, ridicule/discipline), social network (e.g., labeled other person), and circumstantial (e.g., labeled locational) turning point events. the general typology for turning point events asserted by surra and huston (1987) is therefore represented effectively in the typology advanced in the current study aimed at the teacher-student relationship. bullis et al. (1993) intimate that a typology of turning point events in any context should account for the entire spectrum of events. the typology of relational turning point events docan-morgan, t. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 93 articulated in the current study is descriptive of actual events and accounts for a wide range of behaviors enacted in the teacher-student context. the macro-categories in the current study clearly describe the nature of turning point events in adjective form (e.g., instrumental, personal, rhetorical), and the sub-categories offer a specific classification of events. the descriptive focus on types of events in the current study versus perceptions more generally offers a richer account of the many interactions that teachers and students engage. the typology advanced here also recognizes the diverse contexts in which teachers and students interact (e.g., rhetorical or during lecture, personal which is often one-on-one). the current study elucidates that relational turning point events in college teacher-student relationships are dynamic. first, a diverse range of events can facilitate turning points, including using of a student’s name, discussing a course policy, grading, ridiculing, sharing personal information, or interacting in a new environment, for example. second, turning point events can occur in virtually any context of interaction (e.g., during a fieldtrip, lecture, office hours, walking around campus, in public, in a hallway). finally, turning point events appear to come in a number of types in the teacher-student context, including dyadic, social network, and circumstantial. overall, then, relational turning point events are dynamic in that nearly any event may work as a turning point for a student, turning points can occur in all communication contexts, and they can vary in type. v. implications, future directions, and conclusions. one of the major implications of this study is that teachers have the capacity to change their relationships with students in a multitude of meaningful ways. indeed, college teachers should understand that their everyday behaviors, or lack thereof, whether calling a student by name, providing a compliment, engaging in a discussion of common interests, discussing a course policy, ridiculing a student, disclosing personal information, or their own particular teaching style has the potential to and often does shape students’ perceptions of their relationship. further, taking steps proactively to change relationships with students will likely be advantageous (e.g., setting up clear grading procedures, calling students by name, engaging students in discussions about internships and careers, giving compliments, sharing personal information, avoiding ridicule). these behaviors may also have strong implications for instructional outcomes, including cognitive and affective learning and student motivation, each of which will be explored in a future study. additional studies investigating the turning points in teacher-student relationships will likely lead to other meaningful implications. researchers need to examine relational turning points from college teachers’ perspective as well as assess the ways in which teachers and students work together in their understanding of the event. seeing that relationships involve interdependence (wood, 2000), develop through interaction (canary and dainton, 2006), are maintained through communication (canary and dainton, 2006), and entail a bond that unites partners (mccall, 1970), teachers’ experiences must be captured to understand the complexities of relational turning points with students. a study similar in nature to the current analysis of turning points (i.e., large sample, use of the cit), but from teachers’ perspectives, may provide telling similarities and differences when compared with students’ reporting of turning points. important questions include: how are teachers’ perceptions of their relationships with students affected by relational turning points? docan-morgan, t. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 94 how are important teacher outcomes such as teacher efficacy, job satisfaction, and motivation (see mottet et al., 2006) affected by relational turning points with their students? the current study opens up an assortment of other instructional issues to investigate. keeping in mind that change is inherent in relationships (wood, 2000), stage models of teacherstudent relationships that identify when and how stages change have the opportunity to advance previous models of the teacher-student relationship (devito, 1986) and provide a fuller explanation and description of this dynamic between teachers and students. some key questions include: what are the stages of teaching assistant-undergraduate student, teacher-undergraduate student, and professor-graduate student relationships? what prompts change from one stage to the next? at what stage, if any, is student learning at its peak? how can teachers facilitate their relationships with students to arrive at a stage optimal for learning? the results of the current study expand and present new insight into the literature on relational turning points and the larger body of teacher-effectiveness literature. in particular, turning point events reported by students ranged from instrumental, personal, rhetorical, ridicule/discipline, locational, and other person events. as important, perhaps, the current study paves the way for future studies of relational turning points in the academic context. finally, the current study offers teachers an understanding of how to interact with students in a way that establishes and builds meaningful relationships with students. appendix 1. turning point events. turning point event frequency instrumental discussion of grade discussion of course assignment, course content, course more generally discussion of college, major, independent study, and/or internships discussion of course policy/rule n = 170 (43.1%) n = 21 (5.3%) n = 111 (28.2%) n = 26 (6.6%) n = 12 (3.0%) personal discussion of coursework and personal information discussion of common interest compliment invitation name used n = 104 (26.4%) n = 46 (11.7%) n = 15 (3.8%) n = 21 (5.3%) n = 8 (2.0%) n = 14 (3.6%) rhetorical lecture topic or claim teaching style n = 57 (14.5%) n = 31 (7.9%) n = 26 (6.6%) ridicule/discipline n = 31 (7.9%) locational n = 25 (6.3%) other person n = 7 (1.8%) docan-morgan, t. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 95 references baxter, l. a. and bullis, c. (1986). turning points in developing romantic relationships. human communication research, 12, 469-493. bullis, c. and bach, b. w. (1989a). socialization turning points: an examination of change in organizational identification. western journal of speech communication, 53, 273-293. bullis, c. and bach, b. w. (1989b). are mentor relationships helping organizations? an exploration of developing mentee-mentor-organizational identifications using turning point analysis. communication quarterly, 37, 199-214. bullis, c., clark, c. and sline, r. (1993). from passion to commitment: turning points in romantic relationships. in p. j. kalbfleisch (ed.), interpersonal communication: evolving interpersonal relationships (pp. 213-236). hillsdale, nj: lawrence erlbaum associates. canary, d. j. and dainton, m. (2006). maintaining relationships. in a. l. vangelisti and d. perlman (eds.), the cambridge handbook of personal relationships (pp. 727-743). new york: cambridge university press. chesebro, j. l. (2003). effects of teacher clarity and nonverbal immediacy on student learning, receiver apprehension and affect. communication education, 52, 135-147. christophel, d. m. (1990). the relationships among teacher immediacy behaviors, student motivation, and learning. communication education, 39, 323-341. devito, j. a. (1986). teaching as relational development. in j. m. civikly (ed.), communicating in college classrooms (pp. 51-59). san francisco: jossey-bass. dobransky, n. d. and frymier, a. b. (2004). developing teacher-student relationships through out of class communication. communication quarterly, 52, 211-223. docan, t. (2006). positive and negative incentives in the classroom: an analysis of grading systems and student motivation. journal of scholarship of teaching and learning, 6, 21-40. docan-morgan, t. (2007). writing and communicating instructional objectives. in b. hugenberg, l. hugenberg, s. morreale, d. worley and d. worley (eds.), basic communication course best practices: a training manual for instructors (pp. 25-41). dubuque, ia: kendall/hunt publishing company. dwyer, k. k., bingham, s. g., carlson, r. e., prisbell, m., cruz, a. and fus, d. (2004). communication and connectedness in the classroom: development of the connected classroom climate inventory. communication research reports, 21, 246-272. flanagan, j. c. (1954). the critical incident technique. psychological bulletin, 51, 4, 327-358. docan-morgan, t. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 96 frymier, a. (1994). the use of affinity-seeking in producing liking and learning in the classroom. journal of applied communication research, 22, 87-105. frymier, a. b. and houser, m. l. (2000). the teacher-student relationship as an interpersonal relationship. communication education, 49, 207-219. frymier, a. b. and wanzer, m. b. (2006). teacher and student affinity-seeking in the classroom. in t. p. mottet, v. p. richmond and j. c. mccroskey (eds.), handbook of instructional communication: rhetorical and relational perspectives (pp. 195-211). boston: allyn and bacon. golish, t. d. (2000). changes in closeness between adult children and their parents: a turning point analysis. communication reports, 13, 79-97. graham, e. e., west, r. and schaller, k. a. (1992). the association between the relational teaching approach and teacher job satisfaction. communication reports, 5, 11-22. johnson, a., wittenberg, e., villagran, m., mazur, m. and villagran, p. (2003). relational progression as a dialectic: examining turning points in communication among friends. communication monographs, 70, 230-249. kain, d. l. (2004). owning significance: the critical incident technique in research. in k. b. demarrais and s. d. lapan, (eds.). foundations for research: method of inquiry in education and the social sciences (pp. 69-85). mahwah, nj: lawrence erlbaum. lowman, j. (1984/1994). mastering the techniques of teaching. san francisco, ca: jossey bass. masheter, c. and harris, l. m. (1986). from divorce to friendship: a study of dialectic relationship development. journal of social and personal relationships, 3, 177-189. mccall, g. j. (1970). the social organization of relationships. in g. j. mccall, m. m. mccall, n. k. denzin, g. d. suttles and s. b. kurth (eds), social relationships (pp. 3-34). chicago: aldine. millar, f. e. and rogers, l. e. (1976). a relational approach to interpersonal communication. in g. r. miller (ed.), explorations in interpersonal communication (pp. 87-103). beverly hills: sage. o’neill, k. l. and todd-mancillas, w. t. (1992). an investigation into the types of turning point events affecting relational change in student-faculty interactions. innovative higher education, 16, 277-290. podsakoff, p. m. and organ, d. w. (1986). self-reports in organizational research: problems and prospects. journal of management, 12, 531-544. docan-morgan, t. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 97 schwartz, n. (1999). self-reports: how the questions shape the answers. american psychologist, 54, 71-84. strauss, a. and corbin, j. (1998). basics of qualitative research: grounded theory procedures and techniques (2 nd ed.). thousand oaks, ca: sage publications. surra, c. (1987). reasons for changes in commitment: variations by courtship type. journal of social and personal relationships, 4, 17-33. surra, c. and huston, t. l. (1987). mate selection as a social transition. in d. perlman and s. duck (eds.), intimate relationships: development, dynamics, and deterioration (pp. 88-120). beverly hills: sage. teven, j. j. (2001). the relationships among teacher characteristics and perceived caring. communication education, 50, 159-169. wood, j. t. (2000). relational communication: continuity and change in personal relationships (2nd ed.). belmont, ca: wadsworth. woolsey, l. k. (1986). critical incident technique: an innovative qualitative method of research. canadian journal of counseling, 20, 242-254. wulff, s. s. and wulff, d. h. (2004). “of course i’m communicating; i lecture everyday”: enhancing teaching and learning in introductory statistics. communication education, 53, 92102. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009, pp. 98 – 116. student stalking of faculty: impact and prevalence robin k. morgan 1 abstract: this study investigated the incidence of faculty stalking by students in a large university system (eight campuses). a subsample of stalked faculty members was interviewed. results are discussed in relation to categorization schemes for stalking, faculty-student interaction, changes in teaching methods, and the unique problems engendered by students stalking faculty members. keywords: stalking, faculty-student interaction, harassment, teaching climate, social constructivism i. introduction. in october of 2002, three nursing professors at the university of arizona were killed by a despondent student who claimed, in a suicide letter, he murdered the professors for giving him failing grades (lenckus, 2002). in the year prior to this attack, these nursing professors were repeatedly harassed and stalked by this student (j. haase, personal communication, june 20, 2007). fortunately, not all such incidents result in murder, as exemplified by the charges filed against a student at the university of maryland at college park for threatening a professor with a handgun in an attempt to manipulate the professor into providing him an a (schneider and basinger, 1998). more recently, a former graduate student at loyola university attempted to burn his professor’s house down in response to having received a failing grade (collins, 2006). in the year prior to setting the fires, the student had made repeated harassing phone calls to the professor (chronicle of higher education, 2006). ii. stalking in the united states. since the early 1990's stalking has emerged as a significant social and policy concern (fisher, cullen, and turner, 2002). today all fifty states and the district of columbia have implemented anti-stalking laws (marks, 1997), yet state-level statistics on the number of people charged, prosecuted, or convicted of stalking are not readily available, with estimates varying widely. this discrepancy reflects the ambiguity associated with the definition of stalking behavior itself. several researchers have attempted to isolate and describe stalking behavior into categories that are easily accessible to both law enforcement and mental-health professionals. although studies differ in their definition of stalking, some elements are fairly consistent (romeo, 2001). stalking involves repeated and persistent unwanted communications and contacts that create fear in the target. stalking differs from harassment in that harassment is annoying while stalking leads to fear, feeling threatened, or intimidated (purcell, pathé, and mullen, 2004). a standard list of stalking behaviors might include: abusive/excessive telephone calls, letters, or emails to the person's home/work; trespassing, following or threatening the target or the target's friends/relatives, obsessively observing the target from a distance, driving by the 1 school of social sciences, indiana university southeast, 4201 grant line road, new albany, in 47150, rmorgan@ius.edu. morgan, r. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 99 person's home, school, or work, and vandalizing the person's property. many of these activities can be seemingly innocuous in the beginning, but progress into a pattern of activities that introduces terror into the lives of victims. the term “stalking” has only recently been used to describe behavior directed towards the general population (i.e., to someone other than a celebrity). the term first appeared during the 1970s primarily in the context of obsessed fans who intrusively followed and interacted with famous individuals, such as movie stars (mullen, 2003). since these initial accounts, researchers have reported rates of stalking between 2 and 33%, with a national average between 5 and 6% (spitzberg and cupach, 2003). basile, swahn, chen, and saltzman (2006), for example, report stalking in 7% of the women and 2% of the men contacted in a cross-sectional, random digit-dial telephone survey. similarly, in the national violence against women survey (tjaden and thoennes, 2000), 2.2% of men and 8.1% of women reported being stalked. some of this variability may be a result of how those who have experienced stalking perceive the level of threat. turmanis and brown (2006), for example, revealed that 23.4% report stalking when they are asked about behaviors that led to their being ‘a little or somewhat’ fearful, 12.3% when fear was moderate, and 4.7% when fear was of a ‘significant degree.’ across studies, women appear to be at higher risk for being stalked, especially when they are single and below the age of 55 (basile, et al, 2006). the most common patterns of stalking involve a stalker who had a prior intimate relationship with the person being stalked (roberts and dziegielwski, 2006). pathé and mullen (1997) reported stalking behavior is frequently triggered when a close relationship ends, with the stalker attempting to affect reconciliation or gaining revenge. key characteristics of stalking behavior in such situations are jealousy and possessiveness (mullen, pathé and purcell, 2000). dziegielwski and roberts (1995) proposed three categories of stalkers: the domestic violence stalker (the most common representing 7580% of stalking cases involving a need to establish, continue, or re-establish a domestic relationship), the erotomanic/delusional stalker (where the stalker becomes fixated on a person with whom no prior relationship may have occurred), and the nuisance stalker (where the stalker continually harasses with emails, telephone calls, or shows up at the victim’s workplace or home). hall (1998) describes two categories of stalkers: the domestic violence stalker and stalkers who seek revenge. similar to dziegielwski and roberts, hall also found the domestic stalkers to be the most common type. iii. stalking on college campuses. studies investigating stalking on college campuses have almost consistently focused on student stalking of other students. fisher, cullen, and turner (2002), for example, surveyed a nationwide sample of almost 4,500 randomly selected female college students. approximately 13% of these women reported being the victims of stalking incidents, a figure much higher than the 5-6% generally accepted as the national average. interestingly, 83% of these women had not reported the stalking to any university official. bjerregaard (2002) also found that 21% of her sample of college students (24.7% of female students and 10.9% of male students) reported past experiences of stalking and 6% reported currently being stalked. overall, female students are at a greater risk of being stalked and male students, typically an ex-boyfriend, are most likely to be their stalkers. little attention has been given to whether faculty members are at risk of being stalked by their students. this apparent discrepancy is even more disconcerting when considering the morgan, r. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 100 pedagogical changes recommended to college faculty over the past two decades. partly as a function of constructivist theories, faculty members have been encouraged to engage in teaching practices that increase faculty-student interaction. social constructivism, most closely identified with lev vygotsky, argues that learning is enhanced when instructors create an interactive environment designed to enhance learning experiences (llewellyn, 2002). such interaction has been shown to enhance the quality of education provided to students (astin, 1994) and has been the focus of numerous studies. student achievement, whether measured by grades, standardized tests, or self-reported learning, increases as a function of quality faculty-student interaction (anaya, 1999). accordingly, faculty are encouraged to increase interaction with students (holmes, rupert, ross, and shapera, 1999). faculty may become involved with students in multiple roles – as an academic advisor, an instructor, a mentor for research, a supervisor for internship – resulting in a blurring of boundaries (e.g., biaggio, paget, and chenoweth, 1997; feldman-summers, 1989; kitchener, 1992). increased faculty-student interaction may open the door for increased harassment or stalking of faculty by students. batty (2004) reported that female academics in great britain reported physical attacks, stalking and heckling by students. it would seem evident that such experiences would have a significant impact on future faculty-student interactions. for example, bloom (2000) reports on a course he taught where he was heckled by students in the classroom. as a result, bloom changed his teaching strategy by providing an introductory motivational anecdote on the history underlying each topic, leading to less heckling and more positive interactions with his students. in this case, it can be argued that negative interaction with his students led bloom to make a potentially positive change in his teaching methods. however, bloom’s experience may have been idiosyncratic. it seems just as likely that professors may be making changes in their teaching in an effort to protect themselves from such harassment and stalking that detract from teaching effectiveness. in many ways, it is questionable whether the majority of research that has been conducted on stalking would apply to faculty who are stalked by students. as reported by roberts and dziegielwski (2006), in most stalking situations there is a prior intimate relationship between the stalker and the person being stalked. in the case of faculty, there should be no prior intimate relationship as such relationships are frequently forbidden by university policies (wilson, 2007). the clearest parallel to faculty being stalked by students might be cases in which mental health professionals are stalked by their clients. mcivor and petch (2006), for example, found that among mental health professionals male therapists were more likely to be stalked by female clients. this contrasts with the findings of females being more likely to be stalked in relationship based stalking incidents. mcivor and petch also reported three distinct patterns of stalkers among clients who stalked their therapists: those who suffered from personality disorders, those who experienced drug and alcohol problems, and those who had a history of behavioral problems. in a similar vein, hudsonallez (2006) reported 24% of her sample of mental health professionals had experienced at least one incident of being stalked by a client. the clients most likely to stalk their therapist in this study were described as needy clients who made early attachments to their therapists, clients who were sexually attracted to their therapists, and clients who suffered from personality disorders, especially narcissistic personality disorder. the present study, then, addresses three major sets of questions regarding the similarities between student stalking of faculty and stalking in the general population. first, what is the incidence of student stalking of faculty? is the incidence of student stalking of faculty consistent with the rate of stalking in the general population, more similar to the rate of student stalking of morgan, r. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 101 other students, or consistent with client stalking of therapists? given the prior literature, there are several obvious sub-questions relating to incidence. is stalking more prevalent on smaller, commuter campuses, or on larger, residential campuses? is such stalking more common among faculty teaching smaller or larger classes? are female faculty members or male faculty members more likely to be stalked? second, what types of stalking behaviors do faculty most commonly report? given the prior literature, there is an obvious sub-question relating to stalking behaviors. the literature reports domestic stalking, that is, stalking arising from a previous relationship, as being the most common form of stalking. would this be true among students stalking faculty where no intimate relationships may exist prior to classroom interaction? finally, the third major question concerns whether student stalking of faculty impacts future teaching behaviors of faculty? specifically, does student stalking of faculty change the interactional patterns of faculty with their students? if so, what types of changes are faculty making in response to such incidents? vi. phase one. a. method. participants: nine hundred and sixty-eight full-time faculty members from all eight campuses of a large midwestern university volunteered for the present study; of these, 934 surveys were fully completed and usable for data analysis. full-time faculty members were contacted via a university supplied email list, with a response rate of 31%. response rates by campus varied from 23% to 70%. faculty reported teaching a wide variety of classes with many teaching at both the undergraduate and graduate level; sixty-one percent of faculty taught graduate classes, 91% taught undergraduate classes, and 6% taught medical students. twenty percent of the faculty reported teaching large courses (over 100 students). the age range of full-time faculty members was 27 to 81 with a mean age of 49 years (sd = 9.71). of those participating in the study, 55% were men and 45% were women. this contrasts with a university-wide gender ratio of 62% men and 38% women. time spent teaching in the university system ranged from year to 40 years with a mean of 13 years (sd = 10.1). materials: a demographic questionnaire and a modified form of the obsessive relational intrusion scale (ori) short form was given to all faculty members choosing to participate in the present study. the demographic questionnaire consisted of the participant’s sex, age, campus of employment, how long they have been teaching at the college level, and what type of students were taught. the obsessive relational intrusion scale-short form (cupach and spitzberg, 1998) consists of 28 questions assessing stalking behaviors on a five point likert scale ranging from 0 (never) to 4 (over 5 times). for example, the ori provides the stem, “has anyone ever undesirably and obsessively pursued you by:” followed by possible stalking behaviors such as, “threatening to hurt him-or-herself,” “verbally threatening you personally,” “showing up at places in threatening ways or physically hurting you.” the modifications for this study simply involved changing the wording to reflect faculty and students rather than intimate relationships. cupach and spitzberg (1998) report that the ori has been shown to have satisfactory reliability and validity. procedure: all faculty members listed on the university supplied email list were asked to read an informed consent statement and, if choosing to participate, to click on a link to an online survey. the online survey included the demographic questionnaire and ori. the instructions to morgan, r. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 102 faculty members specified that behaviors on the ori should only be reported if a student “undesirably and obsessively pursued” the faculty member and only if the faculty member felt “intimidated, anxious, or fearful” as a result of the student’s behavior. once the faculty member had completed the questions, the faculty member was asked to hit the submit button. all data were compiled and sent to the researcher with no identifying email addresses or names. completing the demographic questionnaire and the ori required approximately 10-15 minutes. b. results. research question 1: incidence of student stalking of faculty: to determine the percentage of faculty reporting stalking, responses to the ori were evaluated. to be considered as experiencing stalking, a faculty member needed to report repeated incidents where a student engaged in at least two separate behaviors on the ori. out of the 28 different stalking behaviors on the ori, the range of behaviors reported by faculty who were classified as having experienced stalking ranged from 3 to 25 with a mean of 6.29 (sd=4.29). in addition, the instructions on the ori specified that behaviors must lead to feeling intimidated, anxious, or fearful. using this definition of stalking, 32.97% (n = 308) of faculty members who completed the survey could be classified as having experienced stalking by a student. the average age of those reporting being stalked (48.98 years) did not differ from the overall sample mean age of 49 years. research question 1a: incidence of student stalking of faculty on smaller commuter campuses vs. larger, residential campuses: the percentage of faculty members who reported being stalked by a student varied by campus, ranging from 22% to 57%, and these differences were statistically significant, 2 (7) = 45.86, p<0.001. overall, more stalking was reported on the six smaller campuses (campuses 3-8) than on the two larger campuses (campuses 1 and 2). another distinction was found with respect to residential vs. commuter campuses. on campus 1, a traditional residential campus, 31% of faculty respondents reported having been stalked. however, on the six primarily commuter campuses (campuses 3-8), the average percentage of faculty reporting stalking incidents was 44%. twenty-two percent of faculty on campus 2 reported stalking incidents. this campus was importantly different from all others, in that it had the lowest response rate of any campus (23%), and, residing in a large city, has a mix of both residential and commuter students. research question 1b: incidence of student stalking of faculty in smaller vs. larger undergraduate classes: no significant difference was found in stalking between faculty members who taught in large or small undergraduate classes. research question 1c: incidence of student stalking of female vs. male faculty: of those being stalked, 54% were female and 46% were male, a significant difference ( 2 (1) = 11.37, p. <0.001). research question 2: types of stalking behaviors experienced by faculty: although not all faculty experienced all 28 behaviors identified on the ori, each of the 28 behaviors was identified as having occurred at least once. among those faculty who were identified as having been stalked, the most common behaviors reported were students invading the faculty member’s personal space and students intruding uninvited into the faculty member’s interactions with colleagues and students by 61% and 57% of faculty respectively (for all percentages, decimals points were eliminated by rounding). the least common behaviors, reported by only 2% of the faculty, were leaving or sending the faculty member a threatening object and kidnapping or morgan, r. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 103 physically constraining the faculty member. table 1 presents the percent of stalked faculty reporting specific stalking behaviors from the ori. as can be seen in this table, 12 of the 28 stalking behaviors listed on the ori were reported by approximately 25% or more of the faculty who had been stalked. table 1. percent of stalked faculty reporting on the ori-short form (n=308). stalking behavior from ori % of faculty reporting behavior invading faculty member’s personal space 61% intruding uninvited into faculty member’s interactions 57% leaving unwanted gifts 47% leaving unwanted messages of affection 46% leaving unwanted threatening messages involving faculty member in activities in unwanted ways 36% 36% making exaggerated expressions of affection verbal threats 32% 32% following the faculty member around, watching the faculty member 28 % 28% engaging in regulatory harassment 27% student threatening to hurt him or herself 25% showing up in places in threatening ways intruding on family/coworkers 17% 17% invading faculty member’s personal property 16% approaching faculty member in public places 14% monitoring faculty member’s behavior 11% obtaining private information student threatening to hurt others 10% 10% stealing/damaging valued possessions sexual coercion physical threats invading faculty member’s property physically hurting 8% 8% 8% 8% 8% physically restraining faculty member physically endangering the faculty member’s life leaving or sending faculty member threatening objects kidnapping or physically restraining faculty member 7% 7% 2% 2% v. phase two. a. method. participants: for this phase of the study, participants were recruited through self-selection. after completion of the online survey and demographic questionnaire used in phase one, faculty members who were willing to be interviewed about their experiences were asked to email the author. fifty-five faculty members volunteered to be interviewed. of these 55, 52 were interviewed. of the three who were not interviewed, one decided not to be interviewed due to scheduling conflicts and the other two withdrew because they reported that it would be too painful to talk about their experiences. similar to faculty in phase one, faculty in phase two taught both undergraduate and graduate students; 17.3% of faculty taught only graduate or morgan, r. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 104 medical/dental classes, 55.8% taught only undergraduate classes, and 26.9% taught both undergraduate and graduate classes. approximately twenty percent (20.38%) of the faculty reported teaching large courses (over 100 students). the age range of full-time faculty members in this phase was 32 to 64 with a median age of 49, mean age of 48.27 years (sd = 6.56). of those participating in this interview phase, 55.77% were women and 44.23% were men. time spent teaching in the university system ranged from 2 to 35 years with a mean of 17.92 (sd = 6.53) years teaching. of those interviewed 19.23% had a rank of assistant professor, 48.08% had a rank of associate professor, 3.85% had a rank of clinical professor, and 28.85% had a rank of professor. no one discipline predominated, with faculty representing almost every school. in comparison to the faculty who participated in phase one, the participants in phase two were approximately the same age with a mean age of 48.27 as compared to a mean age of 49 for those in phase one. likewise, approximately 20% of the faculty in phase one and in phase two reported teaching large courses (over 100 students). two significant differences occurred between participants in phase one and phase two. in phase one, 55% of respondents were men while in phase two, 45% of respondents were men. in addition, the mean time spent teaching in the university was 13 years in phase one but 17.92 years in phase two. materials: no additional materials were used in this phase of the study. the basic questions used to initiate discussion during the interviews may be found in the appendix. procedures: interviews of each professor were conducted on the campus of the faculty member being interviewed at the location of their choice. each interview began by explaining the study and gathering demographic information. following this, the faculty member being interviewed was asked to describe their stalker(s), the incident(s), and their reactions in their own words. follow-up questions were asked to clarify the incident(s) and the reactions of the faculty member. following this open-ended account, the interviewer asked the faculty member to describe the impact of the stalking on their subsequent teaching and interaction with students. the author conducted all interviews and data were recorded by hand. tape recorders were not used to preserve confidentiality. b. results. research question 2: types of stalking behaviors experienced by faculty the stalkers: the number of stalkers reported by each faculty member varied from one to six with a mean of 1.67 (sd = 0.92) and a total of 87 stalkers among the 52 faculty members interviewed. of the stalkers reported, 51.72% were men and 48.27% were women. sixty-seven percent of stalkers were undergraduates, 31.03% were graduate students, and 2.3% were medical students. the age range of stalkers was reported by faculty as 19 to 45 with a mean age of 25.16 (sd=5.87). faculty were not always sure of the student stalkers’ ages so these are their best estimates. behaviors reported by faculty: the 52 faculty members interviewed reported on the stalking behaviors they experienced in the 87 incidents described in the interviews. faculty reported a wide range of problematic behaviors with a mean of 8 behaviors per incident and a total of 696 behaviors. using coding procedures similar to those outlined in other studies utilizing interviewing and open ended questioning (dupuis, bloom and loughead, 2006; holmes, 2005), all 696 of these behaviors were organized into categories by the author and two research students based on similar characteristics; agreement on categories was 100%. for example, being sent numerous email messages, letters, or being called repeatedly on the morgan, r. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 105 telephone was placed into one category, unwanted messages. as can be seen in table 2, the most commonly reported behaviors were unwanted messages, with 92% of faculty who had been stalked reporting receiving repeated unwanted messages, following the faculty member around campus and off-campus and obsessively watching the faculty member (73%), and regulatory harassment, that is, threatening to or reporting the faculty member to their superior (61%). fiftysix percent of faculty reported being verbally or physically threatened and 32% reported sexually coercive behavior on the part of the student stalker, including attempts at kissing the faculty member, requests to engage in sexual activity, and unwanted caresses. seventeen percent of faculty reported incidents in which their life had been endangered, in each case where a student either attempted to kill or actually killed another faculty member being stalked, and 46% of faculty reported incidents in which the student threatened to harm him or herself. table 2. stalking behaviors reported by interviewed faculty (n=52). stalking behaviors % experiencing behavior unwanted messages 92.31% (n=48) following around, watching 73.07% (n = 38) regulatory harassment 61.54% (n = 32) verbally or physically threatening the faculty member, showing up 55.78% (n = 29) spreading misinformation about the faculty member, exaggerated expressions of affection 53.85% (n = 28) obtaining private information 51.92% (n = 27) student threatening his or herself, monitoring behavior, intruding uninvited 46.15% (n = 24) sexually coercing, intruding upon family and coworkers 32.69% (n = 17) invading personal space, unwanted gifts 30.77% (n = 16) physical restraint 21.15% (n = 11) damaging or stealing possessions, involving in activities, endangering life 17.31% ( n = 9) invading personal property invading property 15.38% (n = 8) 11.54% (n = 6) research question 2b: what is the most common category of stalking in this population? previous literature has reported that stalking may be subdivided into various types. as indicated earlier, a common categorization scheme for stalking has been suggested by dziegielwski and roberts (1995), who outlined three subtypes: the domestic violence stalker, the erotomanic/delusional stalker, and the nuisance stalker. the 87 reported stalking incidents were reviewed by the author and two research students to determine if they fell into one of these three subtypes. stalking was labeled as domestic violence if the stalking incident was viewed as resulting from a need to establish, continue, or re-establish a previous relationship even if the prior social contact was misperceived by the student. for example, one incident of stalking in this category occurred following attending an off-campus conference with a faculty member. stalking was labeled as erotomanic/delusional if the stalking was viewed as a result of becoming fixated on a faculty member with whom no prior outside-of-class social contact had occurred. for example, one incident of stalking in this category involved the student interpreting faculty gestures in the class as conveying personal, sexual messages to the student. finally, the stalking was labeled as nuisance if there was repeated harassment of a faculty member without the intent of building a relationship with that faculty member. students falling into the last category typically were perceived as trying to manipulate the faculty member into changing course morgan, r. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 106 requirements or changing their grade. this category might be most similar to hall’s (1998) category of stalkers seeking revenge. there was 100% agreement between the three raters on placing each of the 87 stalkers into the three categories. fourteen percent (n = 12) of the stalking incidents were classified as falling into the domestic violence or prior social contact category, 42.53% (n = 37) fell into the erotomanic/delusional category, and 43.68% (n = 38) fell into the nuisance/manipulative category. research question 3: impact of student stalking on teaching responses reported by faculty: faculty reactions were organized into categories based on similar response characteristics by the author and the two research students. for example, if faculty reported that they told their dean or their assistant dean, this response was categorized under contacted administrator. the responses reported by faculty could be organized into two categories: general behavioral responses, and emotional responses. in terms of general behavioral responses, 67% of faculty had told their colleagues, 58% had told an administrator, and 38% had contacted city or university police; however, 27% had told no one about being stalked. characteristic comments from faculty included: “administrators only interested in solving problems, can’t count on them to help,” “told a colleague about this latest incident and colleague blew it off … not very supportive … made me mad,” “student would have to injure me to be taken off campus… someone has to get hurt before something is done,” and “how has academia allowed these things to happen?” those faculty who had told no one about the stalking typically continued to work with the student, reporting, “teacher in me feels i can’t ignore students – feel an obligation to help students.” forty percent of faculty reported directly telling the student stalker to stop. twenty-seven percent of faculty members reported that they had been accused by a colleague, an administrator, or the student stalker of having a sexual relationship with the student stalker. one faculty member, echoing the comments of several others, stated, “there is a tendency to immediately take the student’s side over the professor … very unfair … professor has no rights in this process,” and “i did not feel i had any rights, the students have all the rights – nothing i know of in system to protect the faculty member unless the student does something criminal and you can prove it.” emotional responses were a large category of responses reported by the interviewed faculty. as can be seen in table 3, the most commonly reported faculty responses included emotional reactions with 52% of faculty reporting embarrassment, 42% reporting helplessness, and almost 37% reporting feeling responsible for the student’s behavior. common responses of faculty included: “we’re supposed to be accessible to our students. where do faculty member boundaries come in? we don’t have a life outside their (the student’s) world,” “dose of reality, you lose your sense of trust – can never be fully replaced – can be somewhat mended but there’s a scar – always in the back of your mind – could this be a situation?” and “made me question what i was doing to promote this – what would make them think they could do this to me?” impact on teaching and future interactions with students reported by faculty: responses to this question were also categorized based on similar reactions by the author and two research students. for example, if faculty reported that they had stopped using personal examples while teaching or no longer shared personal stories with students, this was placed in the ‘less personal when teaching’ category. relative to the current incident of stalking, 40% of faculty reported giving a higher grade to the student stalker than what the student earned. for example, one faculty member reported, “just easier to give them all b’s than to have to deal with morgan, r. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 107 table 3. responses of interviewed faculty members (n = 52) to stalking incidents. response of faculty member % reporting general behavioral reactions of faculty: told colleagues 67.31% felt supported by colleagues 31.25% contacted administrator 57.69% felt supported by administrator 16.67% directly told student to stop 40.38% accused of sexual contact with student by colleagues, administrator, or student 26.92% contacted city or university police 38.46% told no one 26.92% considered leaving university 23.07% caller id installed 21.15% changed home telephone number 21.15% coordinated with colleagues 17.30% panic button installed in office 17.30% divorced 17.30% consulted union 17.30% obtained a personal protection order 17.30% faculty member moved to a safer community 9.61% emotional reactions of faculty: faculty member embarrassed 51.92% faculty member felt helpless 42.31% faculty member felt responsible 36.54% stress related health problems 32.69% contacted psychological clinic 23.08% emotional reaction to student committing or attempting to commit suicide 17.31% them.” another commented, “give all multiple-choice objective test items now so students can’t argue lack of objectivity.” twenty-three percent of faculty reported that the student was banned from campus email, almost 10% of faculty reported that the student stalker was banned from campus, and almost 6% of faculty reported that the student was jailed as a result of the stalking incidents. almost 83% of faculty reported that they were less personal when teaching subsequent to the stalking incident. fifty-four percent reported reduced spontaneity when teaching and 56% reported that they now keep their office door open when meeting with students. additional reports from faculty about how the stalking impacted their teaching are listed in table 4. in general, the responses of the faculty members reflect their efforts to reduce any but the most necessary contact with students. for example, one faculty member commented, “every year (i) get more and more distant; very cautious – treat them with kid gloves”. morgan, r. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 108 table 4. impact on teaching reported by interviewed faculty members (n=52). impact on teaching % reporting change reactions of faculty to current stalking: gave higher grade to student than deserved 40.38% refused to talk with student or answer emails 36.54% avoids office 34.62% professor tried to be nicer to student 32.69% attempted to cajole student 30.77% banned student from email 23.08% banned student from campus 9.62% pursued legal action leading to student being jailed 5.77% made changes to syllabus 98.08% removed phone number/contact information 88.46% reduced difficulty level/number of assignments 53.85% increased structure/rules on syllabus 40.38% substituted objective grading (multiple-choice) for subjective (essays, papers) 30.77% less personal when teaching 82.69% keeps office door open when meeting with students 55.77% reduced spontaneity in classroom 53.85% more professional, no longer meets with students outside of office 50.00% changed or cancelled office hours 40.38% second guesses self 30.77% only responds to certain emails 26.92% vi. discussion. stalking was operationally defined in the present study when a faculty member reported repeated incidents of at least two separate stalking behaviors by a student that led to the faculty member feeling fear, threatened, or intimidated. in the present study, an average incidence rate of 33% was found. this rate is concerning given the national rate of stalking among the general population is about 6-7%. however, the rate seems more consistent with the rate of 24% among mental health professionals. although female faculty members were slightly more likely than male faculty to report stalking, the discrepancy is not as large as found in the general population (e.g., basile, swahn, chen, and saltzman, 2006). likewise, the most common types of faculty stalking differ in significant ways from the most common types of stalking found in the general population. among the faculty interviewed in the present study, it was clear that three distinct patterns of stalking were occurring, similar in some ways to the three types proposed by dziegielwski and roberts (1995). to recall, dziegielwski and roberts outlined the domestic violence stalker, the erotomanic/delusional stalker, and the nuisance stalker. in the present study, faculty reported what can be called a prior social contact based stalking where the student perceives an intimate relationship with the faculty member and begins the stalking as a way of increasing intimacy or of punishing the faculty member for refusing deeper intimacy. this might be similar to the domestic violence stalking described by dziegielwski and roberts. however, in the case of students stalking faculty, there has rarely been any intimate relationship. in the present study, for example, only one faculty morgan, r. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 109 member admitted a previous romantic relationship with the student who later stalked her. in addition, most studies in the general population report that this type of stalking is the most common, accounting for up to 80% of all stalking. in the present study, this type of stalking accounted for only 14% of the stalking. although there was only one instance of a faculty member reporting a prior romantic relationship with a student, there were ten additional incidences of student stalking following what might be perceived as blurred boundaries with faculty members. for example, one instance of stalking occurred following a pattern of interactions with the student that included going to the professor’s house (working in the professor’s bedroom) and attending conferences with the professor. in a second incident, the student had met with the professor off-campus at restaurants and in bars. a second type of stalking occurs in students who appear, to faculty members, to be experiencing significant psychological difficulties. these students may have had little to no contact with the faculty member but have fixated on this faculty member in a delusional manner. this is what dziegielwski and roberts call the erotomanic/delusional stalker. in the general population, this type of stalking is considered relatively rare. in the present study, this type of stalking accounted for 42.53% of the stalking. in many ways, this higher figure parallels the findings of mental health professionals who are stalked by their clients. in those studies, the clients who were most likely to stalk the mental health professional suffered from personality disorders, had experienced drug and alcohol problems, had a history of behavioral problems, had quickly attached to the therapist, or were sexually attracted to their therapists. this seems very similar to the descriptions of the student stalkers in the present study. the fact that this percentage is so high is of concern as the majority of college faculty are not trained as mental health professionals. college campuses are filled with students at a very vulnerable stage of development (fromme, corbin, and kruse, 2008). these students may be away from home for the first time, may be trying to attend school and work full-time, and on many commuter campuses, may be attending school full-time, working full-time, and caring for children. the pressure for all students to attain a college degree has become more intense. it may be that students who attend commuter campuses are even more stressed than those on residential campuses, increasing the dangers to faculty who teach on these campuses. the third distinct category, that may be specific to faculty, is where students stalk faculty in a seeming effort to influence the grade the faculty member assigns. although this may be considered a subtype of nuisance stalker, it seems evident that this subtype is specific to this environment and is relatively common, with 43.68% of the incidents in the present study falling into this category. given the pressures that students are under to achieve high grades and an increasing consumer attitude that if they pay for their education, they are entitled to a good grade (ritter, 2008), these attempts at manipulation of faculty are not surprising. faculty responses to incidents of student stalking included both teaching-related responses and more personal responses. although personal responses were not a direct focus of this study, some reactions are noteworthy. interestingly, most faculty members (84%) attempted to balance what they perceived as the best interests of their students with the stress being caused by the student. for example, although one professor was experiencing physical symptoms of stress, crying, intense fear, and embarrassment, this professor continued advising the student stalker as the professor believed that no other faculty member would be as familiar with the student and the student’s issues. this was not always the case; several faculty members reported that they simply refused to teach students sharing characteristics of their stalker (for example, undergraduate morgan, r. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 110 status or a major in, say, psychology), even though they recognized that this was not in the best interests of students. one of the primary concerns of the present study was to assess the impact of student stalking on faculty teaching and interaction with students. faculty have been encouraged to increase active learning approaches in the classroom and to engage with students outside the classroom in order to increase learning. social constructivism—an instructional philosophy most closely associated with lev vygotsky—argues that learning is culturally and socially influenced; that is, the community around us affects the way we see the world. the classroom, from this social constructivist perspective, must be "active, constructive, intentional, and cooperative" (jonassen, howland, moore, and marra, 2003, p. iv), with instructors creating learning experiences that students find challenging and personally meaningful (llewellyn, 2002). within this framework, the classroom is much like a community, with students participating in both individual and group roles and seeking support and encouragement from their instructor as well as their peers. in order for such a classroom to succeed, it seems evident that the professor must be able to feel safe in the community of students. unfortunately for students, however, every professor interviewed directly identified changes in their willingness to interact with students. these changes varied from changes in the syllabus – more rules, more structure, and fewer opportunities for students to interact with the faculty member outside of the classroom – to increasing the difficulty level for students in contacting the professor by no longer providing a home telephone number, to preventing student access to the faculty member’s office without supervision. faculty members who were able to do so reported decreasing their exposure to students by refusing to teach particular classes or even particular populations of students (students from a particular major or students at a particular academic level, such as undergraduates). these changes may decrease student choices in terms of courses and may decrease the community cohesiveness in the classroom. not all of these changes are necessarily negative. for example, several professors reported no longer providing students with their home telephone number. although students might enjoy contacting professors at home, there is no pedagogical reason to believe that this is necessary for student learning. likewise, three professors noted that the stalking incident led them to reevaluate their course assignments, leading them to improve the clarity of their requirements. such a change would benefit all students. as a result of their experiences with stalking, many faculty reported a change in how they viewed colleagues, administrators, and the general climate of the university. almost every faculty member attempted at some point to communicate their concerns about the student stalking to a colleague (67%) or an administrator (58%). in 62% of these cases, the faculty member reported a lack of concern, disbelief, or recommendations to ‘let it go and not cause any trouble.’ both male and female faculty reported that frequently other colleagues viewed the professor as the instigator of problems or that the professor should be flattered by the student’s interest. in these cases, the stalker was perceived by other faculty and, at times, by administrators, as having a ‘crush’ on the faculty member. since students are almost always of legal age, the perception of many was that as long as the student was no longer in the professor’s class, there was no problem with the professor and student dating. conceptualizing stalking behaviors as simply dating attempts was frustrating and humiliating for the professor involved. in two cases, the student stalker sought out colleagues of the professor to try and help convince the faculty member to date the student, and the colleagues did so. morgan, r. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 111 the results of this study indicate a potentially critical problem on university campuses. that is, stalking appears much more common than in the general population and highly likely to lead to changes in the professors’ treatment of students. faculty have a tendency to withdraw from students following stalking incidents, decreasing the very warmth so many researchers have argued is necessary for better classroom learning environments. equally important is the significant distrust engendered by such incidents between the faculty members and the academic community. almost two-thirds of faculty report negative experiences when attempting to report stalking incidents to their immediate superiors – deans or assistant deans – or to their colleagues. faculty do not feel supported when these incidents occur, reporting that the students are more likely to be believed than they are. efforts to address these issues will require significant campus efforts to educate faculty and administrators about how to handle potential stalking situations and the development of protocols to allow faculty concerns to be heard openly without fear of repercussions. although only a few faculty members reported feeling supported by colleagues and administrators, those individuals were unlikely to have made potentially negative changes in their teaching or interactions with students. vii. limitations and future directions. several limitations of this study caution against overgeneralization. although eight campuses were surveyed, all campuses were within one university system. it is possible that the findings within this system would not be representative of other universities. obviously, the next step would be to survey a larger set of faculty from a variety of campuses across the united states. likewise, the present study used the ori – short form. this survey instrument was designed for use in a more general population and with more traditional stalking situations, that is, in situations where there had been a previous relationship between the stalker and the person being stalked. there is no guarantee that this scale reliably and validly measures stalking in the present population. future studies should modify this form to reflect the findings of this study. specifically, the form needs to reflect situations in which no prior personal relationship existed between the professor and the student as well as asking more questions about how the faculty responded to the stalking incidents. although there was considerable overlap between the stalking behaviors identified on the ori and during the interview, the category rankings were not identical. ninety-two percent of faculty who were interviewed described a pattern of unwanted messages including email, telephone, and traditional notes/letters from students. the ori did not have a category that specifically meshed with this. on the ori, unwanted messages were categorized into either unwanted messages of affection or unwanted threatening messages. at times, the types of unwanted messages described by faculty were rambling diatribes reflecting confusion, hatred, or incoherence and were not perceived by faculty as falling into either of the two ori categories. likewise, in light of the research conducted by turmanis and brown (2006), indicating that rates of stalking vary when assessing level of concern, it seems critical that this be assessed as well as simply stalking behaviors experienced. in the present study, although all faculty members reported the unwanted behaviors as threatening or intimidating, thereby fulfilling the definition of stalking, the level of concern about the stalking was not specifically addressed. in addition, since not all faculty chose to complete the survey, it is impossible to know how the characteristics of those choosing to participate differed from those who did not choose to complete the survey. it may be that those who had experienced stalking incidents were more morgan, r. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 112 likely than those who had not experienced such incidents to complete the survey. likewise, female faculty who had experienced stalking incidents may have been more likely to respond to the survey than male faculty members. on the other hand, it seemed clear from the interviews with faculty who had been stalked, that they were reluctant to share their experiences with others. it may be that faculty experiencing stalking incidents are less likely to categorize their experiences as stalking and thus, were less likely to respond to the survey. in either of these scenarios, the actual incidence of student stalking of faculty may be misrepresented by the present study. despite the above limitations, the present study clearly illustrates the need for additional research in this area. the types of behaviors experienced by faculty who are being stalked by students identified in this study overlap with the general findings in the stalking literature but present some unique challenges in that such stalking incidents are less likely to be the result of prior, mutual emotional relationships. this suggests that student stalking of faculty may be more similar to clients stalking mental health professionals. these differences need to be added to future surveys of faculty stalked by students. the detailed interviews with the faculty in this study clearly suggest the significant emotional impact of student stalking and the toll it takes on the future interactions between faculty and students both within and outside of the classroom. finally, the interview data suggest that college campuses need to develop a more supportive environment for those faculty members who may be experiencing student stalking. more specifically, the development of campus wide training for faculty and administrators on the potential problem of student stalking of faculty and appropriate measures would be of use. such training needs to be coordinated with campus security and the campus improvement of teaching center, if available. the inclusion of administrators such as deans, assistant deans, and program chairs ideally should allow for a greater sense of support. clear guidelines for faculty to document such incidents should also be developed. in addition, faculty need to discuss the advantages and disadvantages of meeting with students off-campus or in more isolated parts of campus. how stalking impacts teaching and the campus community also needs to be directly addressed. as stated by one of the interviewed faculty members, “everyone knew this was going on…no one was stopping her…my colleagues were just glad it was happening to me and not to them.” acknowledgements this research was supported in part by a p.a. mack fellowship. i would like to thank my research students tara white, nolan brewer, and erin emington for their assistance with this study. appendix 1. standard questions used to initiate discussion in interviews. for each stalking incident: • tell me a little about yourself: age, gender, race, marital status, length of time at university, rank, discipline, etc. • tell me a little about the stalker: age, gender, race, level of program, etc. • when did these events take place? • describe the behaviors of the student – follow-up questions as needed. • did you tell anyone while this was occurring? who? what was their reaction? o if not, why did you tell no one? • what impact did this have on your teaching? morgan, r. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 113 o do you have any indications that this impacted your teaching? o do you believe this was a positive or negative change in your teaching? why? • did you change anything on your syllabi as a result of this incident? o get specific information on changes • did you change any of your classroom procedures as a result of this incident? o get specific information on changes • how do you believe this incident impacted your students? o get specific information o can i use this in any way? references anaya, g. (1999). college impact on student learning: comparing the use of self reported gains, standardized test scores, and college grades. research in higher education, 40(5), 499526. astin, a. (1994). what matters in college: four critical years revisited. san francisco: jossey-bass. basile, k.c., swahn, m.h., chen, j., and saltzman, l.e. (2006). stalking in the united states: recent national prevalence estimates. american journal of preventative medicine, 31(2), 172175. batty, p. (2004). female staff face abuse. times higher education supplement, 1639, 1. biaggio, m., paget, t.l., and chenoweth, m.s. (1997). a model for ethical management of faculty: student dual relationships. professional psychology: research and practice, 28(2), 184-189. bjerregaard, b. (2002). an empirical study of stalking victimization. violence and victims, 15, 389-406. bloom, r. (2000). have you ever been heckled in the classroom? accounting education, 9(4), 413-415. collins, s. (august 26, 2006). ex-student charged in arson of professor’s home. retrieved july 9, 2007, from http://wjz.com/local/local_story_236211205.html. chronicle of higher education (august 25, 2006). disgruntled ex-student charged with trying to kill professor. retrieved july 24, 2008 from http://chronicle.com/news/article/897/disgruntledex-student-charged-with-trying-to-kill-professor. cnn.com (april 18, 2007). retrieved may 29, 2007, from http://www.cnn.com/2007/us/04/17/vtech.shooting/index.html. morgan, r. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 114 cupach, w.r., and spitzberg, b.h. (1998). obsessional relational intrusion and stalking. in w.r. cupach and b.h. spitzberg (eds.), the dark side of close relationships (p. 112-137). new york: springer publishing company. dupuis, m., bloom, g., and loughead, t. (2006). team captains' perceptions of athletic leadership. journal of sport behavior, 29, 60-78. dziegielwski, s.f., and roberts, a.r. (1995). stalking victims and survivors: identification, legal remedies, and crisis treatment. in crisis intervention and time –limited cognitive treatment, a.r. roberts, ed., pp. 73-90. newbury park, ca: sage publications. feldman-summers, s. (1989). sexual contact in fiduciary relationships. in g. o. gabbard (ed.), sexual exploitation in professional relationships. washington, dc: american psychiatric association. fisher, b.s., cullen, f.t., and turner, m.g. (2002). being pursued: stalking victimization in a national study of college women. criminology and public policy, 1, 257-308. fromme, k., corbin, w.r., and kruse, m.i. (2008). behavioral risks during the transition from high school to college. developmental psychology, 44(5), 1497-1504. hall, d. (1998). the victims of stalking. in j. reid meloy (ed.). the psychology of stalking: clinical and forensic implications. p. 113-117, san diego, ca: academic press. holmes, d.l., rupert, p.a., ross, s.a., and shapera, w.e. (1999). student perceptions of dual relationships between faculty and students. ethics & behavior, 9(2), 79-107. holmes, r.m. (2005). working to play: college student athletes' conceptions of play and work. in f. mcmahon, d. lytle and b. sutton-smith (eds.), play: an interdisciplinary synthesis: play & culture studies, vol. 6 (pp. 209-231). landham, md: university press of america. hudson-allez, g. (2006). the stalking of psychotherapists by current or former clients: beware of the insecurely attached! psychodynamic practice, 12(3), 249-260. jonassen. d.h., howland. j., moore, j., and marra, r.m. (2003). learning to solve problems with technology: a constructivist perspective. 2nd ed. upper saddle river, nj: merrill prentice hall. kitchener, k. s. (1992). psychologist as teacher and mentor: affirming ethical values throughout the curriculum. professional psychology: research and practice, 23, 190–195. lenckus, d. (2002). college shooting underscores risk. business insurance, 36(44), 3-4. llewellyn. d. (2002). inquire within: implementing inquiry-based science standards. thousand oaks, ca: corwin press. morgan, r. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 115 marks, c.a. (1997). the kansas stalking law: a “credible threat” to victims. washburn law journal, 36, 468-498. mcivor, r.j., and petch, e. (2006). stalking of mental health professional: an underrecognized problem. british journal of psychiatry: the journal of mental science, 188, 403-404. mullen, p.e. (2003). multiple classifications of stalkers and stalking behaviors available to clinicians. psychiatric annals, 33(10), 651-656. mullen, p.e., pathé, m., and purcell, r. (2000). stalkers and their victims. cambridge, england: cambridge university press. new york times (july 19, 1997). man gets three life terms for killing three professors. retrieved july 18, 2007, from http://query.nytimes.com/gst/fullpage.html?res=9c00e4dd173bf93aa25754c0a961958260& n=top%2freference%2ftimes%20topics%2fsubjects%2fp%2fplea%20bargaining. pathé, m., and mullen, p.e. (1997). the impact of stalkers on their victims. british journal of psychiatry, 170, 12-17. purcell, r., pathé, m., and mullen, p.e. (2004). when do repeated intrusions become stalking? the journal of forensic psychiatry and psychology, 15(4), 571-583. ratelle, l. (1994). reports on 1992 campus murders bring shakeup to concordia university. chronicle of higher education, 40(41), a37. ritter, k. (2008). e-valuating learning: rate my professors and public rhetorics of pedagogy. rhetoric review, 27(3), 259-280. romeo, f.f. (2001). a recommendation for a campus anti-stalking policy and procedures handbook. college student journal, 35(4), 514-516. schneider, a., and basinger, j. (1998). faculty notes. chronicle of higher education, 45(10), a14. spitzberg, b.h., and cupach, w.r. (2003). what mad pursuit? obsessive relational intrusion and stalking related phenomena. aggression & violent behavior, 8(4), 345-376. tjaden, p., and thoennes, n. (2000). report of the prevalence, incidence and consequences of violence against women: findings from the national violence against women survey (ncj 183781). washington, dc: u.s. department of justice, national institute of justice. turmanis, s.a., and brown, r.i. (2006), the stalking and harassment behavior scale: measuring the incidence, nature, and severity of stalking and relational harassment and their psychological effects. psychology and psychotherapy: research and practice, 79, 183-198. morgan, r. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 116 wilson, r. (2007). a right to romance. chronicle of higher education, 53(50), 6. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 117 mission founded in 2001, the journal of the scholarship of teaching and learning (josotl) is a forum for the dissemination of the scholarship of teaching and learning in higher education for the community of teacher-scholars. our peer reviewed journal promotes sotl investigations that are theory-based and supported by evidence. josotl’s objective is to publish articles that promote effective practices in teaching and learning and add to the knowledge base. the themes of the journal reflect the breadth of interest in the pedagogy forum. the themes of articles include: 1. data-driven studies: formal research projects with appropriate statistical analysis, formal hypotheses and their testing, etc. these studies are either with a quantitative or qualitative emphasis and authors should indicate the appropriate domain. acceptable articles establish a research rigor that leads to significant new understanding in pedagogy. 2. reflective essays: integrative evaluations of other work, essays that challenge current practice and encourage experimentation, novel conclusions or perspectives derived from prior work 3. reviews: literature reviews illuminating new relationships and understanding, metaanalysis, analytical and integrated reviews, etc. 4. case studies: these studies illustrate sotl and its applications, usually generalizable to a wide and multidisciplinary audience. 5. comments and communications: primarily, these are comments based on previously published josotl articles, but can also include book reviews, critiques and evaluations of other published results in new contexts or dimensions journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 118 submissions authors are encouraged to submit work in one of the following categories: • traditional research reports: data driven studies with either a quantitative or qualitative emphasis • reflective essays on sotl • reviews of current themes in sotl research including meta-analysis • case studies illustrating sotl and its applications • comments and communications on previous journal articles, or book or software reviews in your e-mail with your submission, please indicate which of the above categories most applies to your submission. despite their differences, all of these types of submissions should include the author’s expression of the implications their work has for the teaching-learning process. this reflective critique is central to our mission in furthering understanding of sotl. authors are encouraged to review the guidelines for reviewers in order to understand how their submissions will be evaluated. authors are strongly encouraged to study the reviewer’s rubric that reviewers shall apply in evaluating their submitted work. authors should submit their article to josotl@iupui.edu. submissions must be prepared in an electronic format using microsoft word on either pc or macintosh platforms. submissions should be uncompressed files attached to an e-mail, not in the body of an e-mail text. all submissions must be prepared following the guidelines below. while there is no formal page limit, authors should adhere to recent article lengths, typically 20 pages or less. authors are expected to include proper referencing for their sources, especially urls for web sites that might contain material of interest to our readership. every submission must include a cover page preceding the article with the following information: title of article for each author: name and affiliation postal address e-mail address telephone number abstract (less than 100 words) keyword list related to the submission (less than eight words or short phrases) this cover page should be followed by the article formatted according to the josotl style sheet (available in either .doc or .pdf format). accepted work authors will be required to sign a copyright agreement with the trustees of indiana university. authors must be prepared to sign this agreement upon acceptance of their work and prior to journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 119 publication in the journal. for more information regarding copyright, please see the statement of copyright and terms of use. if you have any questions regarding the submission policy, please e-mail kimberly olivares (josotl production coordinator) or call her at 317-274-0086, or contact a member of the editorial board. journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 120 editorial board all members of the josotl editorial board are affiliated with facet, the faculty colloquium on excellence in teaching, at indiana university. don coffin associate professor of economics dcoffin@iun.edu division of business and economics indiana university northwest 219.980.6913 david j. malik chancellor's professor of chemistry facet director executive vice chancellor of academic affairs dmalik@iupui.edu or dmalik@iun.edu http://chem.iupui.edu/faculty/malik.html school of science and facet indiana university purdue university indianapolis 317.278.3314 administration iu northwest 219.980.6707 eugenia fernandez associate chair of computer & information technology efernand@iupui.edu http://www.engr.iupui.edu/~efernand purdue school of engineering and technology indiana university purdue university indianapolis 317.274.6794 joan e. lafuze professor of biology jlafuze@indiana.edu instructional programs indiana university east, richmond, in 765.973.8246 julie saam assistant dean for program review and granduate studies associate professor of secondary science education jsaam@iuk.edu division of education indiana university kokomo, in 765.455.9302 ellen a. sigler professor and department head elsigler@wcu.edu educational leadership western carolina university 828.227.7415 carol hostetter director, the mack center for inquirty on teachingand learning associate professor of social work chostett@indiana.edu school of social work indiana university bloomington 812.855.4427 
 journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009, pp. 121 – 123. style sheet for the journal of the scholarship of teaching and learning john dewey1 and marie curie2 abstract: this paper provides the style sheet for the journal of the scholarship of teaching and learning. manuscripts submitted for publication should adhere to these guidelines. keywords: radiation, metacognition, identity theory, constructivism, educational philosophy. i. general guidelines for the manuscript. the final manuscript should be prepared in 12-point, times new roman, and single-spaced. submissions should be double-spaced. all margins should be 1 inch. the text should be fully leftand right-justified. the title (in 16 point bold) and author’s name (in 12 pt. bold) should be at the top of the first page. the author’s name should be followed by a footnote reference that provides the author’s institutional affiliation and address. the abstract should be indented 0.5" left and right from the margins, and should be in italics. except the first paragraph in a section subsequent paragraphs should have a 0.5" first line indent. use only one space after the period of a sentence (word processors automatically adjust for the additional character spacing between sentences). the keywords should be formatted identically to the abstract with one line space between the abstract and the keywords. keywords currently in use are indexed at the end of each volume. authors should use keywords that are helpful in the description of their articles. common words found in the journal name or their title article are not helpful. pages should be unnumbered since they will be entered by the journal editorial staff. we will also insert a header on the first page of the article, as above. references should be incorporated in the text as authors name and date of publication (coffin, 1993), with a reference section at the end of the manuscript (see below for the desired format for the references). titles of articles should be included in the references in sentence case. unless instructed otherwise in this style sheet, please use apa style formatting. footnotes should incorporate material that is relevant, but not in the main text. ii. section and sub-section headings. a. major sections. major section headings should be flush-left, bold-faced, and roman-numeral numbered. major section headings should have one-line space before and after. the first paragraph(s) of the article do not require a major heading. 1department of educational philosophy, indiana university northwest, 3400 broadway, gary, in 46408, jdewey@iun.edu. 2institut pasteur, university of paris, 75015 paris, france. dewey and curie journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 122 b. sub-sections. sub-section headings should also be flush-left, in italics, and alphabetically numbered. subsection headings should have a one-line space before and after. sub-sub-sections should appear at the beginning of a paragraph (i.e., with an 0.5" indent, followed immediately by the text of the sub-sub-section), with the heading also in italics. iii. tables and figures. tables and figures should be inserted in the text where the author believes they best fit. they may be moved around a little to better correspond to the space requirements of the journal. if necessary, tables and figures may occupy an entire page to ensure readability and may be in either portrait or landscape orientation. insofar as possible, tables should fit onto a single page. all tables and figures should be germane to the paper. tables should be labeled as follows with the title at the beginning (in bold), with data entries single-spaced, and numbered. column labels should be half-line spacing above data. table 1. the title of the table. unit length, inches point 1/12 pica 1/6 figures should have their captions follow the image. captions should be single-spaced, with title in bold. additional text should not be in bold. the editorial staff may adjust layout to allow optimal use of space. figure 1. color wheel with wavelengths indicated in millimicrons. opposite colors are complementary. dewey and curie journal of the scholarship of teaching and learning, vol. 9, no. 2, june 2009. 123 acknowledgements acknowledgements should identify grants or other financial support for this research by agency (source) and number (if appropriate). you may also acknowledge colleagues that have played a significant role in this research. appendix please insert any appendices after the acknowledgments. they should be labeled as follows: appendix 1. the title of the appendix. references coffin, d.a. (1993). using the competitive edge. journal of economic education, 24 (1), 62-69. garcia, j. and rodriguez, p. (2002). the determinants of football match attendance revisited: empirical evidence from the spanish football league. journal of sports economics, 3 (1), 18-38. hamilton, s. j. (1995). my name’s not susie: a life transformed by literacy. portsmouth, nh: boynton/cook publishers. pappas, d. (2004). fixing the fan cost index: a more realistic view. retrieved april 21, 2004, from http://www.baseballprospectus.com/article.php?articleid=2790. v9n2sullenberger v9n2vieira v9n2malone v9n2benander v9n2taylor v9n2glaserbingham v9n2weeden v9n2docan-morgan v9n2morgan v9n2taylor_.pdf v9n2taylor.pdf v9n2taylor.2 v9n2taylor.3 v9n2taylor.4 v9n2taylor.5