Integrating Course 1 1 Running Head: INTEGRATING COURSE AND INSTRUCTIONAL EVALUATION Integrating Course And Instructional Evaluation With A Learning History Approach Alan Clardy Towson University July 30, 2002 Key Words: Educational Evaluation, Learning History, On-line Learning System Integrating Course 2 2 Abstract Student ratings are the traditional approach to course and instructional assessment, yet this procedure has questionable reliability and validity. Teacher portfolios, while an improvement, also have some shortcomings. An on-line “learning history” approach to course and instructional assessment was tested in a graduate course in the fall, 2000. The characteristics of this technique as applied to instructional assessment are reviewed and compared to the student ratings and teacher portfolios approaches, and results of this initial pilot test are presented. A generic model for adapting this approach to any course is provided along with recommendations for further testing. Integrating Course 3 3 Integrating Course And Instructional Evaluation With A Learning History Approach Evaluation – of faculty, instruction and/or learning – is a time-honored component of the educational process. Presumably, evaluation should serve several functions, including assessing instructor performance for merit and other personnel reviews, identifying effective and ineffective instructional practices, gauging student reactions to courses and faculty, and providing a context for judging the quality of student learning, to name a few (Braskamp and Ory, 1994). These often competing goals and multiple uses, though, require different kinds of information, making the task of finding and implementing the most constructive and efficient process for evaluation a pernicious issue. This paper will report on an action learning adaptation of the “learning history” technique to evaluation in higher education. The first portion of this paper will review the use of and problems with the traditional student ratings and the more recent teaching portfolio approach. Then, the learning history technique will be reviewed, followed by a discussion of how this technique was used in a graduate course. Finally, the implication of this approach for practice and future research will be considered. Evaluation in Education Often, in higher education, course evaluation is equated with end-of-semester student ratings of faculty. A 1982 survey of more than 600 liberal arts colleges found that two-thirds (67.5%) always used “systematic student ratings” for evaluating teaching performance, a noticeable increase from the 55% who had reported on such usage five Integrating Course 4 4 years earlier (Seldin, 1984). And in both surveys, student ratings of faculty were the most frequently used method of evaluation. By 1993, Seldin found that student evaluations had risen to 86% of the 600 colleges surveyed. Evidence on the effectiveness and validity of student ratings of faculty, though, has been mixed (Abrami, d’Appollonia and Cohen, 1990, Braskamp and Ory, 1994). In spite of their widespread use, the traditional method of student evaluation can be dogged by three problems: bias, coverage, and focus/contamination. The net effect of these problems raises questions about their suitability for evaluation purposes. Student rating bias is an obvious possibility when adverse treatment (for example, due to grade disappointments or because of disciplinary actions against the student) is joined to a procedure in which anonymity provides a lack of rating accountability. A resulting student animus to the instructor can easily become translated into ratings that may have little to do with the faculty member’s actual performance in the course. In a field experiment with community college students in which respondent anonymity and course grades were manipulated and communicated immediately prior to student evaluations of their instructor, Blunt (1991) found that both factors affected ratings of instructors. Under conditions of anonymity, ratings of faculty were consistently lower than when students were asked to sign the evaluations. Likewise, students in deflated grade conditions also provided lower instructor ratings. Aware of this possibility, instructors can easily gravitate to a position of either complete indifference to student commentary, or to a modification of teaching practices (by limiting demands and assignments, grading easily, and so on) in order to increase the ratings they are given by their students. In a review of grading levels before and after the introduction of student evaluations of Integrating Course 5 5 teaching (SET), Stratton, Myers and King (1994) found that grade levels increased 11% after the introduction of SETs. These effects did decay over time, however. Birnbaum (n.d.) surveyed 208 faculty at California State University, Fullerton on their judgments about the effects of student evaluations on student learning and their teaching practices. Almost three of four (72.1%) faculty respondents believed that the use of student evaluations encouraged faculty to dilute the rigor of their courses in order to curry student favor. In the same study, Birnbaum also found that, in a sample of 142 undergrads, virtually all (97.9%) gave higher ratings to easier rather than harder courses. A second problem with traditional student-based evaluations arises from a confusion over whether the coverage of the evaluation process should be formative or summative (Adams, 1997). A formative evaluation would collect information about the quality, adequacy and usage of various instructional inputs (like instructor classroom practices, readings, learning activities, instructional media used and so on). Further, since the ideal purpose of formative evaluations is to provide data so that improvements can be made as the course is being conducted, evaluative information should be collected and made available for use on a timely basis. A summative evaluation, on the other hand, provides an end-of-course review about course outcomes, specifically the quality and extensiveness of student learning in the course. Formative and summative evaluations are not necessarily incompatible. Even so, the requirements for doing adequately either formative and/or summative course evaluation are typically much more extensive than what is produced through standard student evaluations. By the time student evaluations are collected at the end of the semester, processed and reported back to the instructor, for example, it is too late to be of much contemporary formative value. In short, the timing Integrating Course 6 6 of such traditional student ratings of faculty makes them de facto summative, regardless of any formative aspirations. Moreover, it is questionable how much of the information collected from students can be meaningfully used in the way of summative results. For example, how much does student opinions of instructor teaching practices (a common item included in surveys) say about the quality of student learning (a key criterion of a summative evaluation)? Third, there is the Janus-like problem of focus or construct validity and contamination. In this regard, the key issue is what is being assessed? (Adams, 1997; Scriven, 1995) There are several conceptually distinct aspects of the instructional, educational and learning process that can be assessed, including in-class teacher performance (particularly lecturing style and quality); course planning, design and management (which includes any number of less visible and out-of-the-classroom activities); and student learning. Each construct requires its own distinctive evaluation procedures; for example, evaluating a teacher’s performance during a class is a much different task than evaluating how much students learned by the end of a course. Without a clear definition of what is to be evaluated, the risk of contamination – gathering information about conditions irrelevant to the construct under study – increases. Ideally, an effective evaluation procedure would include the following components. First, the coverage and focus of the evaluation should be well defined in terms of the construct being assessed and its formative/summative intent. A comprehensive and focused evaluation process would include a systematic means for compiling important and representative information about both instructor and instructional practices as well as showing linkages and connections between these Integrating Course 7 7 practices and student learning achievement and reactions to the course. Second, the major instructional inputs and processes used in the course should be identified in order to gather information about how well those inputs and processes worked. Third, this information should be collected at several times during the course and made available to the parties involved as quickly as possible. Fourth, the data-gathering and compilation process should not be too time consuming and/or labor intensive. Finally, a comprehensive evaluation would include multiple voices, including the instructors, students, and any other meaningful stakeholder (like peer evaluators, outside speakers, and so on). Clearly, student evaluations do not meet these standards. There are at least three reasons why full and complete course evaluations meeting these criteria are not systematically conducted. First, a complete course evaluation meeting the standards noted involves gathering a lot of information, which can make it seem both labor intensive and time-consuming. Second, systematic course evaluation is something that is typically not encouraged or rewarded in faculty evaluation systems (Cerbin, 1994). For the typically harried faculty member, a task that requires a lot of time but which has little if any value in promotion and tenure criteria is a task that will probably not be done. Third, there is a certain methodological inhibition to evaluation. Unlike syllabi, test construction, and other aspects of instructional practice that are more frequently done and have exemplars of practice that can be easily inspected and copied, course evaluation suffers in comparison; there is an aura of procedural uncertainty that shrouds the assessment process. Knowing what kind of information to collect, how to collect it and when is not immediately obvious or clear. Integrating Course 8 8 In addition, the first desiderata of correlating evidence on student learning with instructional practices are seldom met. Cerbin (1994), for example, notes that assessments tend to focus exclusively on either student learning or faculty teaching, yet seldom are these two domains of evaluation examined in tandem. Particularly difficult to examine are “how classroom practices contribute to learning outcomes” (p. 95). In this gap, Cerbin recommends a “learning-centered assessment” that would look at the interaction between instructional practices and student learning. This approach is based on using a course portfolio design1 as a way to carry out a learning-centered assessment. For Cerbin, a course portfolio should contain four core components: (1) a statement of the teacher’s assumptions about teaching and learning, goals for the course and the rationale linking the two; (2) an analysis of student learning based on class work and assignments; (3) an analysis of student feedback; and (4) a course summary. A portfolio can include examples of both instructional inputs and student outputs (like test scores, copies of graded student papers with comments, and so on). The result should be a four to seven page overall summary of the course. As a way to focus on how the instructional process impacts student learning, the portfolio approach offers an advance over traditional student evaluations. Still, compared to the ideal checklist of elements of a thorough course evaluation, there are several potential limitations. The instructor tends to be the exclusive author and voice, particularly in terms of defining student learning. “The course portfolio has as its center of gravity the data the teacher gathers about students’ learning and development….” (American Association for Higher Education, 2001). While student accomplishments should be included in the portfolio, less certain or systematic is the inclusion of students’ Integrating Course 9 9 voice. One set of guidelines from Samford University (2001) suggests that examples of student voices like interviews, journal entries, student notes, or reports about instructional experiences are optional. Further, course evaluations in general should be learning activities for the instructor about his or her learning product; when done well, a course evaluation (here, done as a portfolio) should be designed to help the instructor learn and improve. Yet a learning-centered assessment process is summative in nature, putting together the portfolio after the course has been completed. While appropriate on its own terms, this approach does not lend itself to exploring and improving educational processes and their effects as they occur during a course and/or over a semester. As such, this approach would have limited utility as a formative tool to aide in improving the instructional process as it is occurring. The process of evaluation, assessment and critique can provide an opportunity for students to reflect on their own involvement in the learning process. Yet, assessments do not necessarily involve students in critiquing or managing their own learning while in the course nor is assessment information shared with students. While certainly true in the case of student evaluations, this is also true of portfolios, and the potential value of portfolios as a device for aiding student learning is also missed. On the other hand, an additional benefit of a process that involves students in the assessment process would be to provide a means to help empower the students in gaining influence over their learning. Are there other approaches to evaluation that may overcome some of these limitations and integrate evaluation efforts, that systematically includes the student’s voice on a more real-time basis with a reflective critique of instruction and learning, and that can also provide evidence of the learning that is taking place? Can a procedure be Integrating Course 10 10 created that collects evaluative information from students on a continuous basis and then make that information available to students to assist in their learning and development while also serving as a guide for the instructor in course evaluation? Can such a process be done in an economical, efficient manner? One potential solution to these questions is the use of a “learning history” approach to program evaluation. The remainder of this paper describes what this procedure is and the results observed in testing it in a graduate course. From this experience, the underlying characteristics of a learning history approach to evaluation will be identified in order to generalize how this procedure could be applied to course, instruction and student evaluation. A comparison of the three approaches to evaluation reviewed here is included in Table 1. TABLE 1 ABOUT HERE The Learning History Technique In the 1990’s, management consultants Roth and Kleiner (1998) began developing a new procedure to be used for organizational change and knowledge management in work organizations. Calling it a “learning history,” this procedure involves collecting multiple perspectives on some corporate event, like a company reorganization or new product launch (Kleiner, Roth, Thomas, & Hamel, 2000). The resulting report captures the rich tapestry of thoughts and reactions experienced by people throughout the organization as the event unfolded. Since the primary purpose in creating a learning history is to spur “reflective conversations,” the resulting document is made widely available to members of the organization and is used as the basis for a Integrating Course 11 11 collaborative review of the process in order to promote organizational learning. According to Kleiner and Roth (1997, page 176-177), “learning histories seem particularly effective at raising issues that people would like to talk about but have not had the courage to discuss openly.” As a technique, a Learning History is defined by both the procedure used and the resulting report format. As originally formulated, the procedure was designed to elicit and record multiple voices of organizational members across the time span of an organizational change project. Specifically, the procedure involves several steps, beginning with planning the boundaries and scope of the project. Then, a series of “reflective” or depth interviews are conducted by a team of insider participants and outside researchers with a diverse and large sample of people. Even though the primary data collected through a learning history are the reflections and observations of people involved in the program, other data, like documents, records, observations, and so on, might also be gathered. Then, the data are scoured for common and recurring themes, and a first draft of the report (using the prescribed format) is produced. This report is double-checked for accuracy and significance by showing it to participants. Finally, the report becomes part of a workshop where participants go over the history, consider how typical the events were, and look for how they can use the learning history to extract lessons from the experience. The format requirements for writing and presenting the report are specific. First, the program under study should be decomposed into its major segments, stages or episodes. These stages become provocatively titled chapter headings of the report. Second, there should be a brief introductory segment to each chapter that describes the Integrating Course 12 12 nature of what happened, approximate timing, participants, conditions, and so on. Third, the data should be presented in a two-column format using the following guidelines: (1) representative or significant quotes taken directly from the interviews should be listed in the right column; participants are identified by job title only; (2) the left column is reserved for interpretations, questions, observed themes or commentary about what is happening. In effect, the right column is for the voices of the participants, while the left column is for analysis. The document may also contain other information than quotes, such as memos, announcements, or other documents. (A copy of the full report generated for the course reviewed here is available at www.towson.edu/~aclardy. A sample of the learning history for one class session is enclosed in the Appendix.) Adapting the Learning History A Learning History approach satisfies many of the desiderata for course evaluation procedures. For example, the learning history for any particular course would be comprehensive, covering both instructor and instructional practices as well as student achievement and reactions to the course. Second, the history should contain a complete and thorough description of both the inputs and processes used in the instructional process and in turn, there should be information about how well these inputs and processes worked. Using procedures described below, all of this information can be collected and reported regularly during the course and should not create excessive demands on time or labor. This approach also gathers multiple voices for the evaluation. I modified and pilot tested the basic approach for creating a learning history in a graduate course I taught in the fall, 2000. There were 17 students in the class; the typical Integrating Course 13 13 student was a 30 year old, working adult. The course, entitled Change in the Workplace, is a required three credit hour course in Towson University’s masters program in Human Resources Development (HRD). The course met for two and one-half hours once a week for 14 weeks. I had previously taught the course four times. The course has a double focus: to review the historical and current changes impacting the contemporary workplace, and to teach principles of change management. In the field of organizational change, the twin concepts of a “learning organization” and “knowledge management” (Davenport & Prusak, 1998) are important and were an organizing theme for the course. Within the HRD field, the idea of a “learning organization” has become very popular (see, for example, Senge 1990). The fundamental concern is how an organization can improve its performance by a self-conscious effort to critique its internal processes and to find and adopt new, more effective practices. In short, the idea is to promote learning. One result of improved learning is that the organization’s storehouse of knowledge will continually be improved; in turn, this also means that knowledge should be better stored, coded and made accessible to all members of the system. To demonstrate these ideas in practice, I treated the course and all class members as an organizational change project with a learning history approach as a centerpiece to the process. The platform which enabled this demonstration to be tested was the Learn on Line, BlackBoard 5 (BB5) system, the e-learning based instructional system from BlackBoard, Inc. This system allows an extensive range of course management and instructional options through web-based delivery. In 1999, Towson University installed the Blackboard 5 system; it is available to all students at no cost and can be accessed from either on- or off- campus locations. BB5 allows the instructor to create a website Integrating Course 14 14 for the course. The homepage for the course website contains a set of menu options that allows the following functions: posting of course documents (like overheads, text files, case studies, and so on); providing assignments and assessments (such as course and individual surveys, complete with tabulated results); email communications to members of the class and individual emails directly with the instructor; a section for links on the worldwide web to other, related websites; and a “Discussion Board” format that collects and maintains a historical file of all individual comments and opinions about a “forum” topic. In addition to posting individual comments, the Discussion Board function accepts, stores and makes available for general inspection attachments such as student papers. In the initial pilot of this project, the primary element used to support the creation of the learning history was through the Discussion Board.2 Over the period of the semester, students were required to make and post comments to specific questions on the discussion board. The first major modification of this approach to the normal learning history method was that no interviews were conducted; instead, student comments were taken from their posted comments and remarks. For example, during the first week of classes, students were required to post information about their learning goals for the course and about their personal and/or occupational background with organizational change. Then, at three, equally spaced times during the semester, students were required to make entries about what they were learning to a forum called the Knowledge Management Set (KMS). These were preset discussion fora with the same basic question: what have you learned about organizational change to date? In addition, students had to post two class assignments to the Discussion Board. One was a group Integrating Course 15 15 project designing a Diversity Management program, and the other was a copy of a major paper on some organizational change technique they prepared for the class. In order to demonstrate the knowledge management intention for this course, students posted several of their individual learning products, making what they learned available to all. Students were encouraged to download copies of other papers that they might want to use latter in their careers. Students were expected to complete an assessment of the course and instructional practices. This assessment survey was posted and answered on-line (instructions in class and with the survey emphasized that all responses were anonymous). A week later, the class was given the composite data from the assessment as part of a instructional module on survey feedback.3 The class was held on Thursday nights. In adapting the learning history approach, each class was treated as a separate event. To write the learning history, I would write a brief summary of what happened the night prior, along with any of my comments or analyses. In effect, in a second modification to the learning history technique, I wore two hats: course participant qua instructor and program observer. Where possible, I would take student comments from various sources, including the Discussion Board or emails to me directly. Even though student comments to the Discussion Board were posted for all to see, if their comments were included in the history, students were only identified as a “student” and not by their name. Then, the updated learning history of the class would be posted as a course document every few weeks. Students could see the evolving history of the course as it was being developed and written. Three points deserve note here. First, it took about half an hour on average each week to write each entry to the learning history. Other than typing my own comments Integrating Course 16 16 while writing the history, the inclusion of student comments was a cut and paste operation. Students typed their own comments in response to the questions and assignments they were prompted to provide. As a result, a large of amount of material taken directly from students was included fairly easily. Second, because the history was updated on a weekly basis, the result was an on-going and emergent document that provided an almost real-time account of what was happening in the course. By writing my comments within 24 hours after the class was completed, the record of what happened was reasonably contemporaneous. Wearing the hat of instructor, my personal comments ranged from critiques of my own instructional practices, to observations about student involvement in the course, and to ideas for improvements to instruction the next time the course was to be taught. Third, at the end of the semester, I used the complete document as the basis for evaluating what worked well and what did not in the course; these results will be discussed more fully below. Results Based on the pilot testing of a learning history approach to course evaluation, the following effects were observed. First, in the regular learning history protocol, the authors are different than the participants. In this class, though, I had to wear both hats as author and participant. One risk in participant observation studies is bias in what is seen and recorded, which may be colored by personal feelings and attitudes about the topic. While my affective responses were included (see the next section), I believe I completed the authoring duties without too much distortion. The main job of the author is to create a short summary of events, obtain and include comments from a variety of people Integrating Course 17 17 representing a range of viewpoints, and then offer interpretive comments and questions. The descriptive summary that introduced each section (i.e., each class) was a paragraph that chronicled the sequence of events as they played out in each session. Since this summary was almost a narrative reporting of the agenda followed in the class, there was little opportunity for bias to be introduced. In addition, I made a conscious effort to find both typical as well as atypical comments and include both in the history. Second, as would be expected from any truly formative evaluation, the act of preparing a learning history for the class had an impact on my performance in the class. The process of regularly writing about my intentions for, interpretations of and reactions to each class shortly after each session as well as seeing and including student reactions to those same events did affect my classroom instructional practices. For example, in one message, a student complained that the class did not always start precisely on time; this was due in large part to the vagaries of waiting on a working, commuting graduate population to park and arrive in class. Nonetheless, I made a conscious effort to start on time thereafter, regardless of the numbers of students present. Likewise, when I wore the participant hat and could present my own personal feelings about my experiences, the result was cathartic. I was able to vent any frustrations and/or doubts I was having with the class or my instructional design. For example, after class 8, I made the following entry after grading the first set of papers required for the course: I was disappointed by the general level of scope in the first papers, particularly given what I thought approached a cut and dried assignment…. More often than not, the reports were incomplete in covering [some of the key points covered in a history of workplace change]….In the syllabus, I should explicitly mention including a historical summary as part of the account of workplace change. After class 10, in an entry labeled “How much is me? How much is them?”, I noted Integrating Course 18 18 There is a recurring pattern of students not completing what I think are simple, clear and straightforward instructions. I’m getting both pissed and concerned about this. Why is it so hard to get the basic instructions through? Normally, I would not think of sharing this type of information with students, but, for me, this means of presenting my version of what was transpiring in the course allowed me to raise issues I was observing in the unfolding of the course. I could step outside the role of distant and isolated instructor to become more of a participant in the learning organization that was the class. Third, the use of the BB5 system did not require a lot of additional time. On-line collection of student comments made it easy to cut and paste their voice into the history. By channeling student comments into specific sections of the website (the Discussion Board, in particular), I was able to quickly find examples of student voices. The convenience of this means of collecting information comes at the expense of an important feature of the traditional learning history method (personal communication, Roth. 2000): there was no intensive interviewing of participants. Further, a situation in which students are asked to provide critical information without anonymity does raise flags about compromised and self-serving reporting due to fear of reprisal or a desire to curry the instructor’s favor. I did not have any reason to believe, though, that punches were being pulled. For example, after class 10, one student emailed me with this comment: I have a concern that I have to “get off my chest” regarding the last project [a review of a student-selected book on change]. I felt insulted in the presentation on [the book] because I had the understanding that we (the grad students) had to pursue more proven or “researched” methods of change management – not a self- help book….I felt that [that student’s] selection was both degrading and diminishing to the level of studies that we have pursued in this course so far. To make matters worse, [this student] was bragging that he was reading the book for another class. Integrating Course 19 19 By the end of the semester, the complete learning history contained 37 basically single-spaced pages. After the history was finalized, I reviewed all the remarks and observations it contained. Through this analysis, I was able to identify 16 specific and substantial items to change when I taught the course for the following semester. One would expect that while there might a large number of corrections the first few times a course is taught, one would also expect that the number of corrections should also taper off each additional time the course is taught. As noted, this was the fifth time I taught the course, a point by which one might expect the number of substantial modifications to be few. This suggests that the learning history approach may be a robust method for reflecting on practice in order to generate a continuously constant stream of ideas for course improvement and innovation. I do not believe I would have generated as many corrections in so systematic a manner through any other venue, nor, given my recordkeeping, would I have retained or stored these ideas in a single source for convenient future reference. The learning history was not without its drawbacks and limitations. Even though the emerging Learning History was posted for student inspection at the website every few weeks, I did not require them to look at it, and most students did not read the history as it was occurring. In addition, I did not include any of the following potentially helpful items in this history. First, the BB5 system provides various aggregate data about when students used the site and what features of the site they used. In general, neither did I include student comments picked up from class discussion. Both kinds of data would have added to the depth of the account. Second, I did not include actual copies of assignments (such as the diversity training plan the students had to design using virtual Integrating Course 20 20 groups) or the results of a survey which the students anonymously completed about me and the course. At least in this first attempt 4, examples of student learning were not as systematic as might be found in a Portfolio. Finally, the length of the resulting document (almost 40 pages) is a disincentive for other faculty to review. Toward a General Learning History Procedure for Evaluation Based on this pilot test, a more general model for how to apply and use a learning history approach to course evaluation can be proposed. While it is conceivable that a learning history approach can be implemented manually, clearly the advantage goes to using some type of web-based electronic communication process. In principle, while a basic email system could serve the same basic purpose, a web-based instructional support system is the preferred. Given a web-based e-learning system, the underlying design of a modified learning history approach to course and instructional evaluation can be built using the following six-point blueprint. First, the course itself would be divided into segments; depending on the type of course and scheduling cycle used, segments could be based on each class, each week, or each instructional topic (that might span class sessions). Soon after completing each segment, the instructor would write a brief description of what happened in each segment. The description would report primarily on the instructional events used but could include other significant events, such as spontaneous discussions of current events or assignment clarifications. The two-column format would be used to record both participant observations and commentary on those observations. Second, the voices of all classes of participants must be solicited and included. Certainly, instructor Integrating Course 21 21 thoughts, assessments and feelings about the events within each segment would be expected. As a co-participant in the learning enterprise that is a course, the instructor should make comments and observations. At times, those comments may focus on student involvement in the course (what students were doing well or badly); at times, those observations could be self-reflective commentary on what the instructor was doing well or badly. Third, student input would be expected. This can be done by requiring responses to specific questions at several times during the course. For example, students could be asked to identify their learning goals for the course or by asking about their expectations or preferences for classroom management or instructional practices. Other prompts might ask students to identify the key lessons they have learned about a topic, to report on (and even include copies of) activities they are completing (such as lab work or artistic creations), to record their critical opinions about a subject, or to make suggestions and recommendations. Some modest participation grade can be used to encourage thoughtful involvement. Fourth, while more optional, it is possible to include data from survey assessments, like anonymous course evaluations. (In the BB5 system, survey data is anonymous; descriptive data is instantly available.) Fifth, various kinds of student learning outputs, like copies of papers (perhaps already posted and available on-line), should be included. Finally, the Learning History of the class would be posted continuously during the semester. One requirement could be to have students read and respond to the History. A graphic depiction of this application model is shown in figure 1. FIGURE 1 ABOUT HERE Integrating Course 22 22 Further Study The learning history approach to evaluation needs further study. For example, since a learning history is posted during the course, making instructor thinking and reactions more transparent and accessible, it would be interesting to see what effects, if any, a learning history might have on motivation, student learning or instructional quality. For example, would there be any differences in motivation or learning if students were given responsibilities for preparing or maintaining portions of the learning history for the course. A student in a subsequent section of this course made this comment about the learning history I had prepared for her class: “It really gave me a different perspective on the class. The comments and observations were surprisingly honest and forthcoming. I can see how this could be a useful tool …, it seems that it could make individuals more aware of and accountable for their actions/behavior.” Does the use of an e-based medium make any difference in the quality, depth and “honesty” of student comments? Would student comments made directly to an email prompt be any different compared to data collected in interviews by outside third parties? The learning history was used in a course because it was actually part of the instructional content (a technique for knowledge management and organizational change). Can a learning history approach be used in courses where it is not part of the instructional content? That is, can it be used equally well in science, literature or physical education courses? This approach was used with a somewhat older and more mature population of graduate students. Can it be used with a more traditional undergraduate population? Finally, how do faculty and/or administrators compare a learning history report in value to the traditional student surveys and/or the portfolio model? Since this size of a learning history might be four to Integrating Course 23 23 five times that of a course portfolio, it is important to look at the relative value of this approach in terms of quality, specificity and value of the information provided. Improving the quality of learning and instruction is a lofty and important goal. Unfortunately, the most common methods of student evaluations are also the weakest in terms of supporting the achievement of that goal. The portfolio approach, clearly an advance, is still somewhat limited in terms of timing and perspective. The learning history approach, using an on-line capability, offers a new way to move toward this goal by integrating various forms of evaluation into one common procedure. Even though this approach is not without its limitations, there are also potential benefits. Based on the first results of the pilot test reported here, a learning history can be a new tool in the quest for educational excellence and deserves further consideration. Integrating Course 24 24 Endnotes 1. This procedure has been amplified and promoted by the American Association for Higher Education’s Teaching Initiatives project (www.aahe.org/teaching/Teaching_Initiative_Home.htm). Braskamp and Ory (1994) distinguish between course portfolios (for one course at a time) and teaching portfolios (a composite of several courses). 2. Unavailable and hidden to student view in BB5 is a set of usage reports that indicate when students use the system, how often they access various elements, and who completed what assignments. While this information could be included, it was not in this pilot test. 3. Survey feedback is a technique of organizational change. Members of a social system complete some kind of attitude or opinion survey, say about the leadership practices of their boss, or their job satisfaction. Those results are compiled and then a summary of the results is presented back to the members, typically meeting as a group. The members then study the data to recommend changes and improvements to the system. 4. I continue to use a Learning History for the same reasons in the same course. Now, I include a variety of student products and outcomes, like examples of papers, assignments and survey results. Integrating Course 25 25 References Abrami, P.C., d’Apollonia, S., & Cohen, S. (1990). Validity of student ratings of instruction: what we know and what we do not. Journal of Educational Psychology, 82, 2, 219-231. Adams, J.V. (1997). Student evaluations: the rating game. Inquiry, 1, 2, 10-16. American Association for Higher Education. (2001). Teaching Initiatives. Retrieved on June 12, 2001 from http://www.aahe.org/teaching/Teaching_Portfolio.htm. Birnbaum, M.H. (no date). A survey of faculty opinions concerning student evaluations of teaching. Retrieved on September 24, 2001 from http://psych.fullerton.edu/mbirnbaum/faculty/3.htm. Blunt, A. (1991). The effects of anonymity and manipulated grades on student ratings of instructors. Community College Review, 18, 4, 48-53. Braskamp, L.A., & Ory, J.C. (1994). Assessing Faculty Work, Enhancing Individual and Institutional Performance. San Francisco: Jossey Bass. Cerbin, W. (1994). The course portfolio as a tool for continuous improvement of teaching and learning. Journal of Excellence in College Teaching, 5, 1, 95-105. Davenport, T.H., & and Prusak, L. (1998). Working Knowledge, How Organizations Manage What They Know. Boston: Harvard Business School Press. Kleiner, A., & Roth, G. (1997). How to make experience your company’s best teacher. Harvard Business Review, 97, 5, 172-177. Kleiner, A., Roth, G., Thomas, T., & Hamel, E. (2000). Oil Change: Perspectives on Corporate Transformation. New York: Oxford University Press. Integrating Course 26 26 Roth, G.L. (2000). Constructing conversations: lessons for learning from experience. Organization Development Journal, 18, 4, 69-78. Roth, G.L., & Kleiner, A. (1998). Developing organizational memory through learning histories. Organizational Dynamics, 26, 43-60. Samford University. (1998). The Samford PBL [Problem-Based Learning] Initiative, Course Portfolio Contents. Retrieved on June 12, 2001 from http://www.samford.edu/pbl/aboutsu3.html. Scriven, M. (1995). Student ratings offer useful input to teach evaluations. ERIC/AE Digest. Retrieved on September 24, 2001 from http://www.ed.gov/databases/ERIC_Digests/ed398240.html Seldin, P. (1984). Changing Practices in Faculty Evaluation. San Francisco: Jossey Bass. Seldin, P. (1993, July 21). The use and abuse of student ratings of instruction. The Chronicle of Higher Education, A-40. Senge, P.M. (1990). The Fifth Discipline, The Art and Practice of the Learning Organization. New York: Doubleday. Stratton, R.W., Myers, C.M. & King, R.H. (1994). Faculty behavior, grades and student evaluation. Journal of Economic Education, 25, 1, 5-15. Sustaining Change by Learning from Collective Experience. Retrieved on July 9, 1999 from http://www.learning.mit.edu/res/projects/change.html. Integrating Course 27 27 Appendix. Sample Record from the Learning History Document _____________________________________________________________________ Class 7. LET THE GAMES BEGIN! After collecting the assignments due for tonight, we spent about 10 minutes going over assignments again. Dr. Clardy then made the transition into the shift in the focus of this course from looking at the forces impacting the workplace to the question of how to manage change. Tonight was the inauguration of my change management model. We covered the material up to resistance, then did the assignment about force field analysis for poor levels of supervisory performance. This took about 45 minutes. We ended by the Assignment 3 project teams meeting again. Frustration with students understanding the assignments Instructor: Again last night, we killed about 15 minutes at the start of class going over what the assignments will be. This included me spending 10 minutes at the break going over Assignment 2. All that plus the first few papers I’ve graded for Assignment 1 were not complete. I’m getting very frustrated. How can we be spending some much time going over the same assignment expectations and still not getting it? I don’t know how I can be any more specific than what I’ve already included in the syllabus, plus the supplemental resources (professional practice paper and examples of prior papers) in Bb5. This is a real problem and I need to investigate what’s causing it. We’ve probably killed close to an hour of class time (excluding the first class) beating these dead horses. I’m running out of patience on this matter. Instructor: The force field analysis exercise probably should be presented as text, rather than as a hypothetical example circling around examples of poor supervisory performance. Or, perhaps better still, keep the brainstorming listing of poor supervisory practices, but provide a simple version of the conditions in place in the organization: for example, here’s the supervisor evaluation form, here’s a story from the culture, etc. This spoon feeds the answer a bit, though. Integrating Course 28 28 Dr. Clardy sent email announcements to all students with a notice about an upcoming SHRM meeting as well as a notice about having posted Learning Organization overheads. Also, the instructor survey ratings were obtained. There were 11 responses of 16 students. How to be able to use survey data most effectively? This suggests the importance of training and discipline in: * rational emotional therapy and dealing with irrational beliefs * humility Things done well:  Humor  Encouraging participation  Feedback  Use of exercises and activities Instructor: Opening up personal survey data is always a challenge. Curiously, I was more nervous and concerned about seeing this data than I am when I get course survey data (from the university). Possible reasons: it’s more timely and it involves a set of factors which were developed from and with the students. Interpreting results is always a challenge. Anything less than perfection is riveting and produces feelings of despair or defensiveness or both. How to train oneself to accept the information humbly and to use it constructively without getting defensive or depressed? After 25 teaching and training, that skill still eludes me. Student 1. Encourage student participation. Offer ideas in group discussion. Ie: Have you thought about it this way... Student 2. He should continue to show videos and have small group discussions because the hour is late and I'll fall asleep if he doesn't. Student 3. He motivates the class. Often uses humor to motivate the class. Always answer class questions. Provides feedback on all assignments. I enjoy the mini projects that assigned during class. Student 4. Very good use of class time; exercises are engaging and appropriate (Dr. Clardy is very good at showing connection/relevance); sometimes feel like a guinea pig, although stretching comfort zone, it is challenging and thought provoking. Student 5. Willingness to adjust class schedule and due dates to class needs. Ability to make material interesting - incorporation of exercises and media, etc. Student 6. I enjoy Dr. Clardy's playfulness and creativity in class, and it is a change for me, most instructors being more focused on lecture. He seems very contemporary, very concerned with the well-being and fairness of each of the students. He keeps it light, which is a nice change of pace. Integrating Course 29 29 Things to change or improve: Several comments about the amount of web-based activity and the fact that they are ungraded Is student saying that more time should be devoted to group work and discussion? It is difficult to identify action suggested if it is not listed. Student 1. Start on time & end on time. Student 2. He should stop giving so many course info assignments. The projects and the readings are plenty of work in themselves. Student 3. Way too much work on the web for students. Even if Dr. Clardy sees this as the way the world is going in terms of communication, it might be too soon to introduce it. I suggest he reduces the amount of web assignments for the students. Perhaps he is before his time? Student 4. There are assignments on course info that are not graded, but they seemed like a mini project. Course info may be over-used. Student 5. I think that this class has a lot to offer and I would like to learn from some of my fellow students (including & beyond the syllabus). They've shown some great ideas and creativity in group work and class discussions. This class has a lot of people who have diverse experiences and backgrounds. No responses or not sure: 4 Instructor: I’m wondering if there were any concerns from students about the anonymity of their responses and whether they hedged their answers in any way. Students were to complete the first Knowledge Management Set (KMS) entries by this time. All but 3-4 students had done so. A summary of their comments is shown below. The larger context in which change is to implemented must be taken into account. This includes the global setting as well as the full range of organizational practices that either support or inhibit change. Same point by another student. Student 1. In a more micro view of change, individualistic resistance to change must be approached by a strong organizational culture AND action. How often organizations talk about change management, yet do not set up the environment or reinforcers for such behavior. Change has to be managed at ALL levels of the organization. I feel that two important characteristics of organizational change is proactive responsiveness and adaptation. By “proactive responsiveness” I mean that an organization may project what customers want and test the product in all forms before distribution, … and fix them in a timely manner. This can apply not only to products issues, but also management policy. Adaptation is simply this: moving out the “comfort zone” of business practices. In the technological revolution going on, it is difficult to develop an original idea and keep it as one. As soon as it gets out, it is changed and adapted into hundreds of other forms. Integrating Course 30 30 Change applies to the traditional role of HRM as a rule maker. Must keep some ethics as rules. Creating alliances to help make change. Imperatives for change call for new or different roles for HR. Resistance to change is seen in the Ford Motor example. Resistance is a recurring theme in these comments. Yet companies need to change in order to survive. Another student agrees about the extent and amount of change taking place. Recognizes a contingent approach to change management. People involvement is important. Techniques: SWOT and SMART Student 2. Human resources has seemed to exist in it's own world for years. Acting as the creators and deliverers of "the rules" have been the norm…. Business Partnering within an organization can help to facilitate this move and link HR with the rest of an organization. This partnering can promote organizational change. But, sometimes it seems that we are so eager to change an organization and move forward, that we may forget the "rules" we must keep in order to remain ethical. Student 9. I agree with Jeannine in terms of ethics & the role HR. I think that for years we've had a mechanistic approach to change (linear cause and effect and "cog in the wheel"), but we're moving into an organic approach (everything is connected and affected by each other and the environment). This certainly opens HR into a role as team member, diplomat, consultant & facilitator. This shift means that the HR professional not only needs to know employee/employer rights, but also have developed skills to work effectively in teams, skills in diplomacy, and interpersonal communications Student 4. So far, I have learned a number of important ideas about change: 1. People are extremely resistant to change (as in) the movie that we viewed about the Ford Motor Company, we saw the extent to which the employers resisted the change. 2. While companies like to bury their heads in the sand when it comes to change, analysis and change is the key to a company's survival. If they don't grab the opportunity to change, others will pass them by and they will soon be non-existent. 3. Nowadays, it seems that organizations are in a constant state of change so that it is almost dizzying to think about. 4. It is vital to weed out the important changes an organization must make from the trends and styles of society. To follow every whim of society is a sure key to failure. Student 5. I feel the biggest thing I have learned so far is that organizational change is not easy to understand or manage. I have also realized that the changes in my office have not been handled in the best ways. Every situation requires a different approach and a different solution. I think the most important thing is to involve people in the change process. No one likes to be forced around when it comes to change. Student 6. The hardest part about change is preparing for it. I think that in order for companies to be successful in their change, they need to take into consideration two models (Swot and Smart). First the companies need to figure out their strengths, weaknesses, opportunities, and threats. Second I think that the companies need to understand whether or not their change strategies are specific, measurable, achievable, realistic and timely. Integrating Course 31 31 Three generators of change: Employers, employees, government. Must look at broad complex of factors. Problem: how to generate change? How do you create a motivation for change, particularly among long- term staff? Is it a “discussion” board or a “posting” board? Student 7. I believe there are three different types of organization change. The first type of change is being lead by the organization. This incorporates organization hierarchy, policy, procedures, missions, anything that is catered to the organization. Another type of change is lead by the employees (Ford). Employee influenced organization change would include unions and civil rights groups. Finally, government mandates is the third method of change. Examples of mandated change would be EEO, AA, FMLA, ADA, etc. Student 10. So far, I have learned that diagnosing, planning, conducting and/or evaluating organizational change has to encompass many factors outside of the change itself. What is/will be the impact of the environment? technology? people involved? etc. I've never considered the components that forced the transition from colonial to industrial to contemporary workplaces. And I've been a part of the change due to technology, so I've not known what it would be like to work without a computer and networks. And I believe honesty is one of the most important concepts that should be involved in any change process. Secretive or dishonest methods of instituting change just make people resist the ultimate change. Student 13. The question I have about organizational change appears to be easy to answer. What can a manager do when an organization implements change in procedures and policies -BUT NOTHING CHANGES! For example, a new program has been implemented in an organization along with new staff to train the old. The problem is the veteran employees are resistant to the change-what does a manager do to change their mindset? Management does not want to terminate the employee/employees because they are valuable to the corporation. Instructor: I’m noticing that the students as a rule are simply posting their comments and not engaging in any discussion on the items posted by others. On the other hand, I’ve resisted responding to each posting, other than the diversity reports. I probably need to prime the pump but responding more frequently. Integrating Course 32 32 Table 1. A comparison of three evaluation procedures Components of an effective evaluation process Student Evaluations Portfolio Learning History Well-defined focus Questionable Yes, summative only Yes, both formative and summative Major instructional inputs and processes used in the course identified No Yes Yes, along with on- going critique of those processes Regular collection of information as the course unfolds No No Yes Easy to use process Yes No Yes Includes multiple voices No May Yes Integrating Course 33 33 Figure 1. A Blueprint of a Learning History Course Evaluation Structure REGULAR WEEKLY INPUTS Instructor inputs:  Descriptions of what happened in each segment  Assessments and commentary COURSE Segments of the course (here, weekly class sessions) PLANNED PERIODIC INPUTS DURING THE COURSE Student voice inputs Student learning goals and expectations Open discussion boards Email communications Periodic reports on what they have learned Posted student assignments with instructor comments Survey data SUMMATIVE INFORMATION AND OUTPUT PRODUCTS System Administration Information Usage data from the system: timing, participation Examples of Student Achievements Copies of papers and reports Test scores Q&A forum