STIUDENT PERCEPTIONS OF EFFECTIVE TEACHING BEHAVIORS REVISITED Developments in Business Simulation & Experiential Exercises, Volume 9, 1982 184 STIUDENT PERCEPTIONS OF EFFECTIVE TEACHING BEHAVIORS REVISITED George E. Stevens, Arizona State University Sheila A. Adams, Arizona State University ABSTRACT This paper is an extension of an earlier study examining student agreement on the importance of course- related teacher behavior. One finding of the earlier study was substantial agreement between males and females on the characteristics considered important to teaching effectiveness. However, the results seemed to indicate a greater emphasis given by students to those characteristics which impact directly on grades than those which impact on leaning. The results of this study indicate that students do differentiate teacher traits on the basis of importance to leaning or to good grades. INTRODUCTION As a device for evaluating the performance of college teachers, the student evaluation of teacher effectiveness has become widespread. Typically, the student fills out a standardized evaluation form near the completion of the class in which he or she is enrolled. Evidence for the extensive use of teacher evaluation has been documented by a number of researchers [e.g. Lein & Merz, 1978; Peterson, Kerin & Martin, 1978]. For example, in a study designed to identify the methods used for evaluation of business faculty, Lein and Merz [1978] received responses from 374 business schools. Although respondents indicated that they used various combinations of methods to evaluate business faculty, over 70 percent of the schools used some form of teacher evaluation by students. Not only are more schools using this method of assessing teacher effectiveness, many also use the results to make administrative decisions (e.g., faculty retention, promotion, salary and tenure). As the use of student evaluations of teacher performance has increased, so has the amount of literature reporting the uses and abuses of these devices [see, for example, Miller, 1978; and Miller, Brokaw & Shaaban, 1977]. It is evident that there are both proponents and opponents of the use of student evaluations of teachers as input into personnel decisions. Most faculty members agree that these evaluations have value if used for faculty development purposes, but are leery of their usage for other purposes. One reason for this concern is the many reliability and validity issues related to teacher evaluations--issues which have been investigated by a number of researchers. Researchers have discovered, for example, that many who construct such ratings are not sufficiently qualified to do so [Costin, Greenough & Menges, 1971]. Furthermore, when colleague and supervisor ratings of teacher effectiveness were also obtained, low correlations were found between colleague or supervisor ratings. One controversial study [Rodin & Rodin, 1972) concluded that students are not able to judge teaching effectiveness. Many. variables have been identified which influence student perception of teacher effectiveness. In many cases, either the teacher cannot control the variables or the variables may be difficult to measure. Studies undertaken include those examining student attributes such as student achievement [Banziger & Smith, 1978; Costin et al., 1971]; achievement factors [Banziger & Smith, 1978]; personality traits [Warren & O’Connell, 1978]; and leader behavior or style [Swanson, 1975; Kinicki & Schriesheim, 1978; Baba & Ace, 1978]; type of course, i.e., required vs. elective (Miller, 1978; Miller et al., 1977]; course content, i.e., nonquantitative, primarily conceptual such as organizational behavior and marketing to more quantitative, less conceptual such as finance and operations management [Neely & Schaffer, 1979]; teacher demands [Sullivan & Skanes, 1974]; class size [Miller, 1978; Miller et al., 1977]; sex of teacher [Elmore & LaPointe, 1975; Wilson & Doyle, 1976]; and teacher personality [Elmore & LaPointe, 1975; Witty, 1947]. Although full discussion of these issues is beyond the scope of this paper, the interested reader is directed to reviews such as those of Costin, Greenough & Menges [19711 or Sullivan & Skanes [1974]. A recent study [Stevens & Marquette, 1979] examined differences between faculty and student ratings of the importance of teachers’ course-related traits. Another [Stevens, Adams & Stevens, 1980] investigated differences among the students themselves about the importance of course-related teacher behaviors. If differences exist in terms of faculty and student ratings of teacher effectiveness then the potential value of student evaluations may be severely limited. Other studies have found that supervisors and subordinates tend not to agree as to the relative importance of dimensions of the subordinates’ job [Cummings & Schwab, 1973] and, in fact, disagree over subordinates’ jab duties and requirements [Maier, Hoffman, Hooven & Read, 1961]. One result of the Stevens & Marquette [1979] study was the finding that both students and faculty disagree with the statement that they “used the same criteria to evaluate performance.” Stevens & Marquette concluded that differences do exist between student and faculty perceptions of important teacher traits. One result of the Stevens, Adams & Stevens [1980] study was the conclusion that students seemed to place more emphasis on characteristics which have impact directly on grades more than those which have impact on learning. If teachers base their effectiveness standards on leaning criteria and students base theirs on grading criteria, the use of student evaluations may be inappropriate as a means of judging teacher effectiveness. The question then arises, do students differentiate behaviors on the basis of the importance of those behaviors to grading and to learning? Is there a difference between the importance ratings of undergraduate and graduate students? Of males and females? Of older and younger students? In an effort to answer these questions the present study was designed as an extension of the earlier Stevens, Adams & Stevens [1980] study. For the present study a sample of undergraduate and graduate students was asked to rate the same 17 traits used in the 1980 study. These traits are commonly regarded as characteristic of effective teaching in courses where the teaching mode is predominantly lecture. METHOD Sample and Procedure A convenience sample of 409 students enrolled in a large southwestern state university was used. Responses were received from 134 undergraduates and 275 graduate students enrolled in principles of management or beginning personnel courses. Questionnaires were administered to the students by a faculty member during regular class time. The questionnaires were accompanied by a cover letter assuring respondents of complete confidentiality. Developments in Business Simulation & Experiential Exercises, Volume 9, 1982 185 Of the total number of students completing questionnaires, 74.3 percent were mature (that is, 23 and over) while 25.7 percent were in the traditional age range of 22 or younger. Sixty-three percent of respondents were male and 36 percent female. Graduate students made up 67.2 percent of the sample (at least partly accounting for the skewed age distribution) with 32.8 percent of the returns coming from undergraduates. These data are shown in Table 1. Instrument The original questionnaire was composed of two parts. For purposes of this report, however, data from only the first part, “Traits,’ will be analyzed. This part of the instrument contained 17 statements describing teacher traits. Respondents were asked to indicate on a scale of 1 to 7 how important they believed each trait to be in helping “you get good grades.” The 17 traits were repeated and the respondents were then asked to indicate on a scale of 1 to 7 how important they believed each trait to be in “helping you lean.” Demographic data were also requested indicating age, sex, class, and grade point average. Results A comparison using t-tests was made of the means of each pair of variables, that is, each one listed under importance to grades with the same variable listed under importance to leaning. The means and the 2-tail probability value for each trait are shown in Table 2. Significant differences between the means are seen in 13 of the 17 items. In seven of those the importance to leaning is greater than the importance to grades. Of these seven only one, entertaining lectures, seems to contradict our traditional thinking about teacher effectiveness. Six items are seen as more important to grades than to leaning. All of these items are consistent with popular wisdom about influences on grading. In an earlier study [Stevens, Adams & Stevens, 19801 it was noted that the ranking of traits seemed to yield a clue as to which characteristics of teachers are considered important by students. The first three items in the ranking, we suggested, had a clear, direct impact on a student’s grade in the course. These items were “tests related to course materials” (#16), “clear expectations” (#17), and “fair tests” (#15). Some comparisons may be made between data presented in Tables 2 and 3. As can be seen from Table 3, in the present study these three items all appear in the top five mean rankings for all demographic breakdowns as Important for Grades. Furthermore. Table 2 shows that students differentiate between the importance to grades and to leaning for these three traits. Not only is the difference in means statistically significant (p < .001), but there is a difference of more than 1.0 between each pair of means indicating a difference of practical significance as well. While the three traits are considered by respondents to be somewhat important to learning (means of 5.0 to 5.4), they are seen as extremely important (means of 6.3 to 6.6) to grades. As can be seen on Table 2, a statistically significant difference exists between the means for 13 of the 17 items when importance to grades is compared with importance to leaning. Table 3, by highlighting the five highest ranking traits in each category, further illustrates this distinction. Note that traits numbered 16 and 15 appear as first and second (i.e., their mean scores were the highest) in every sub-group of Importance to Grades. Only once (as fifth for the undergraduates) does either of these traits appear as Important for Leaning. On the other hand, traits 7 and 6 appear as first and second in Importance to Learning for all sub-groups, but neither appears in any sub-group of Importance to Grades. This result tends to support popular wisdom in that we would expect stressing applications (7) and emphasizing understanding of concepts (6) to contribute importantly to learning. It may be reassuring to some that the students, in this sample at least, agree. It may be less comfortable to note that the item with the lowest mean rank for either Grades (4.438) or Leaning (4.069) is ‘has high grading standards.” Faculty often maintain that such standards contribute to learning; students seen less sure. In a similar vein, it is interesting to note that item number two, “lectures in an entertaining manner” is seen, by the younger students at least, as being important to learning (fifth ranked), while it does not appear in the top five of any other sub-group. Tables 4 through 9 (these tables are available upon request from the authors) provide the complete breakdown of item means for grades and learning for each of the sub-groups. Students’ ages made little difference in how they scored the importance of teacher traits to leaning or to grades. Younger students seemed to place more weight for learning on entertaining lectures (#2), teacher’s concern for students (#10) and the availability of teachers outside of class (#11) than did more mature students. (These results are reported in Table 4.) This is not unexpected. Younger students, when considering importance to grades, also scored organized lectures (01), entertaining lectures (#2), predictable exams (#3), and concern for students (#10) higher than the mature students did. However, the mature students scored exams requiring creative thinking significantly higher than did their younger classmates. (See Table 5.) More differences were noted when the undergraduates’ means were compared with graduates’ means. Table 6 indicates that undergraduates were significantly more concerned about the impact on grades of eight items than were graduate students. Graduates scored only one item, requires creative thinking on exams, higher than undergraduates. This difference also existed in the age breakdown which, since one could reasonably expect graduate students to be over 23, is a predictable similarity. The same item was scored as significantly more important to leaning by the graduates than by the undergraduates. These results are shown in Table 7. Undergraduates rated five items as significantly more important to learning than did graduates. These items included entertaining lectures, predictable exams, concern for students, accessibility outside of class, and objectives which reflect teaching. Indicating the importance of class material for exams is clearly related to grades; however, it is, perhaps, harder to see its relationship to learning. Entertaining lectures, similarly, are difficult to relate to learning. Students may be assuming that entertaining lectures will help them to pay closer attention to the course material and, therefore, lean more. Likewise, they may believe that if the importance of material for exams is indicated they will be concentrating on, and remembering, the important material of the class; again, helping them to learn more. The largest number of significant differences between means is apparent in the breakdown by sex. Table 8 indicates 11 items for which there is a significant difference between the mean ratings by females and the mean ratings by males. Curiously, for every one of the 17 items, the mean rating by females is higher than the mean rating by males. We can only speculate Developments in Business Simulation & Experiential Exercises, Volume 9, 1982 186 on possible reasons for this result. Since females typically are better performers tn the student role they may place more emphasis than males do on grades and leaning. If so, they might tend to see relevant traits as more important than males do simply because those traits are related to goals (grades and learning) on which the females place more importance. Only four of the items on Table 9 reflect a significant difference between males’ and females’ ratings of Importance to learning. All four were also significantly different in importance to grades. The difference in ratings by males and females seems to be a departure from the findings of our earlier study [Stevens, Adams & Stevens, 1980] which showed unexpectedly high agreement between the sexes. However, note that the differences found here are all a matter of degree. Females consistently rated all traits higher than males did but both sexes rated the identical traits as the top four of the seventeen. Females simply saw them as even more important than males did. There is, then, agreement between the sexes that, for instance, understanding concepts and stressing applications are both more important to grades and leaning than entertaining lectures are. CONCLUSIONS The analysis of student ratings of the 17 course- related teacher traits in this study indicates that students do tend to differentiate teacher traits on the basis of the importance of those traits to learning or to good grades. Seven items were seen as more important to leaning and six as more important to grades. Although all but one of these traits are consistent with academic expectations, a problem may still exist in the use of student evaluations of teacher effectiveness. Since evaluation forms do not specify whether teachers are to be evaluated on their contribution to leaning or to grades, student evaluations may mix up objectives. If we wish to evaluate effective teaching on the criterion of contribution to student leaning, care could be taken to include those items which relate to leaning, and possibly, eliminate (or at least minimize) those that relate to grades. Furthermore, directions could explicitly ask students to evaluate their instructors contribution to their own leaning. Differences noted between graduate students and undergraduate students were as expected. The similarity of results between class and age categories is not unusual given the expectation that most (but obviously not all) of the mature students will be graduates and few (if any) of the younger students will be graduates. The differences noted in these categories seem to bear out the expectation that graduate students are more independent, more willing to take responsibility for their own learning and grades, while undergraduates are more likely to want to be entertained, to be actively guided and cared for. In a class composed of both graduate and undergraduate students this phenomenon could create misleading evaluations of the instructor as well as confusing feedback to him or her. The differences noted between male and female ratings tend to corroborate our present understanding of differences in male and female classroom performance. On whether there Is any further significance to the oddity that all female ratings exceeded male ratings, we can only speculate. Could there be a similarity to the graduate-undergraduate difference? That is, are males more willing to accept responsibility for grades and learning and are females more dependent on the instructor? The data do not provide an answer. One shortcoming of this study is that no items were included which could reasonably be expected to receive low (not at all important) ratings. It is difficult to determine just how discriminating students are when nearly all ratings are at or above 5.0 on a 7 point scale. Other areas for future research include further attempts to determine those characteristics seen as more important and less important for learning by faculty and comparing student rankings, for learning, of those characteristics with faculty rankings. If a set of traits can be found which both students and faculty consider to be important for student learning we may move closer to making student evaluations of teacher effectiveness a useful tool. Developments in Business Simulation & Experiential Exercises, Volume 9, 1982 187 REFERENCES 1. Baba, V., and H. E. Ace. ”Leader Behavior in the University classroom,” Western American Institute for Decision Sciences Proceedings, (1978), 184-136. 2. Banziger, G. J. and R. K. Smith. “Student Motivation and Course Evaluation.” Western American institute for Decision Sciences Proceedings, 1978). 181-183. 3. Costin, F., W. r. Greenough, and R. J. Menges. “Student Ratings of Collage Teaching: Reliability. Validity and Usefulness.” Review or Education Research, 41(1971), 551-555. 4. Cummings, L. L., and D. P. Schwab. Performance in Organizations: Determinants and Appraisal. Glenview, IL; Scott, Foresman, 1973. 06. 5. Elmore. F, B.. and K. A. La Pointe. “Effect of Teacher Sex, Student Sex, and Teacher Warmth on the Evaluation of College Instructors.” Journal of Educational Psychology. 67(1975), 368-374. 6. Kinicki, A., and C. A. Schriesheim. “Teachers as Leaders: A Moderator Variable Approach. Journal of Educational Psychology, 70(1978), 928- 935. 7. Lein, C. D., and M. Merz. Faculty Evaluation in Schools of Business: The Impact of AACSB Accreditation on Promotion and Tenure Decisions. Collegiate News and flews. 31(1978), 2, 21-24. 8. Mater, N. a. F. • L. a, Hoffman, 3. J. Hooven, and W. H. Read Superior-Subordinate Communication: A Statistical Research Project. New York: American Management Association. 1961. 9. Miller. D. W. “Dangers of Using Student Evaluations for Administrative Purposes.” Collegiate News and Views, 31(1978), 3, 2- 6. 10. Miller, D. W., S. Brokaw. and F. Shaaban. “Administrative Uses of Student Evaluation of College Instruction.” Annual Meeting of the American institute for Decision Sciences Proceedings. (1977). 589. 11. Neeley, S. E. and B. F. Schaffer. Student Evaluations of Business Courses: Some Findings s.: Journal of Business Education, 54 (May 1979) 366-368. 12. Peterson, ft. A., R. A. Kerin, and W. S. Martin. “Teaching Effectiveness in the Decision Sciences; The Use of Student Evaluations.” Decision Sciences, 9 (1978). 712-714. 13. Rodin, M., arid B. Rodin, Student Evaluation of Teachers. Science. 177(1972), 1164-1166. 14. Schwab, O. P. “Attitudes Measured Prior to taking a Course and Subsequent Evaluations of That Course and Instructor,” Annual Meeting of the Academy of Management Proceedings, (1977), 454. 15. Stevens, C. E., S. A. Adams, and F. W. Stevens. “Student Perceptions of Effective Teaching Behaviors.” Proceedings of the Association for Business Simulation and Experiential Learning, (1983), 133-136. 16. Stevens. C. K. and R. P. Marquette. “Differing Student and Faculty Perceptions of Teaching Effectiveness and the Value of Student Evaluations.” POD Quarterly. 1(Winter 1979). 207-219. 17. Sullivan. A. H., and G. R. Skanes. Validity of Student Evaluation of Teaching and the Characteristics of Successful Instructors. Journal of Educational Psychology, 66(1974), 584-590. 18. Swanson, R. C. Teacher Evaluation as a Function of Leadership Style: A Multiple Correlational Approach, Unpublished doctoral Dissertation, North Texas State University, 1974. 19. Warren, w., and W. F. O’Connell. “The Effect of Instructor Reputation and Personality Upon Instructor Evaluations”, Annual Meeting of the Academy at Management Proceedings, (1977), 45-69. 20. Wilson, D., and K. O. Doyle. Student Ratings of Instruction.” journal of Higher Education. 47(1976), 465-470. 21. Witty, P. “An Analysis of the Personality Traits of the Effective Developments in Business Simulation & Experiential Exercises, Volume 9, 1982 188 Teacher.” Journal of Educational Research. 40(1947). 662-671. Table of Contents Volume 9, 1982 The Value of Conjoint Analysis in Enhancing Experiential Learning The Effect of Participation in a Collective Bargaining Simulation on the Expectations and Attitudes of Union and Management Representatives Conflict Resolution in Experiential Learning The Use of Simulation to Test Theories of Bargaining in a Business Context The Nominal Group Technique: A Vehicle for Improving Case Method Courses A Framework for Developing a Business Policy Case Produce Your Own Video Cases for Classroom Use: A Demonstration An Experiential Effect from Charismatic Encounters Nine Topic Oriented Mini Simulations: Descriptions, Purposes, and Observations A Hospital Simulator (HOSPSIM) A Report of the Model and Results Expected from Field Testing An Evaluation of SLIM (A System Laboratory for Information Management) An Experiential Exercise Introducing Students to the Role Ambiguity Faced by Salespersons The Advertising Research Project as an Innovative Experiential Learning Technique The Small Group Research Project: An Experiential Learning Approach for Undergraduate Marketing Research Students The Relationship of Cognitive Style Maps to the Preference for Experiential Learning of Undergraduate Students The Learning Style Inventory Debate Revisited: An Empirical Assessment fo the Construct Validity Issue Related to Experiential Learning Theory Conducting a Classroom to Facilitate Career Goal Attainment Corporation Executives' Ratings of Policy Learning Techniques The Generation and Application of Evaluation Criteria for Management Policy and Strategy Simulation Games Experiential Opportunities with Microcomputers Leading Students to Learning: The Teacher's Obligation An Experiment in Teaching Principles Courses: The Mini Debate Formula Developing Creative Thinking Through Experiential Learning Heuristic and Systematic Evaluation of Policy: Exercise in Decision Making A Case Study Approach for the Litigation Decision: Employing Decision Analysis to Determine When a Business Should Settle or Go to Court Learning Negotiation Skills Through Simulation Union vs. Management: A Simulation of Collective Bargaining in Action Inside the Black Box: An Analysis of Underlying Demand Functions in Contemporary Business Simulations A Review of Channel Exercises and the Description of a New Alternative SIMCON I: A computer Based Simulation Model for Evaluating Physical Distribution Strategies Involving Order Consolidation Toward Competency-Based Management Education: The Interpersonal/Communications cluster The Value of Pre-Teaching in Role Playing Some Effects of Positive Personal Reinforcement upon Socializing Students in an Experiential Learning Course Who Gains and Who Does Not from Experiential learning Toward the Ultimate Experiential Exercise, the Student View Simulating Professional Writing Experiences in the Classroom The Johari Window as a Measure of Personal Development Windows Into Management: A Participative Aid to Learning Get Your Faculty Involved A Pedagogical Approach to Business Gaming for the Commuter Student Super Service for Computer Game Administrators Expand the Role of Simulation with Creative Scenarios The Delivery, Administration, and Evaluation of an Executive Development Program Using a Total Enterprise Business Game An Application of Experiential Learning in International Trade and Foreign Direct Investment International Management: Building Bridges Analysis of a Business Simulation Exercise: Organizational Survival and Success The FALRIS Organizational Scavenger Hunt Trainee V. Trainee Subordinates' Evaluation of Experiential Learning Longitudinal Analysis of an Innovative Teaching Intervention Student Perceptions of Effective Teacher Behaviors Revisited Realism and Learning: The Evolution of a Management Game A Merger and Acquisition Simulation A Stock Market Investor Simulation Experiential Learning in Consumer Behavior: Perceptual and Attitude Change Exercises The Recycling Industry Problems of Women in Management: A Role Playing Exercise for a Course in Contemporary Organizational Problems Experimental Three Weekend Course: Empirical Results A Process for the Analysis and Development of Course Content and Instructional Methodology for Large Class Sections Problems Associated with the Assessment of Experiential learning Using the Multiple Choice Test Combining Lecture and Simulation Teaching Methodologies Qualitative Determinants of Team Performance in a Simulation Game The Effects of Different Team Sizes on Business Game Performance Comparison of Problem-Solving Technologies: A Free Simulation Approach Consistency in Business Games A computer Simulation of Personnel Selection Decisions Giving Praise Exercise Job Enrichment A Look at the Spoken and Written Word in Organizations A Pattern of Group Communication A New Generation in Business Simulation Adapting Mainframe Business Simulations to Min Computers File Access is the Essential Prerequisite to Time-Flexible and Interactive Computer Simulation A Microcomputer Simulation for Teaching Retail Location Strategy Blocks & Chips: A computer-Assisted, Geno-Typical Entrepreneurial Game Development of a Self-Paced Course in Business Statistics The Involvement of Student Bodies in the Teaching of Advanced Technical Concepts Systems Learning Sequence: An Experiential Course Module for Management Information Systems An Experiential Approach to Developing Managerial Competencies Communication Research, Inc. An Experiential Learning Activity Developed as a Practicum for a Course in Organizational Communication