ASSESSING EFFECTIVENESS OF AN EXPERIENTIAL ORIENTED COURSE OVER TIME Developments in Business Simulation and Experiential Learning, Volume 26, 1999 ASSESSING EFFECTIVENESS OF AN EXPERIENTIAL ORIENTED COURSE OVER TIME John E. Cook, SUNY Institute of Technology at Utica/Rome ABSTRACT A survey was conducted in 1998 of students who completed an experiential class between 1990 and 1997. The respondents reported that they liked the experiential format and that they learned from the class. They also agreed that they had applied concepts from the course in their subsequent life and/or work. INTRODUCTION It is always important to challenge our biases. Two significant research questions must continually be addressed regarding the learning efforts of students and teaching strategies that might assist students in their learning. A primary assumption is that learning is an internal process that can be both aided and hindered by formal teaching or training processes (Dewey, 1939). A corollary issue is that individuals have different preferences about the style of teaching they encounter, although those preferences may not aid and/or hinder learning (McKeachie, 1997; Russell, 1997). A second basic issue is that learning, being a fundamental change in behavior (Morgan & King, 1966), is difficult to assess on a short term basis although most teaching and training relies on early (“instant feedback”) assessment. At the college level a student is in a course for 15 weeks and the end of semester paperwork must be completed right away, so faculty adopt learning assessment strategies that fulfill those environmental administrative imperatives. In training situations the parameters for delivery or assessment may be more flexible although certainly not always. This report is based upon information most relevant to the college learning and assessment problems. Additionally, some assumptions are important in setting the framework for this discussion. The first is that students “like” the approach to “school” that is called experiential learning, defined for these purposes as involving “classroom” (more a concept than a physical description) exercises, activities, and case studies done both as individuals and in small groups (Keys & Wolfe, 1988). Such a teaching model has been popular in some colleges and courses for many years (Johnson, Johnson & Smith, 1991) and is currently enjoying increased emphasis in public schools under a general name of cooperative learning (Slavin, 1983). The adoption of such a teaching strategy for a course on organization behavior is somewhat easier than in other topical areas because it is a “people” oriented course. Exercises and cases involving behavior fit very well into the theoretical framework of such an offering. Because it is popular, there are numerous resources (texts, cases, films, exercises, etc.) available to the instructor for use in such a course. Since it is easy to adopt an experiential teaching strategy, that decision should be regularly challenged as to whether it fulfills the basic purpose of enhancing learning. The next assumption is that the assessment of learning in such a course requires a longer time frame than the typical college semester. It may be impossible to judge just exactly how long that time frame must be in order to assess any changes in behavior associated with a particular learning experience. The typical end-of- course student survey is probably not far enough removed to measure learning (Cornwall & Manfredo, 1994). Such surveys likely measure student attitude as to whether they think they learned and whether the style of the class or instructor fit their personal learning style preferences (Daft, 1995). The Setting A graduate course in organization behavior has been scheduled on Friday afternoon during the Fall semester for about ten years. The original purpose of the course was as a support for a then new major in Nursing Administration. The MS in Nursing was planned as a weekend program and the Friday offering through the School of Business fit into the overall curriculum plan. The author has taught the course since the Fall of 1990. Although the first section was almost entirely nursing students, the course has subsequently evolved as a popular elective for the MS in Business Management. Enrollments in the last five years have been approximately two-thirds business students enrolled as an elective and one-third nursing students completing a program requirement. The nursing students generally are employed full-time and are 160 Developments in Business Simulation and Experiential Learning, Volume 26, 1999 completing their program on a part-time basis. The MS in Business Management is a part-time program scheduled for students who are employed full-time, but in the last five years approximately 20% of the students enrolled are full-time students, some traditional and some returning students displaced by the changing economy near the campus. Business students selecting the Friday afternoon class as an elective are somewhat more likely to be full-time students, but not necessarily traditional aged students. The content of the course has evolved from approximately 70% discussion and lecture and 30% exercises in the 1990 offering to one in the recent five years that is approximately 65% exercises, activities, and group problem solving and about 35% lecture and discussion. The activities offered are, for example, in- school research exercises such as what is the most popular color of socks (white), autograph bingo, problem solving exercises, role playing, or completion of common research exercises such as the Fiedler LPC questionnaire. More exercises and activities would likely be a preferable learning format. McKeachie (1997) reports better retention, thinking and motivational effects when students are more actively involved in talking, writing and doing. The current mixture seems to satisfice for most students. It is not so non-traditional that they feel threatened by the teaching/learning strategy. As an illustration, two examinations are scheduled, mid semester and end of semester, and both are open resource exams with a three hour time limit. The examinations include individual exercises and activities (a procedure to use a group activity has not yet been devised). An essay question or two is also included and the student needs to complete four of five questions/activities offered. Any essay question usually has been a popular choice and based on student comments a section of 25 multiple choice questions is offered as fulfilling one of the four test activities. It is very rare (perhaps 4 of 100) that a student does not complete the multiple choice section and usually the essay question and then case study or activity questions are almost always least popular choices. Given choices, students appear to “vote” for the more traditional test questions with lower levels of ambiguity regarding the “right” answer. The educational purpose for using such ambiguous techniques is to better prepare students to function in turbulent environments. The Study As an assessment by the author, all students who have completed the course from 1990 through 1997 were surveyed in late Spring 1998. That provided a minimum of six months life experience, or more for most, after the completion of the course. It allowed a large majority of the respondents several years to reflect upon whether learning occurred through the completion of such a course. The goals of the survey were to determine whether some of the major course objectives (see Exhibit One) were met and have stood the test of the passage of time. Secondly, to determine whether the basic teaching strategies emphasizing experiential learning were effective and whether students found utility in completing the course. And finally, whether the instructor was perceived by the students as effective. This is a report based upon the results of that survey. EXHIBIT ONE Course objectives (as stated in class handout): Organization/Management has three broad course objectives: 1) explore the relationship of managing people and the concepts of management within various types of organizations, 2) expand your understanding of individual behavior within the context of diverse organizations, and 3) provide a menu of possible approaches, protocols, or solutions for dysfunctioning organizations. Final grade rosters were used as the basis for the list of students completing the course. One hundred thirty students were on the list and the college student/alumni data base was utilized to prepare a mailing using the last known address. The mailing consisted of a letter reminding the former students of their participation in such a course and requesting their cooperation in completing a brief two page questionnaire. Also included was a postage paid business reply envelope. Fifty six completed questionnaires were received and based on a revised population of 123, the return rate is 45.5%. Ten mailings were returned by the Postal Service, three were remailed with updated information and seven mailings were undeliverable. The student major was known from the final grade rosters so the research plan was to use a chi square analysis to 161 Developments in Business Simulation and Experiential Learning, Volume 26, 1999 compare the distribution of the returns by major to the population distribution. The population had a distribution of 43% Nursing majors and 55% Business majors. The returned questionnaires have a distribution of 43% reporting their major as Nursing and 55% reporting their major as Business. The sample is representative of the population. The first page of the questionnaire was comprised of eight statements with a forced choice response scale for each statement. The choices were strongly disagree, disagree, agree, and strongly agree. No neutral choice was offered so that respondents had to either agree or disagree with the statement. All of the respondents generally made those choices with only 5 items out of 448 possible responses either left blank or a neutral choice written in for a compliance rate of almost 99%. All but one of the statements were designed to elicit a planned affirmative response. One item was worded in such a way that the desired response would be negative (disagree) to help break any pattern of automatically selecting a response choice without thinking about the impact of the selection. It would appear that the responses follow the expected patterns. Responses were rated 1 for strongly disagree to 4 for strongly agree. All of the items designed to elicit an affirmative response had a weighted average more than the 3.0 that would represent the “agree” choice. The lowest overall weighted average for the seven statements designed to be affirmative was 3.3. In dividing the responses by majors for Business and Nursing, the lowest weighted average was 3.2. The item designed for a “disagree” response had a 2.0 weighted average, overall and for both the business and the nursing responses. DISCUSSION The easiest result to discuss is the statement designed to elicit a disagree response regarding the need to include more lecture materials and instructor centered activity. The responses were uniformly in disagreement with including more traditional teaching activities with a weighted average of 2.0 (disagree). Even so, three business majors strongly agreed and three nursing majors agreed that more lecture would have been better for the course. Based upon these results as well as other research conducted with students at this college regarding the use of technology in teaching, from 10% to 20% of students will be dissatisfied with any major teaching strategy employed (Cook, 1998). Russell (1997) reports that “…students are not alike and that individual differences/learning styles determine that technology will improve learning for some but will probably diminish learning for others, while some will experience no significant difference.” The majority of respondents opted to strongly support the use of experiential learning in the form of exercises and activities (Item 3 in Table One) with an overall weighted average response of 3.5 (out of 4.0). Additionally, they supported the idea that a strong emphasis on such techniques enhanced their learning (Item 4 in Table One) with a 3.4 weighted mean. Scriven (1994) affirms that students (here former students as respondents) can rate their own gains in knowledge. In a separate open ended comments section (see Table Two), the largest number of comments regarded the enhanced learning aspects of the course with 13 of 54 total comments (24%). And finally, (Item 8 in Table One) most respondents validated the learning by reporting that they had used the course materials in their work or life since completing the course. In the comments section, 7 of 54 comments (13%) also mentioned the specific applicability of the course materials to work or life. Although almost all respondents supported the experiential learning nature of the course, business students and nursing students responded somewhat differently in their emphasis. Business students liked the experiential techniques (Item 3 Table One) the most and felt that the exercises enhanced their learning (Item 4 Table One). In addition, they strongly remembered that there was a lot of interaction among students and with the instructor (Item 1 Table One). Nursing respondents in contrast were more enthusiastic regarding the theoretical framework provided by the text (Item 7 Table One) and the technique of using open resource examinations as a specific learning activity (Item 5 Table One) rather than as just an assessment tool. Nursing respondents also reported that they have applied the materials and concepts in their work or lives (Item 8 Table One & Table Two). Two summarizing questions were included, one an overall rating of the class and the other an overall rating of the instructor. Such global ratings have been judged as an effective means for student feedback (d’Appollonia & Abrami, 1997). A ten point scale was offered with one being lowest and 162 Developments in Business Simulation and Experiential Learning, Volume 26, 1999 ten being highest. The course achieved an overall rating of 7.9, with business respondents rating at 8.4 and nursing respondents at 7.2. All business respondents rated the course in the upper half of the scale whereas 31% (eight) of the nursing respondents rated the course as 5 or below. The overall instructor rating was 8.3 with business respondents at 8.7 and nursing respondents at 7.8. Five nursing and one business respondents rated at 5 or less. These are consistent with the general results that about 10% of the students would probably have preferred a different course approach. TABLE ONE WEIGHTED MEANS FOR STATEMENTS BY MAJOR Item Statement highlight Overall Business Nursing .1 ‘participant interaction’ 3.3 3.5 3.2 2 ‘enhanced people skills’ 3.3 3.3 3.3 3 ‘liked experiential techniques’ 3.5 3.6 3.3 4 ‘exercises enhanced learning’ 3.4 3.5 3.2 5 ‘exams a learning experience’ 3.3 3.2 3.4 6 ‘needed more lecture’ 2.0 2.0 2.0 7 ‘text provided theory’ 3.3 3.2 3.4 8 ‘have applied course materials’ 3.3 3.3 3.4 TABLE TWO CONTENT ANALYSIS OF OPEN ENDED COMMENTS Comment essence Total Business Nursing learned 13 5 8 enjoyable 11 5 6 Hello 10 7 3 applicable to work or life 7 2 5 professor dominated 3 2 1 stimulating 2 2 2 liked text 2 0 2 needs higher level materials 2 0 2 update AV or technology 2 0 2 style needs mature behavior 1 1 0 sometimes boring 1 1 0 CONCLUSIONS Business students appear to be more satisfied with a course or an instructor using experiential classroom techniques than are nursing students. That result might well be altered if the parameters of major were reversed where the nursing students were electors and the business students were required to complete the course. Coburn (1984) as well as Marsh and Roche (1997) reported that research supports the belief that students who are required to take a course rate it lower than students who elect to take the same course. Goldman (1990) and Marsh and Roche (1997) suggest that students will rate courses in their major field (“prior subject interest”) more highly than supporting courses outside their field and Cashin (1990) partially supports that opinion. The varied nature of the students may also play a role in the overall assessments with part-time students more anxious to get to the “point” and the experiential approach may appear more tangential and time consuming. 163 Developments in Business Simulation and Experiential Learning, Volume 26, 1999 No matter what teaching strategy is designed, no “game plan” will be totally accepted and provision must be made for those with alternatively preferred learning styles to succeed. Some of the detailed comments from the course proponents, however, indicated significant recall of the materials and techniques used and powerful witness that they had learned. REFERENCES Cashin, W.E. (1990, Sep 5). The use of student evaluations of faculty members. The Chronicle of Higher Education, p. B3 Cohen, L. (1984). Student evaluations of teacher performance, ERIC Document Reproduction Service, Document No ED289887 Cook, J.E. (1998). Trend review of educational technology: Impressions from students and faculty. in J. Paknejad (Ed.),Proceedings of the Northeast Decision Sciences Institute (p. 59). Providence, RI. Cornwall, J.M. & Manfredo, P.A. (1994).Kolb’s learning style theory revisited. Educations and Psychological Measurement., 54, 317-327. d’Apollonia, S. & Abrami, P.C. (1997). Navigating student ratings of instruction. American Psychologist, 52(11) 1198-1208 Daft, R.L. (1995). Disaster in Commerce 353. Journal of Management Education, 19(1) 17-30 Dewey, J. (1939). Experience and education. New York: The Macmillan Company Goldman, L. (1990, Aug 8). Student evaluations of their professors rarely provide a fair measure of teaching ability. The Chronicle of Higher Education, p. B2 Johnson, D.W, Johnson, R.T. & Smith, K.A. (1991). Cooperative learning: Increasing college faculty instructional productivity. ASHE-ERIC Higher Education Report No. 4. Washington, DC: The George Washington University, School of Education and Human Development. Keys, J.B. & Wolfe, J. (1988). Management education and development: Current issues and emerging trends. Journal of Management ,14(2), 205-229. Marsh, H.W. & Roche, L.A. (1997). Making students’ evaluations of teaching effective-ness effective. American Psychologist, 52(11), 1187-1197. McKeachie, W.J. (1997). Student ratings:The validity of use. American Psychologist, 52(11), 1218- 1225. Morgan, C.T. & King, R.A. (1966). Introduction to psychology. McGraw-Hill, p73 Russell, T.L. (1997). Explaining, exploring, understanding the no significant difference phenomenon. Adult Assessment Forum, VII,(4), p. 6. Scriven, M (1995). Student ratings offer useful input to teacher evaluations. ERIC/AE Digest, Document No ED398240. Slaven, R.E. (1983). Experience and education. New York: Longman. 164 Table of Contents Volume 26, 1999 ABSEL's Historical Research Interests Back From the Future: An ABSEL Merlin Exercise for the Year 2005 The Contributions of ABSEL During the 1980's ABSEL's Contributions to Experiential Exercises in the 90's ABSEL's Contributions to Experiential learning/Experiential Exercises: The Decade of the 1970's Business Simulations - Algorithms and Model Enhancements: A 25 year Review A Study of the ETS General Field Test as an AACSB Assessment Tool and the Impact of Experiential Exercises and Simulation on Learning A Framework for Assessing the Competencies Reflected in Simulation Performance Developing A Learning Culture: Assessing Changes in Student Performance and Perception Financial Simulation Using Distributed Computing Technology Analyzing Managers' Judgements and Decisions with an Educational Business Simulation Understanding Your Business through Home-Made Simulator Development An Examination of a Reanalysis of the Impact of a Market Leader on Simulation Competitors' Strategies Applying Shocks to TE Simulations: A Demonstration Increasing Efficiency of Management Skill Assessment A Testbank for Measuring Total Enterprise Simulation Learning Developing Leadership Skills - Video Live! LEADSIMM: Collaborative Leadership Development for the Knowledge Society A Team Approach to Producing Multi-Media Laptop and Video Formatted Presentation Tinkertoys Revisited: Exploring Trust Based Relationships Overall Dominance in Total Enterprise Simulation Performance Success or Bankruptcy: The Relationship between Personal and Goal Orientation and Simulation Performance Assessing the Effects of Feedback: Muti-method and Muti-directions in Multi-pedagogical Courses The Missing Ingredients in Experiential Learning Purpose and Learning Benefits of Business Simulations: A Design and Development Perspective Building Capabilities for Change through Laboratory Simulations Modeling Innovation as a Process of Design in Educational Business Simulation The Need to Measurer variance in Experiential Learning and a New Statistic to do so Assessing Effectiveness of an Experiential Oriented Course Over Time Developing Participant Satisfaction Models of Experiential Exercises in Business Education Perceptions of Learning in TE Simulations Students' View on the Use of Business Gaming in Hong Kong Management Gaming's Lost Opportunity: Meditations about the Russian Experience and Prospects for the Future Unanticipated Enhancements in the Business Strategy and Policy Game when Running in Windows 95 Using a Business Game to Demonstrate Broad Business Concepts A New Model for Business Courses (Getting the Student Connected) Seeing the Forest and the Trees: Integrating Knowledge Using Large Scale Simulations in Capstone Business Strategy Classes Is It Here To Stay? A Roundtable Discussion of the Inter-Group Interaction Interactive Tools Used in Applying Financial Concepts Strengthening Essential Skills through a Finance Exercise: Calculation of Beta A Model of Currency Exchange Rates Understanding Currency Exchange Rates: A three-part Exercise Student Experiences in the World Intercollegiate Business Game Competition Student Experiences in the International-Collegiate Business Policy Game Competition Using Boards of Directors in Simulation Environments: Comments From Board Members Sharing Best Practices: Teaching Smarter, Not Harder Star Power: A Simulation for Understanding Power and Empowerment The Marketing Game: A Marketing Principles Simulation Alexander Islands: GSSM Tiny Business Simulator on the WWW Putting Strategy into Strategic Business Games Industry Analysis, Porter's Five Forces Model and Strategic Group Maps in the Business Strategy Game Simulation The Use of Concept Mapping to Improve Student Performance and Understanding of Strategic Management Concepts: A Comparison of Techniques Transformational thinking in the Organizational Behavior Course: The Use of Metaphor as an Assessment Tool Creation of a Virtual Learning Community for the Global Virtual Enterprise Project Inter Institutional use of Educational Resources: Joint Use of Management Simulation Games Business Insights: Theory and Practice with the Aid of a Business Simulation Progress: An Experiential Exercise in Development Marketing The ABC's of teaching the Theory of Constraints to Undergraduate Business Students Demonstrating Principles of Organizational Purchasing Behavior through an Experiential Exercise The Virtual Manager: A Different Simulation for Managing Complexity When the Rules are changing and Chaos Breeds Innovation: Recapturing the Value of Constructive Thinking and Play in Simulation Training The Pitfalls, and Potential, of Actual Events as Problem Drivers Using Business Games to teach Environmental Awareness and Green Management: The International Experience with the ENSIM Game A Review of my ABSEL-Related Work Simulation of Government: A Workshop Using GEO Creating Internet-Based Games Using Perl and JavaScript Who's on First? Exploring the Concepts of Problem-Based Learning, Experiential Learning, and Lifelong Learning Students Learn Customer Service and Selling while Conducting Research So You Want to Run an NFL Football Team–An Honors Interdisciplinary Project Supervised Internship: The Employer's Perspective Using Computer Assisted Simulation to Teach International Business Strategy: A Case of the Multinational Management Game (MMG) Multi-Cultural Experiential Learning: A Computer Simulation in China An Appreciative Stance on Diversity as We Move into the 21st Century: A Timeline Exercise to Identify Key Experiences in Good Work Relationships between Black and White Peers A Day in the Life of an Interactive, Real Time, and Internet Delivered Course: A Demonstration Comparing Internet Search Engines: An Experiential Learning Exercise Total Enterprise Simulations and the Internet: Assessing Student Perceptions and Preferences Teaching Accounting Information Systems in a Practicum Format Providing Experiential Learning in Accounting through a Field Study Payroll Project A Spreadsheet Based Business Simulation Game Computer-Behavioral Simulations Training for Project Managers Simulation Scenarios - Rationale and Illustration