CHANGING THE ASSESSMENT PARADIGM: USING STUDENT PORTFOLIOS TO ASSESS LEARNING FROM SIMULATIONS Developments in Business Simulation & Experiential Learning, Volume 27, 2000 172 CHANGING THE ASSESSMENT PARADIGM: USING STUDENT PORTFOLIOS TO ASSESS LEARNING FROM SIMULATIONS Katrina A. Zalatan, Hartwick College ABSTRACT Evaluating the complex mix of learning from computer-based simulations is a significant challenge. Most faculty employ traditional, teacher-centered assessment methods. In 1996, we changed our management curriculum from traditional lecture-based courses to a sequence of simulation-based courses, with simulation- related assessment accounting for 60 to 90% of students' course grades. Our evaluation methods, however, continued to be traditional. In 1998, we added electronic portfolios to every junior course for 20 to 30% of each course grade. Portfolios enabled students to demonstrate complex competencies learned from simulations regardless of their results. Students perceived increased engagement, increased responsibility, and better preparation for job seeking. Instructors gained more complete and more authentic information to assess student competence, and they observed increased reflection—a skill critical for lifelong learning and professional growth. Despite increased time and effort, both students and faculty overwhelmingly endorsed continuing course portfolios as part of the junior curriculum, expanding their use into the senior curriculum, and supporting the development of career portfolios. BACKGROUND Simulation games can engage students in the discovery of many business competencies. Unlike traditional, teacher-centered pedagogy, simulations make students responsible for analysis, problem solving, and decision-making, often within the real-world context of teams. When married with reflective observation, this active experimentation can empower students to create their own new knowledge (Kolb, 1984). Material has a greater chance of being learned and retained, and students find simulations more interesting than traditional classroom instruction (Randel, Morris, Wetzel & Whitehall, 1992). Higher motivation, effort, and levels of learning have also resulted from computer simulations than from other forms of group projects (Tompson & Tompson, 1995). In 1996, we changed our management curriculum to capitalize on these pedagogical benefits. For many years, the major consisted of traditional, lecture-based courses about separate management functions (see Figure 1). The new curriculum was an innovative sequence of seven cross-functional courses that were each designed around a simulation to create an "active learning culture" (Zalatan & Mayer, 1999). In each course, students collaborated in company teams to compete against each other for up to two simulated years. While students still learned basic concepts through traditional instruction, simulations enabled students to discover advanced concepts and functional interrelationships. NEED FOR CHANGE Interestingly, while our pedagogy changed dramatically, our approach to assessing learning did not. Our simulation-related assessment typically included peer evaluation, student participation, team performance versus other teams, written plans, presentations, and frequently, final "case" exams similar to the simulation. Individual student reflection and self-assessment were not emphasized. In this respect, our approach to evaluation appeared consistent with other faculty using simulations (Anderson & Lawton, 1992). While the Anderson & Lawton study indicated that others counted simulations as 30% of the total course Developments in Business Simulation & Experiential Learning, Volume 27, 2000 grade on average, however, ours accounted for 60 to 90% of the final course grade. Our evaluation m capture the complet in our simulation-b greater motivation a assessment did n changes. Evaluatio results (especially not necessarily m students' learning. In to have little if any their learning, and learning more, the scope and significa improve commensur A POTEN As we investigat methods for 199 framed our inquir methods cause all students to reflect, discover and integrate critical concepts throughout the simulation? Could they increase our students' awareness of their learning and help them better articulate these gains? And finally, could any methods help our students see every simulation—regardless of results—as part of an CURRICULUM COM “Old Curriculum” (8 course units, junior/senio courses) Technical Courses (3 courses) - Marketing Management - Financial Management I - Operations Policy Courses (2 courses): - Case Studies in Leadership - Policy Formation & Administration Additional Management Cour course), either: - Leadership (Outward Bound - Internship Additional Economics Course course), either: - Microeconomic Theory - The International Economy - Comparative Economic Sys - Banking and Financial Mar - Macroeconomic Theory Senior Thesis (1 course unit) FIGURE 1: PARISON (Zalatan & Mayer, 1999) r “New Curriculum” (8 course units, junior/senior sequence) : MGMT I - BRANDS: A ethods, however, did not e, complex mix of learning ased courses. We observed nd critical thinking, but our ot always capture these ns also tended to focus on winning results) which did easure the full extent of dividual students continued responsibility for assessing while they appeared to be ir ability to articulate the nce of this learning did not ately. TIAL SOLUTION ed alternative assessment 8-1999, several questions y. Could any assessment ongoing, lifelong learning process? se (1 ) (1 tems kets Marketing Game (Chapman, 1994) MGMT II - AIRLINE: A Strategic Management Simulation (Smith & Golden, 1994a) MGMT III - PROSIM III: A Production Management Simulation (Chu, Hottenstein & Greenlaw, 1996) MGMT IV- Human Resources Management Simulation (Smith & Golden, 1994b) MGMT V - PC FinGame: The Financial Management Decision Game (Brooks, 1996) MGMT VI - The Multinational Management Game (Keys, Edge & Wells, 1994) MGMT VII - The Business Strategy Game (Thompson & Stappenbeck, 1997) MGMT VIII - Senior Thesis Recent assessment literature indicated that programs ranging from teacher to dentist preparation used "portfolios" to evaluate student performance in the latter parts of their pre- professional education (Borko, Michalec, Timmons & Siddle, 1997; Chambers & Glassman, 1997; Snyder, Lippincott & Bower, 1998). Unlike traditional methods, portfolios gave students the responsibility to collect and present evidence from their educational experiences to demonstrate their achievement (Freeman & Lewis, 1998). Given the realism of our simulations and the student engagement they foster, portfolios seemed like an excellent assessment option. They also seemed consistent with our "active learning culture." The literature also presented other benefits: 1. Portfolios foster the reflection needed for learning from experience (Borko, et al., 1997; Freeman & Lewis, 1998). Portfolios are vehicles through which students could integrate concepts (Magill & Herden, 1998), connect experience with theory (Borko, et al., 1997), and increase their motivation to learn from mistakes (Snyder, et al., 1998). Reflecting about learning in terms that are meaningful to them could also help students "legitimize" and transfer their learning (Burns & Gentry, 1998) and prepare them for lifelong learning as professionals (Borko, et al., 1997; Snyder, et al., 1998). 2. Portfolios engage students in the process of evaluating their own learning, giving them a stake in the process and encouraging them to assume more responsibility for it (Mullin, 1998). Portfolios foster active engagement in the pursuit of both the teachers' and their own 173 Developments in Business Simulation & Experiential Learning, Volume 27, 2000 personal learning goals (Murphy, 1997; Mullin, 1998), and they give students freedom to use their own initiative to demonstrate their learning (Freeman & Lewis, 1998). On a broader level, portfolios can increase students' belief in the importance of looking critically at themselves and the skills that enable that to happen (Snyder, et al., 1997). Students can also learn that not all feedback and reward must come from external sources as it did at the beginning of their educational career (Chambers & Glassman, 1997). 3. Portfolios can help students articulate their learning in courses, and eventually, in job interviews (Freeman & Lewis, 1998; Magill & Herden, 1998). Portfolios leave it up to the student to assemble evidence and argue his or her case for competence (Chambers & Glassman, 1997), and through this process, students learn to present themselves and their work to others (Murphy, 1998). 4. Portfolios can provide faculty with more complete, detailed information for assessment (Freeman & Lewis, 1998). Students and faculty work collaboratively (Mullin, 1998), and this increases teachers' ability to "know" each student as an individual (Murphy, 1998). OUR PORTFOLIO PILOT (1998-1999) In the fall of 1998, we piloted portfolios as a requirement for all junior management courses (MANA I, II, III, and IV). We limited change to junior courses so we could adopt alternative assessment slowly and improve our chances of success (Anderson, 1998). Two instructors were involved, each teaching two sections of their two junior courses (20 to 24 students per section). PILOT PROCESS: The instructors attended relevant conferences and workshops to increase their familiarity with portfolio methods, and they collaborated to design and implement the following pilot process: 1. Competency Area Lists - Brief lists of course and curricular competency areas were presented to all students on the first day of each course. Course competency areas included critical understandings that students should learn during each simulation (see Figure 2). Students were expected to use portfolios and evidence from the course to demonstrate their achievement, and as such, course competencies served as a framework to support discovery. FIGURE 2: COURSE COMPETENCY AREA EXCERPTS (MANA I, Fall 1998) 1. Definition of marketing. 2. Philosophies of marketing. 3. Marketing Management Decision Making Process: a. ANALYSIS - Market situation analysis and SWOT (1) Customers (a) Needs/wants… b. PLANNING (1) Target market and positioning… c. IMPLEMENTATION (including “approach” to decision making) d. EVALUATION (1) Vertical and Horizontal analysis… While each course had a functional emphasis, competencies from other disciplines were also included. In MANA I, for example, competencies were drawn from formerly separate marketing, economics, and accounting courses. Competency lists were organized using the management model taught in introductory courses (Stoner, Freeman & Gilbert, 1995). Curricular competency areas were relevant to all courses (see Figure 3). Students used course portfolios to demonstrate their growth in these areas, comparing and relating experiences between (not just within) courses. Both course and curricular competency areas were left intentionally broad, so students could develop meaning for these competencies throughout their courses. 174 Developments in Business Simulation & Experiential Learning, Volume 27, 2000 175 FIGURE 3: CURRICULUM COMPETENCY AREA EXCERPT 2. Technology Training and Support - Course portfolio media were left to the discretion of instructors. While students had the option of submitting portfolios on electronic media (floppy disk, zip disk, or compact disk) in every course, they were required to do so in MANA I and IV, using Netscape Communicator as their navigational application. All students participated in Netscape Communicator training conducted by library staff outside of regular class hours, and all had access to hardcopy and web-based self-instruction for scanning, CD burning, and Netscape. Students upgraded their college-issued laptop PCs to include Netscape Communicator, and they had access to necessary peripherals (e.g., zip drives, CD burners, scanners, etc.) at either labs or the library reserve. Our department lab was also upgraded to support this project. Library staff were available for consultation throughout the semester, including the "crunch weeks" when portfolios were due. 3. Prescribed Portfolio Organization - While students had the freedom to choose what evidence supported their claims of competence, they were required to use a prescribed format for their portfolio in each course. Faculty designed these formats to ensure enough consistency between students to make assessment less time-consuming (Freeman & Lewis, 1998), and they presented portfolio grading criteria to students when the format was reviewed. Faculty were careful to balance "guidance and structure" with "flexibility and choice," however, as too much structure could lead to a lack of students' personal ownership (Borko, et al., 1997). When electronic portfolios were required, portfolio formats and tutorials were available via course web sites. 1. Critical thinking (about management and about yourself as a manager and a learner) 2. Communication a. Written b. Oral 3. Teamwork a. Specific goals b. Common approach c. Meaningful purpose… 4. Preliminary (Formative) Submission - In each course, students submitted preliminary portfolios after they accumulated enough experience about which to reflect, typically about halfway through the simulation. All were required to be individual efforts. While portfolio assignments varied between instructors, all engaged students in selecting, presenting, and reflecting about experiences to demonstrate competence. In MANA I and IV, for example, students reflected about four examples of their experience that, together, demonstrated their achievement of the course competency areas. In MANA II and III, students were directly asked to demonstrate understanding of pivotal simulation concepts at instructor-identified points during the game, with all competency areas addressed by the end of the course. Portfolios in all courses included reflective narratives and examples of students' simulation- related work. Instructors provided feedback to shift students' focus from "assessment for accountability" to "assessment for improvement" (Cross, 1998) and to enhance learning (Murphy, 1998). Faculty also provided coaching to help each student reflect about and articulate his or her learning. 5. Final (Summative) Submission - Final portfolio submissions were due on the last day of each course, typically two weeks after the last simulation decision period. They were returned after the course, and instructors provided feedback to help students understand that even summative works are formative parts of their larger curricular portfolio. 6. Grading - In the fall, portfolios in MANA I and II accounted for 20% of each final course grade. By the spring, the weight for portfolios increased to 30% of the course grade for Developments in Business Simulation & Experiential Learning, Volume 27, 2000 MANA III and IV. This increase paralleled the increase in responsibility that students were taking on for their own learning and their increasing ability to do reflective self- assessment. PILOT EVALUATION: As a department, faculty shared and discussed student reactions, student work, and instructor reactions to workload and process variations. Since student perceptions were also critical, we administered a Portfolio Project Survey ("Survey") on the last day of spring classes to all participating students (N’46). The survey was anonymous, and it consisted of three sections: 1. Technology - eight items of various formats that measured self-reported use of various technology resources and students' satisfaction with them. 2. General Perceptions - seven questions on a five-point scale (1 ‘ strongly disagree to 5 ‘ strongly agree) that collected perceptions about pilot objectives and assumptions. 3. Comments and Suggestions - three open- ended questions that asked for improvement suggestions, if junior portfolios should be continued, and for expectations about portfolios in their senior year (1999-2000). RESULTS We observed several anticipated results: 1. Portfolios did develop reflection among the majority of students. Since portfolios were not team projects, all students were required to reflect. Interestingly, many students reflected about their learning from the challenges and failures they experienced during the simulation, and they appeared to appreciate the opportunity to treat these events as positive learning experiences. As the year progressed, we observed improvement in students' reflective works. Students earned higher portfolio grades in all spring sections. Students' responses to the Survey also strongly supported that portfolios fostered reflection: 91% agreed that portfolios increased their awareness of what they learned and gained from each course (item mean ‘ 4.43 out of a possible 5); 89% agreed that portfolios provided an opportunity for them to reflect about their development as managers (mean ‘ 4.35); and 83% agreed that portfolios helped improve their understanding of course experiences (mean ‘ 4.20). Overall, both electronic and paper portfolios appeared highly valued despite the extra work required, clearly supporting our shift to an "active learning culture" (Zalatan & Mayer, 1999). Of the 36 students (78% of total survey respondents) who answered the open-response question "Should we continue to use portfolios to assess student learning in our junior courses next year?," all students responded affirmatively. The majority (70%) of the 27 who wrote explanatory comments supported the use of portfolios because of the reflection they fostered. Students' rationale included that the portfolio "enables us to reflect and comprehend," and it "[made it] able to truly convey what you have learned." 2. Students became more involved in assessing their own learning. In the Survey, 91% of the respondents agreed that portfolios gave them the opportunity to assess their own learning in each course (mean ‘ 4.37). Given the strong realism of each simulation, students appeared to perceive authenticity, and they took both simulations and portfolios very seriously. Students also appeared to be motivated by the direct link they perceived between classroom learning and future job requirements (competencies). 3. Students felt more prepared for job seeking. Students perceived that their course portfolios would eventually contribute to career portfolios. Approximately 41% of the 27 Survey respondents who agreed that portfolios should continue to be used gave reasons related to "being able to better convey what you learn" and "being prepared for the future." Also, 81% of all survey respondents agreed that portfolios 176 Developments in Business Simulation & Experiential Learning, Volume 27, 2000 will be worthwhile in job interviews (mean ‘ 4.03). Other findings pertained to the pilot instructors and portfolio-related technology: 1. Instructors worked harder, but they also had more information to coach teams and assess individual students. Portfolios gave instructors detailed insights into each student's learning and self-assessment, and they helped instructors discern the extent to which individuals were really involved in their teams. They also gave instructors more detail to facilitate students' discovery in the simulation. Instructors perceived that they worked harder, especially in the formative submission stage of the process, but the benefits outweighed the extra work. The time to grade final portfolios in all courses appeared comparable to the time required to grade previously used case exams. 2. While each instructor had slightly different approaches to portfolio assignments, most students appeared to accept these differences. Only 13% of the Survey respondents suggested increased consistency. Instructors found that their approaches converged naturally as the year progressed, and increased consistency is expected in assignments and technology in 1999-2000. 3. Most students found content more arduous than technology. Most students seemed to accept the need to produce, store, and communicate their work electronically, and some even saw technology and visual communication skills as a competitive edge for future employment. Even though electronic portfolios were not required in all courses, the Survey revealed that, on average, students stored 79% of their management course work electronically. In courses requiring electronic portfolios, those few who reported problems with the technology also submitted poorer portfolios overall. 4. Students used and were satisfied with the technical support available outside of class. For those courses requiring electronic portfolios, the Survey indicated that 61% of all students used the portfolio tutorial available on their class web sites, with an average satisfaction rating of 3.89. Students suggested adding actual example portfolios to the sites, and this will be possible during our second year of implementation. A reported 72% and 67% of students used the instructional documentation for Netscape and scanning, respectively, and student satisfaction with both were also good (mean ratings of 4.24 and 4.16, respectively). Students were also highly satisfied with technical support (mean rating ‘ 4.60), though we were pleasantly surprised to learn that 20% used "friends" as their primary technical resource. 5. Of the electronic portfolio media available, zip disks were the most utilized, and write-once compact disks (CDs) were the least utilized. Students appeared to prefer the ease and convenience of editing zip disks, even though most students had to use labs or equipment on reserve to use zip disks with their laptops. Storage capacity was critical when portfolios included multiple, simulation-related scanned images. Very few students used CDs even though they were an inexpensive and writers, self-instruction, and technical support were available. We believe this happened because the CDs could not be edited, and students were generally unfamiliar with using CDs for storage. CONCLUSIONS Overall, the results of our portfolio pilot were positive, encouraging, and strongly in support of the "active learning culture" so critical to our simulation-based management curriculum. While portfolios did require more time and effort from faculty and students, we believe that portfolios raised the quality and totality of students' learning experiences from simulations—raising reflection, learning, and assessment to levels not achieved with 177 Developments in Business Simulation & Experiential Learning, Volume 27, 2000 traditional evaluation methods. Portfolios also improved our rapport with students. Since the pilot was well received by students and faculty, we will continue to use portfolios in the junior curriculum and expand their use into the senior curriculum in 1999-2000. We are also coordinating with our career center to help students extend course portfolios into career portfolios. As we move ahead, our continued success will be dependent on several factors including continued technological support, faculty commitment and collaboration, and reasonable class sizes (20 to 24 students). ACKNOWLEDGEMENTS I would like to extend sincere thanks to my colleague Doug Mayer for his partnership in this pilot and his thoughtful suggestions pertaining to this manuscript. I would also like to thank Margaret Schramm, Professor of English and Hewlett Project Coordinator, and Marilyn Dunn, College Librarian and Director of Information Resources, for their support. This project was funded in part by a grant from the William and Flora Hewlett Foundation. References Available Upon Request 178 Table of Contents Volume 27, 2000 Internet International: A Simulation Exercise for Understanding Technological Innovation and customer Service In a Rapidly Growing Internet Server Company Simulations and Learning: Dialog and Directions Endnote Activity: A Tool for Integration of Course Content and Communication Skill Practice Incorporating Video as a Teaching Strategy in Interpersonal Communication Vision Quest: An Alternative Approach to Industry Analysis for MBA Courses in Strategic Strategic Management: An Evaluation of the Use of Three Learning Methods Trainer, Mentor, Educator: What Role for the College Business Instructor in the Next Century? Using the Internet and Shareware to Facilitate Computer Simulation in Distance Learning Classes Visual Modeling of Business Simulations Teaching about Information with Management Games A Self-Evaluation Based on the Discussion and Decision in Experts' Business Gaming The Restaurant Game Using Journals to Enhance Computer Simulation Based Learning Exercises to Facilitate Better Student Writing in the Undergraduate Strategy Class Identifying, Resolving, and Managing Common Ethical Dilemmas in the Workplace: An Experiential Approach Integrating the Digital Revolution into the Classroom The Wheel of Learning: An Integrative Business Curriculum Experiment The Changing Nature of Simulation Research: A Brief ABSEL History Perspectives on Simulation & Gaming's Review Process Experiential Learning Across Disciplines: Mixing International Business and Accounting Simulating Governmental Effects on Economic Development Internationalizing the Introduction to Business Course Using an International Text and Domestic Simulation with a Twist Using Stock Value as the Performance Measure in a Business Simulation Game Introducing Cross-Elasticities in Demand Algorithms Validating a Model of Currency Valuation An Exercise for Exploring the Relationship between Jungian Psychological Types and Organizational Politics Exercise: Preparing Financial Reports Using the Group Categorizing Technique Effect of Trust and Cultural Beliefs on Negotiation Processes: Data from an Experiential Role Play Experiential Learning Gets Stamp of Approval From the Boyer Commission Talent Search 2000 - An Experiential Activity to Help Strengthen Skills in Employee Recruitment and Selection Clemson University's Collaborative Learning Environment Initial Data on a Test Bank Assessing Total Enterprise Simulation Learning Changing the Assessment Paradigm: Using Student Portfolios To Assess Learning from Simulations How We Learn and Why We Don't: The Cognitive Profile Model: A Workshop in Teaching to Reach Your Students Knowing Thyself: A Portfolio Approach to Student Self-Assessment Collaborative Learning and Web-Based Instruction in a Cognitive Apprenticeship Model Teamwork Attributes in a Classroom Simulation Virtual Teams: Meeting the Next Challenge for Experiential Education New Age Learning: Nuance or Nonsense Developing Charisma: An Experiential Exercise in Leadership Problems and Solutions in Going Web-based with an Agribusiness Simulation Creating a Comprehensive Web-Enhanced Classroom Your Class is in Session, Now What? The Challenges of Teaching On-line An Application of Process Control Charts for Attributes as a Form of Classroom Assessment for Experiential Learning Work Goals and Life Aspirations: Do You Have What it Takes to be an Entrepreneur An Exercise to Develop Initiative: Possible Dream? The Ball Point Pen Assembly Company Management Game Review System Development Total Enterprise Simulations and Optimizing the Decision Set: Assessing Student Learning Across Decision Periods Facilitating Learning in the New Millennium with the Complete Online Decision Entry, System (CODES) The Marketing Management Experience The Right Venue for Your Simulation One More Time: Overall Dominance in Total Enterprise Simulation Performance A Profile of ABSEL Conference Attendees Learning Readiness: An Underappreciated Yet Vital Dimension in Experiential Learning Active Learning in a Professional Undergraduate Curriculum The Problem Is - They Think Differently! Cultures Integration in Mergers and Acquisitions: Putting Managers Together in a Business Simulation The Global Business Game: A Strategic Management and International Business Simulation