The Continuing Evoluation of Assessing Project Management as an Academic Learning Outcome (ALO) Page 182- Developments in Business Simulation and Experiential Learning, volume 38, 2011 ABSTRACT The last five years has seen most colleges and universities ask,” how can the outcomes it desires for graduates be assessed to ensure compliance with the educational stan- dards they embrace.” Legislatures are directing universi- ties and colleges to develop Academic Learning Outcomes (ALO) to meet the State’s Academic Learning Compacts being established in curriculums. Students must be able to demonstrate an ability accomplishing an ALO such as Pro- ject Management (PM), but a learning domain like this must be measureable. To evaluate PM, assessing many project management activities require an examination of complex processes required for the project’s completion. Our students do this by competing as teams in a business strategy simulation and being responsible for PM activities for success. This paper examines a year’s worth of pro- gress toward an accurate measurement of the Project Man- agement ALO using team members, not faculty, to rate each team member on PM activity performance. It is hope- ful this evaluation with the final team simulation perform- ance standings, can be used to assess if a PM ALO has an effect on final team simulation performance INTRODUCTION Academic institutions today are assessing students’ academic engagement in studies. The Southern Associa- tion of Colleges and Schools (SACS) and Association to Advance Collegiate Schools of Business (AACSB) assess- ment directions has led to new university- and college-level directives about academically engaging students. These directives led the State and university to mandate five over- arching academic learning outcomes (ALO): a) content knowledge, b) critical thinking, c) communication, d) eth- ics, and e) project management that students be able to demonstrate upon graduation. The development, testing, and measurement of these five ALO is a work-in-process across every college at our university. One difficult rubric to be developed is “how to capture” project management (PM) as an ALO. Our PM rubric began development in 2006. In 2008, the rubric was used to assess the PM learn- ing outcome as part of the COB capstone course: MAN4720 Business Policy and Formulation. This paper’s goal is to report on progress being made to measure PM using our rubric. This report addresses if it is an appropriate measure for an ALO like PM using expe- riential exercises, like simulations. This goal is accom- plished by asking three questions: 1) “how did the PM Aca- demic Learning Outcome (ALO) and rubric develop?” 2) “what was reported concerning the PM rubric to ABSEL in 2010?” and 3) “what have we learned and where is it headed? HOW DID THE PM RUBRIC DEVELOP? The Direction The AACSB adopted new accreditation and maintenance standards in 2003. The State Board of Governors in 2004 required all State universities to imple- ment Academic Learning Outcomes (ALO) for under- graduate and graduate degree programs. These ALO show what abilities a graduating students should be able to dem- onstrate such as: 1) content or discipline concepts, theories, and frameworks, 2) critical thinking by manages informa- tion, higher-level cognitive skill sets, problem solving, and creativity, 3) communications involving appropriate writ- ten, spoken, quantitative, and technological skills in each discipline, 4) integrity/values that embraces the areas of decision making, academic integrity, and professional stan- THE CONTINUING EVOLUTION OF ASSESSING PROJECT MANAGEMENT AS AN ACADEMIC LEARNING OUTCOME (ALO) Martin J. Hornyak University of West Florida mhornyak@uwf.edu Blaine Lawlor University of West Florida blawlor@uwf.edu Stephen Snyder University of West Florida ssynder@uwf.edu mailto:mhornyak@uwf.edu mailto:blawlor@uwf.edu mailto:ssynder@uwf.edu Page 183- Developments in Business Simulation and Experiential Learning, volume 38, 2011 dards, and 5) project management ability to analyze a pro- ject planning and execution functions. Corresponding ru- brics matching each ALO must be developed to help deter- mining if actual student learning is matching the mandated expectations (Association to Advance Collegiate Schools of Business, 2003; Collegiate State Board of Trustees – Academic & Student Services Committee Meeting, August, 2004, Quality Enhancement Plan, January 2005). The Vehicle The COB’ capstone course, MAN4720, is designed for our business students to take it the semester prior to graduation. This timing sequence makes MAN4720 an ideal place to assess learning outcomes for soon to be graduates. The course is designed around a typi- cal business policy and formulation agenda covering basic strategic management theory, case analysis, financial analysis, with a business simulation. By compiling multi- ple course elements in one course, assessments of several college-level ALO and learning domains can occur by separately evaluating exams and quizzes (40%), case analy- sis and discussion (30%) and a business simulation (40%). The percentages reflect the value each course area contrib- utes to the course grade. The simulation is completed by groups of 3-4 self-selected or instructor determined teams. Teams are totally responsible for team decisions and re- ports as identified in the course syllabus. The experiential exercise, a simulation, used in this course is a Total Enterprise Simulation called the CAP- STONE Business Simulation (CAPSIM Student Guide, 2010). The simulation places student teams in charge of $100M sensor manufacturing company. Students must complete rehearsal rounds, four (4) practice rounds, and eight (8) final decision rounds during the semester. Every team develops a strategic plan using strategy models such as Porter’s Five Forces, Value Chain Analysis, etc., learned in the course, while making its weekly simulation deci- sions. With multiple decisions being made, a PM academic learning objective beg measurement. Looking at research findings of Wellington & Faria (1995), Peach (1996), and Gentry (1990) suggests positive relationships existing be- tween simulations and strategic management. Developing objectives, analyzing environments, choosing among strate- gic alternatives, monitoring and reacting to results are ele- mentary to basic strategic management and using simula- tions. With basic PM rubric’s skills being derived from de- fining what a project is: “a complex, non-routine, one-time effort limited by time, budget, resources, and performance specifications designed to meet customer needs” (Gary & Larson, p.5, 2008). With PM typically being managed in three phases: planning, scheduling, and controlling (Heizer & Render, p.56, 2004), it is easy to understand why PM assessments can be made in this course and simulation. In Summary of PM Student Performance Table 1 Rating Unacceptable Acceptable Exemplary Total 2005 Number 7 31 51 89 % 8% 35% 57% 100% 2006 Number 6 73 32 111 % 5% 66% 29% 100% 2008 Number 12 68 3 83 % 14% 82% 4% 100% 2010 Number 0 19 15 34 % 0 56 44 100% ** 2010 Number 5 12 18 34 % 12% 35% 53% 100% **2010 Excluding Self Assessment Page 184- Developments in Business Simulation and Experiential Learning, volume 38, 2011 the course, the simulation event is a semester long, one time activity managed by student teams that address com- plex business situations faced in its sensor industry WHAT WAS THE PM ASSESSMENT RE- PORT AT ABSEL 2010 At the 37th Annual ABSEL Conference, an experiential track presentation was held summarizing the continuing PM assessment efforts at our university (Hornyak, Lawler, & Peach, 2010). The presentation evaluated the multiple PM rubrics used from 2006-2009. To help manage the assessment effort, the university measures and reports each of its five ALO every two years including PM being as- sessed again in fall semester 2010. From this rubric, the university wants to see if the soon-to-graduating students display PM planning skills, both individual and team work skills, and abilities to successfully deliver project results. PM assessment yearly results are seen in Table 1. As dis- cussed at ABSEL 2010, the PM 2008 measurement clearly shows differences from the earlier measures believed to come from the specific directions given to students by all instructors before taking the survey. The instructors em- phasized the rating definition that rating team members a 3 or Acceptable means the team member performed Survey Instructions Figure 1 Please Help the College of Business & Management/MIS Department: The College of Business is developing rubrics to help us assess the Academic Learning Outcomes (ALO) within MAN4720. One ALO, we try to measure is: Do students demonstrate an ability to successfully perform elements of Pro- ject Management as a CAPSIM team member. Project Management is defined as the ability to plan, schedule, and con- trol a project from start to finish. CAPSIM is a project because teams take responsibility to operate a sensor manufactur- ing company with limited budgets, time, and resources…all being accomplished in one academic semester. Project Management: the concept is measured by evaluating the skills needed to complete successful project manage- ment. Who can evaluate what team members did for the project? Team members! Not faculty! Because team mem- bers witness the actual skills team members contribute to the project or lack there of. Project Management Skills Defined:  Project Planning: How did a team member assist in the development of their CAPSIM plan; Did they contrib- ute by identifying required tasks, responsibilities, deadlines, and performance expectations.  Individual Work Skills: As a individual, did the team member set appropriate completion goals, manage per- sonal timeframes & schedule appropriately, and complete all required tasks in a timely and professional quality manner.  Team-Work Skills: As a team member, did the individual positively contribute to achieving team objectives, completing their team responsibilities, mediating any conflicts among members, participating in all scheduled team activities, and responding quickly and effectively to team feedback.  Project Delivery: How did this individual contribute to the final project’s on-time delivery, to it complying with all MAN4720 requirements, making valid product and process suggestions, and to accurately assessing the qual- ity of their personal contribution? Please fill out a SEPARATE SURVEY FOR EACH MEMBER OF YOUR GROUP. This survey is being used by us to assess CAPSIM and WILL NOT be used for grading purposes. Fill out separate sur- veys for each of your team members and yourself. Thank you for your help and thanks for a great semester! Drs Hornyak, Snyder, and Lawlor Page 185- Developments in Business Simulation and Experiential Learning, volume 38, 2011 Assessment of Project Management Skills in the Capstone Course -- Fall 2008 Figure 2 Name of Team Member: _____________________________ Team: _________________ Self-Assessment: ____ INSTRUCTIONS: On a separate form for each team member, including yourself, place an X to identify level project management efforts. Use the rating definitions below for your answers. 1. UNSATISFACTORY: Team member failed to provide minimal contributions for input point. Team member had a significant negative impact on team performance and/or created extra work for other team members through late or unsat- isfactory contributions. 2. BELOW EXPECTATIONS: Significant or repeated shortfalls in performance that negatively impacts overall team performance. Team member did not significantly modify behavior after being advised of problems. 3. MEETS EXPECTATIONS: Team member generally performed in a manner reflective of a serious, contributing mem- ber. May have committed occasional minor errors (e.g., late to meetings) but not to the extent it was a negative impact on the team. May have occasionally done extra work or put in extra effort, but of the type you would expect a team member to reasonably do. 4. EXCEEDS EXPECTATIONS: Team member consistently performed in a manner beyond what would be reasonably expected, and was a strong positive influence on the team and its performance. May have done significant extra work, helped other team members with their tasks, or provided extra effort wherever needed. A. Project Planning 1 2 3 4 5 In this section, assess performance concerning the team’s approach to planning the project. Pts. Unsatis- factory Below Meets Exceeds Well Ex- ceeds Identify Required Tasks 5 Assign Responsibilities for Tasks 5 Establish Deadlines for Tasks 5 Agree On Performance Expectations 5 B. Individual Work Skills 1 2 3 4 5 In this section assess tasks accomplished individually to the best of your knowledge. Pts. Unsatis- factory Below Meets Exceeds Well Ex- ceeds Sets appropriate goals for completing individual tasks 5 Manages timeframe and schedule appropriately 5 Completes all individual tasks in a timely manner 5 Completes all individual tasks with appropriate quality 5 C. Team-Work Skills 1 2 3 4 5 In this section, grade performance as a team member to- wards accomplishing team objectives. Pts. Unsatis- factory Below Meets Exceeds Well Ex- ceeds Contributes positively to accomplishing team objectives 5 Effectively completes responsibilities 5 Effectively mediates conflict among team members 5 Participates in all scheduled team activities. 5 Responds effectively to feedback 5 D. Project Delivery 1 2 3 4 5 In this section, assess the impact of the team member’s per- formance on the overall final project. Pts. Unsatis- factory Below Meets Exceeds Well Ex- ceeds Team projects delivered on time 5 Effectively complied with project requirements 10 Makes valid suggestions for improving process & product 10 Able to accurately assess quality of personal contribution 10 Total Project Management Points: 100 Exemplary: 85 – 100 Acceptable: 73 – 84.9 Unacceptable: <72.9 Page 186- Developments in Business Simulation and Experiential Learning, volume 38, 2011 PM tasks adequately and did contribute to the team’s simulation success. For team members to rate people a 4 - Excellent or 5 – Outstanding that person’s particular behav- ior and skills provided significantly more value to the team’s simulation effort. Because of the ABSEL 2010 discussion and comments received, the PM rubric continues being reviewed for the Fall 2010 survey administration. See Figure 2 for the pro- ject management rubric used in 2008. Any rubric review should start by reassessing the sum- mated rating scale that was used (Spector, 1992). First the authors checked the definition for the PM construct used for the survey was found again to have content validity using university faculty involved with curriculum assess- ment. Once a construct is confirmed, the scale that being used is looked at. The previously used 2008 PM rubric and scale was re- evaluated. Assessment faculty questioned having a five point rating scale. With the university wanting to be com- fortable knowing the graduating seniors have demonstrated good PM skills. What does a PM rubric asking team mem- bers to rate colleagues excellent (4) or outstanding (5) do for the PM assessment? Actually, it does not do much. As Spector (1992) states do not give respondents any more choices than they can use. Despite the fact student know the survey results do not have an impact on their grade, will not stop certain students from not providing accurate scores. This can be the result from people having poor decision-making skills, people not working hard and then trying to give a new image, teams not getting along person- ally with each other, and being too young or experienced about doing personal evaluations. All these factors could affect results. What will be suggested is using an updated PM rubric this Fall 2010 to see if more useful information can be obtained. Survey instructions are being constructed to prepare students to take for the survey. Attachment 1 shows the instruction form to be placed on the front of the two-sided survey. The instructions are provided to give respondents a common frame of reference for the survey (Spector, 1992). The PM survey instructions accomplish several key items: 1) to explain why the survey is being conducted, 2) to de- fine what are the PM skills being evaluated are, and 3) what are is the important submission areas. A last concern out rubric discussions was having vary- ing weights identified for various PM scale items. The last three questions on Project Delivery where scaled to 10 points rather than 5. This was done in an attempt to make sure the PM rubric results can be measured out of 100 points. No need to do this for assessment purposes because just using an 85-point baseline can be just as effective (Spector, 1992). WHERE IS THE PM RUBRIC HEADED? There is now an adjusted PM rubric going to be evalu- ated this fall and data from the survey is going to be col- lected by 30 November 2010. See Figure 3 for a version of the adjusted PM rubric about to be used. The PM rubric data reduction and analysis has been completed by 15 De- cember 2010. Some of the results can be seen in Table 1. Time only allowed 1 class of 34 students to be evaluated and two more classes will be added later. The 2010 data suggests definitions of student PM expectations need to be re-examined. For example, how do you measure good PM? Is it the team member achieving the highest rubric score having the best PM skills or the rubric score be similar hav- ing minimal differences between team members? Interest- ingly, separating PM self-assessments in the analysis showed certain students exaggerated their impact on the team (see Table 1). Is this good PM behavior or is it unac- ceptable? PM is a measure of a person’s ability to work as a team. New PM definitions, measures, and findings con- cerning PM will be discussed at ABSEL 2011. CLOSING THOUGHTS This paper begins answering a series of questions: 1) “How and why was a PM assessment rubric developed? 2) What was reported at ABSEL 2010? and 3) Where is this PM rubric heading? ABSEL is a wonderful organization able to discuss new ideas and areas affecting educators. With curriculum assessment being required throughout education, the measuring of student’s learned abilities against educational standards is at the forefront. Our uni- versity’s assessment experience can offer suggestions on using a designed PM rubric to measure State-directed learning outcomes. Carefully developed rubrics can ensure that students are learning that best PM practices are the powerful tools pushing organizations toward operating ex- cellence and better execution. For learned PM skills to develop into becoming a “best practice” of graduating stu- dents, it must able to be demonstrated successfully over time, delivering quantifiable, positive results, and be re- peatable (Thompson, Gamble, & Strickland, 2004). Our goal then becomes to verify over time and document that students can identify and rate PM team activities effec- tively. This leaves us, again, at the point of looking toward future investigations in our evolving PM ALO. REFERENCES Association to Advance Collegiate Schools of Business (AACSB) International (2003). Eligibility procedures and standards for business accreditation, AACSB In- ternational.St. Louis, MO CAPSIM Student Guide (2010) Management Simulations, Inc. Northfield, Illinois, 80093. Gentry (1990). Guide to Business Gaming and Experiential Learning, Nichols/GP Publishing, East Brunswick/ Kogan Page, London. Reprinted in the Bernie Keys Library 6th edition [Available from http://ABSEL.org] Gray & Larson (2008), Project Management, The Mana- http://absel.org/ Page 187- Developments in Business Simulation and Experiential Learning, volume 38, 2011 Assessment of Project Management Skills in the Capstone Course -- Fall 2010 Figure 3 Name of Team Member: __________________________________ Team: __________Self-Assessment: ____ INSTRUCTIONS: On a separate form for each team member, including yourself, place an X to identify level the project management efforts demonstrated by each person on your team. Project management is defined as the task of getting required activities done on time, within budget, & meeting pro- ject specifications. Use definitions below when answering this survey. 1. UNSATISFACTORY: Team member failed to provide minimal contributions for input point. Team member had a significant negative impact on team performance and/or created extra work for other team members through late or unsat- isfactory contributions. 2. BELOW EXPECTATIONS: Significant or repeated shortfalls in performance that negatively impacts overall team performance. Team member did not significantly modify behavior after being advised of problems. 3. MEETS EXPECTATIONS: Team member generally performed in a manner reflective of a serious, contributing mem- ber. May have committed occasional minor errors (e.g., late to meetings) but not to the extent it was a negative impact on the team. May have occasionally done extra work or put in extra effort, but of the type you would expect a team member to reasonably do. 4. EXCEEDS EXPECTATIONS: Team member consistently performed in a manner beyond what would be reasonably expected, and was a strong positive influence on the team and its performance. Performed significant extra work, helped other team members with their tasks, or provided extra effort wherever needed. A. Project Planning 1 2 3 4 In this section, assess performance concerning the team’s approach to planning the project. Pts. Unsatis- factory Below Ex- pectation Meets Ex- pectation Exceeds Ex- pectation Identify Required Tasks 5 Assign Responsibilities for Tasks 5 Establish Deadlines for Tasks 5 Agree On Performance Expectations 5 B. Individual Work Skills 1 2 3 4 In this section assess tasks accomplished individually to the best of your knowledge. Pts. Unsatis- factory Below Ex- pectation Meets Ex- pectation Exceeds Expectation Sets appropriate goals for completing individual tasks 5 Manages timeframe and schedule appropriately 5 Completes all individual tasks in a timely manner 5 Completes all individual tasks with appropriate quality. 5 C. Team-Work Skills 1 2 3 4 In this section, grade performance as a team member to- wards accomplishing team objectives. Pts. Unsatis- factory Below Ex- pectation Meets Ex- pectation Exceeds Ex- pectation Contributes positively to accomplishing team objectives 5 Effectively completes responsibilities 5 Effectively mediates conflict among team members 5 Participates in all scheduled team activities. 5 Responds effectively to feedback 5 D. Project Delivery 1 2 3 4 In this section, assess the impact of the team member’s performance on the overall final project. Pts. Unsatis- factory Below Ex- pectation Meets Ex- pectation Exceeds Ex- pectation Team projects delivered on time 5 Effectively complied with project requirements 5 Makes valid suggestions for improving process & product 5 Able to accurately assess quality of personal contribution 5 Total Project Management Points: 85: Exemplary: 72-85 points Acceptable: 51–71.9 points Unacceptable: <=50-9 Page 188- Developments in Business Simulation and Experiential Learning, volume 38, 2011 gerial Process, 4th ed., McGraw Hill Irwin, Boston. Hornyak, Lawler, & Peach (2010) “Continuing to Assess Project Management: What’s Been done? What’s it Means Looking for New Twists” 37th Annual Associa- tion for Business Simulation & Experiential Learning (ABSEL) National Conference, Little Rock AR Heizer & Render (2004) Principles of Operations Manage- ment. 5th Ed, Upper Saddle River, NJ: Pearson/Prentice Hall. Peach (1996) “Enhancing Simulation Learning Through Objectives and Decision Support Systems.” Develop- ments In Business Simulation & Experiential Learning, 23, 61-67. Spector, (1992) Research Design, Newbury Par4k: Sage. Thompson, Gamble, & Strickland (2004, 2006, 2008) Strat- egy - Winning in the Marketplace, New York, NY: McGraw-Hill/Irwin. University of West Florida Board of Trustees. (2004). Aca- demic & Student Services Committee Meeting Minutes, August 2004. University of West Florida Quality Enhancement Plan. (2005). Enhancing Student Learning: Creating a com- munity of learners through active learning and student engagement. January, 2005. Wellington, W. & Faria, A. (1995) “Are good simulation performers consistently good?” Developments in Business Simulation & Experiential Exercises, 22, 5- 11. Table of Contents Volume 38, 2011 Simulated Tabletop Exercise for Risk Management - Anti Bio-terrorism Scenario Simulated Tabletop Exercise From Business Games to Simulations - Simuworlds & Microworlds Demand Equation Redux: The Design and Functionality of the Gold/Pray Model in Computerized Business Simulations Managing Client-Based Learning: Insights from Successful Teaching Project Courses in Marketing Tracking Forecast Error Type, Frequency and Magnitude with the Forecast Error Package Responding to Facilitate Collaboration Simulating Sudden Change and the Value of Timely Information Managing Human Resources Simulation A Study on Collectivism and Group Decision-Making: An International Comparison of Japan, China, and Russia Using a Gaming Simulation The Use of Management Games in the Management Research Agenda Gaming On-Line: A Simulation Application Positioning and Performance in Simulated Networks Supply Chain Management: A Simulation Application Simulation as a Teaching Method in Strategic Management Distance Studies Entrepreneurship: A Game of Risk and Reward Phase II: The Start-Up Return to the Paradise Islands: From Confrontation to Cooperation Effect on Market Performance of Displaying Supply and Demand Curves in a Business Simulation Appreciating Complexity: The Chief of Staff of the Army Game Managing Organizations: Experiential MBA Course Teaches Alternatives to the Machine Model A Simulation Model for Analyzing the Night-Time Emergency Health Care System in Japan The Continuing Evoluation of Assessing Project Management as an Academic Learning Outcome (ALO) Should College Instructors Change Their Teaching Styles to Meet the Millenial Student? The Mouse Game and its Effects on Team Interdependence Learning from the Gulf Oil Spill to Prepare for a Brighter Future: A New Game Engaging Stake Holders in Triple Bottom Line Accounting & Strategic Planning Video Killed the Biblio Star: The Impact of Digital Media on Student Learning Outcomes Exploring Motivation: Using Emoticons to Map Student Motivation in a Business Game Exercise An Alternative to PC and Internet Based Simulations: The Internet Integrated Mode MiddleState University -- A Crisis in Education Complexity Avoidance, Narcissism and Experiential Learning Examining the Cognitive, Affective, and Psychomotor Dimensions in Management Skill Development Through Experiential Learning: Developing a Framework A Situational Leadership Exercise Based on the Biology of a Starfish JOGAI CEFET -- The Industrial Administration Undergraduate Game Would You Take a Marketing Man to a Quick Service Restaurant? Modeling Corporate Social Responsibility In A Food Service Menu-Management Simulation Use of a Simulation in a Large Class Environment for a Marketing Principles Class: A Qualitative Analysis of Whether Learning Objectives were Met A Team Based Information Literacy Exercise ABSEL Marketing Communications Plan An Interdisciplinary Study of the Impact of Playing a Marketing Simulation Game on Student Knowledge of Management Accounting/Finance Principles Analyzing Construction Planning of Interiro Finish Work of Apartment Building by Simulation Doing Murder One Again The Simple Business Game and Simulation Transfering the Knowledge of Middle Management to Novices Infectious Disease Simulation Model for Estimation of Spreading Understanding the Relative Influence of Several Factors in ERP Simulation Performance: An Exploration of Ecological Validity Tragedy of the Commons: An Exercise Using Clickers to Illustrate and Teach a Key Concept in Negotiations The Meaning of Firm Demand in Business Simulations If the Games Work, Why Aren't More Faculty Willing to Play? Those Who Do and Those That Don't: A Study of Engaged and Disengaged Business Game Players