Evaluating Business Plans in a Simulation Environment EVALUATING BUSINESS PLANS IN A SIMULATION ENVIRONMENT Annette L. Halpin Arcadia University halpin@arcadia.edu William D. Biggs Arcadia University biggs@arcadia.edu ABSTRACT This paper describes a template for evaluating business plans in a simulation environment. The template incorporates rubrics to help students understand the evaluation criteria and process, and to facilitate evaluation by multiple parties (instructors, outside executives, faculty members). The authors argue that simulation research can be enhanced through the use of common scoring instruments that identify specific, observable attributes that distinguish between high and low levels of performance on simulation-related business plans. INTRODUCTION A review of the ABSEL literature demonstrates that the possibility of using business plans in simulation environments is of interest to users and researchers in the field. There are over 60 articles that mention business plans and one of these (Schreier and Komives, 1977) goes back to the third meeting of the organization; however, this was the only article in the proceedings in the 1970’s. There were three articles during the 1980’s; 14 during the 1990’s; and 41 articles between 2000 and 2008. Thus, there is evidence of an increasing interest in business plans by those who conduct research in the areas of simulations and experiential learning. It is also the case, however, that the majority of these articles merely mention business plans. For example, in the Schreier and Komives (1977:55) article, which focuses on entrepreneurship, the only use of the term ‘business plan’ is in the statement, “Can this team take the business plan, run with it, and make it happen?” Likewise, Kenner and Uretsky (1986) make only a passing mention of business plans, despite the fact that their article is focusing on the use of supplemental activities to enhance learning in simulation environments. Since the interest in this article is on the use and evaluation of business plans in simulation environments, the focus is on articles that have more extensive coverage of business plans. There are a number of researchers who have argued that there is value to having students develop business plans in simulated environments. Cherukuri and Cannon (1988) focus on using a computer-based business plan assistant, such as those developed by the SBA, to assist students with developing their plans. They do not, however, provide any information on the quality of the plans developed or ways to evaluate these documents. Malik, Howard and Morse (1997) discuss the value of business plans as they argue that they are an integrative tool that may be used as substitutes for simulations when teaching strategy, although they also say that they can be used as a supplemental activity. They cite two pieces of research, Anderson and Lawton (1992) and Curran and Hornaday (1987), which looked at students who had to develop a business plan for a total enterprise simulation they were playing. However, neither of these studies addressed the evaluation of these plans. Hornyak and Peach (2004, p. 274) had students submit business plans that were evaluated according to, “1) the quality of the plan (completeness, challenging/reasonable objectives, and well- articulated strategy) and 2) consistency of the plans with the team’s actual decisions.” While these are certainly reasonable criteria, they do not provide solid benchmarks against which comparisons can be made about the quality of one plan versus another. In light of the increased interest by accrediting bodies in the assessment of student work, evaluating business plans in a systematic fashion is a reasonable expectation. ASSESSMENT OF STUDENT WORK Two major accrediting bodies of business programs, The AACSB (The Association to Advance Collegiate Schools of Business) and ACBSP (Association of Collegiate Business Schools and Programs) state in their standards that student learning is a process that requires a formal program of assessment (AACSB, 2008; ACBSP, 2008). Institutions seeking accreditation establish program goals and course learning outcomes and are expected to evaluate student performance to determine the extent to which students meet some pre- determined set of expectations. The understanding is that the process of formally setting goals, assessing progress, and correcting weaknesses will assure external constituencies (employers, trustees, prospective students, etc.) that the business program is providing a valuable learning experience for its primary internal constituency, its students. It also assists deans, chairs, and faculty members in determining the strengths of the program as well as areas that need improvement. The senior-level Business Policy course is a common course requirement of majors at the end of their undergraduate business program. Szczewbaci, Duserick, Rummel, Howard, and Viggiani (2000) note that within this class the use of a business plan is one way to show fulfillment of the school’s mission while meeting accreditation requirements for assessment. Drost and Chaney (2001) address this issue more directly when they 148 | Developments in Business Simulation and Experiential Learning, Volume 36, 2009 mailto:halpin@arcadia.edu mailto:biggs@arcadia.edu describe how their institution assesses student knowledge of core business subjects (accounting/finance, information and decision sciences, management and legal environment, marketing, public administration, and strategy) by requiring the preparation of a business plan. They describe the evaluation criteria for the assignment and identify seven items that all plans must include: a Vision and Mission Statement; Company Overview; Product Strategy; Marketing Analysis and Marketing Plan; Financial Analysis; Governmental Strategy; and Overall Strategic Plan for the Business. The eighth item, a PowerPoint© presentation, is included in the assessment but is not part of the written business plan. To help students develop their plans they require the use of Biz Plan Express (Kapron & Reidel, 2006), a software package with templates that correspond to each of the areas required. Drost and Cheney (2001, p. 46) conclude that this integrative exercise achieves its goal by providing their program with a method “to assess the degree to which students have applied core business concepts”. EVALUATION CRITERIA Assessment is clearly at the forefront of the goals for institutions seeking accreditation or renewing their accreditation status. The challenge for leaders of business programs is to establish methods to determine what to evaluate and how to do it. While feedback from multiple reviewers may be received for assignments, such as the business plans discussed above, the specific criteria used to assess student performance have not been provided (Cherukuri & Cannon, 1998; Drost and Chaney, 2001; Malik, Howard, & Morse, 1997). Hall and Ko (2006) address this issue, in part, by arguing for the use of rubrics to evaluate business plans in simulation environments. They offer some examples but do not provide the benchmarks needed for assessment purposes. One of the problems with much of the ABSEL literature is that researchers exploring a particular topic develop their own instruments to collect the data. Thus, three articles that look at a topic such as the impact of group cohesiveness on simulation performance can’t be directly compared because the instruments used are not comparable. In some instances the problem is extreme since a theoretical base for the instrument being used is lacking. It is clearly easier to compare the results of studies looking at cohesiveness if they all used some generally accepted instrument. Obviously, this point can be made about other areas of research as well. This article offers a specific set of criteria to evaluate business plans in a simulation environment and then presents them in a format with a scoring rubric that will be useful for those involved in the assessment process. Ideally, the community of researchers working on the topic of evaluating business plans in simulation environments will move to use some generally accepted instrument in order to show agreement on what is of value in a business plan as well as to strengthen the theoretical base for such an instrument by allowing comparisons of results across studies. What is proposed in this work should be considered a first step towards this goal. DEVELOPMENT OF THE RUBRIC TO EVALUATE BUSINESS PLANS For many years the second author required students in Business Policy to prepare a business plan for their firms in a simulation environment. These plans were read by the instructor, fellow faculty members, and business executives who were acting in the role of board members. To facilitate evaluating the reports and providing feedback to students the form shown in Exhibit I was developed. A number of years ago this form was adopted for use in the International Collegiate Business Strategy Competition (ICBSC), which also uses business executives to evaluate student business plans in a simulation environment. Thus, a number of different users in two different simulation environments have been using this instrument. While the instrument was helpful in obtaining some feedback, it did not address specific characteristics of business plans such as mission/vision statements, goals/objectives, and functional area decisions. In addition, since the range of evaluation categories (‘Strongly Agree’ to ‘No Basis for Judgment’) did not provide clearly defined descriptors of what would constitute a particular level of performance, one cannot assume that all evaluators were using the same criteria. To address these problems the first author developed the form shown in Exhibit II. This instrument lists specific criteria that are related to items students are told should be covered in their business plans. Students are familiar with these expectations from in-class discussions as well as guidelines for developing a business plan offered on The Small Business Administration website (United States Small Business Administration, 2008) and BizPlan Builder Express (Kapron & Reidel, 2006), resources mentioned earlier as used in other studies. The rubric provided in Exhibit II draws on the concept of Behaviorally Anchored Rating Scales - BARS (Smith & Kendall, 1963) used for performance appraisal in human resources (Catano, Darr, & Campbell, 2007). BARS combine the use of critical incidents (behaviors) and rating scales (strong, average, weak) in an instrument that provides examples of what constitutes a particular level of performance. Judgment is based on specific behaviors rather than on abstract constructs or adjectives (Maiorca, 1997). BARS attempt to provide reviewers with an objective and systematic method of evaluation (Campbell & Cairns, 1994). The scoring rubric presented in Exhibit II lists seven qualities expected in a business plan. Different score levels (1 through 4) are described based on the amount of evidence for each trait in the plan. Each score category provides specific guidelines for the reviewer and may include both qualitative (‘organized’) and quantitative (‘0-3 grammatical errors’) descriptors. As argued by Moskal (2000, p. 2), “By having a description of the characteristics of responses within each score category, the likelihood that two independent evaluators would assign the same score to a given response is increased.” For their part, reviewers were asked to read through a plan, score the plan on each trait by placing a number under the ‘Score’ column, include additional comments on page 2 if desired, and submit the completed form to the instructor/facilitator. Assigning scores that are half-way between two levels such as 3 ½ is acceptable so total scores could include fractions (20 ½, 23 ½, etc.). Each 149 | Developments in Business Simulation and Experiential Learning, Volume 36, 2009 plan was reviewed by three to four evaluators so scores were totaled across reviewers to arrive at a final score for each plan. Converting the ‘final score’ to a grade required some thought since an average score of 14 (all 2’s) does not necessary indicate a score of 50% or ‘F’. Trice (2000) suggests that there are more scores that fall in the ‘average’ range than in the ‘below average’ range. For the proposed form, a score of 14 was considered a ‘C’, a score of 21 a ‘B’, and a score of 28 an ‘A’. Scores below 14 were assigned a letter grade of either ‘D’ or ‘F’. The approach taken in this paper helps to address an issue raised by accrediting organizations, the desire for assessment to be conducted in a transparent and meaningful way. The revised instrument was used during the spring 2008 semester at the authors’ university as well as in the 2008 International Collegiate Business Strategy Competition (ICBSC). Informal feedback from board members of both groups indicates that they found the new instrument to be superior to the earlier one. Comments from long-time board members noted that the new format ‘provided clear guidelines for judging’ and ‘listed expectations in several areas’. At the end of the process students at the authors’ institution, as well as, participants in the ICBSC received a copy of the rubric with their score on each item and a summary of the open-ended responses from board members. By viewing their strengths and weaknesses, performance on future writing assignments (annual report and management report) may be strengthened since areas of improvement are linked with behaviors that students can affect in the next report. CONCLUSION The process of evaluating student work at the collegiate level continues to evolve as business accreditation organizations require more attention be paid to assessing performance and providing meaningful feedback to students. The authors provide a suggested format to evaluate business plans in a simulation environment and which will offer meaningful feedback to students. It is argued that simulation research could be enhanced by the use of the same instrument across studies. Instructors are encouraged, therefore, to use the template presented here and provide feedback concerning: (1) successes (pluses); (2) failures (minuses); and (3) potential modifications By using a common instrument for collecting and analyzing data related to business plans in simulation environments researchers can eliminate one variable across studies. This should facilitate making generalizations and aid in theory building. It is the authors’ belief that this type of collaboration will help to strengthen simulation research. REFERENCES Anderson, Philip and Leigh Lawton (1992). “A survey of methods used for evaluating student performance on business simulations.” Simulation and Gaming, 23,490-498. Association of Collegiate Business Schools and Programs. (2008). Standards and criteria for demonstrating excellence in baccalaureate/graduate degree schools and programs. Retrieved November 12, 2008, from http://www.acbsp.org/p/cm/Id/fid=81 Campbell, Terry and Heather Cairns (1994). “Developing and measuring the learning organization: From buzz words to behaviours.” Industrial and Commercial Training, 26 (7), 10-15. Catano, Victor M., Wendy Darr, Catherine A. Campbell (2007). “Performance appraisal of behavior-based competencies: A reliable and valid procedure.” Personnel Psychology, Vol.60 (1), 201-231. Cherukuri, Vijay and Hugh M. Cannon (1988). “Using a computer-based business plan assistant in conjunction with a marketing simulation game.” Developments in Business Simulation and Experiential Learning, Vol. 15, 262-264. Reprinted in the Bernie Keys Library, 9th edition [Available from http://ABSEL.org] Curran, K. and Robert Hornaday (1987). “An investigation of the relationships between formal planning and other performance measures of learning on a simulation exercise.” Developments in Business Simulation and Experiential Learning, Vol. 14, 43-46. Reprinted in the Bernie Keys Library, 9th edition [Available from http://ABSEL.org] Drost, Donald and John Chaney (2001). “Core course outcomes assessment and continuous improvement using an integrative business plan – an empirical evaluation.” Developments in Business Simulations and Experiential Learning, Vol. 28, 42-48. Reprinted in the Bernie Keys Library, 9th edition [Available from http://ABSEL.org] Hall, Owen and Kenneth Ko (2006). “Learning assurance using business simulations applications to executive management education.” Developments in Business Simulation and Experiential Learning, Vol. 33, 1-6. Reprinted in the Bernie Keys Library, 9th edition [Available from http://ABSEL.org] Hornyak, Martin J. and E. Brian Peach (2004). “Reporting lessons learned: What gets reported; who gains value?” Developments in Business Simulation and Experiential Learning, Vol. 31, 273-275. Reprinted in the Bernie Keys Library, 9th edition [Available from http://ABSEL.org] Kapron, Jill & Jim Reidel (2006). BizPlan Builder Express. Jian Tools for Sales, Inc. Kenner, Joe and Mike Uretsky (1986). “The management decision laboratory at New York University.” Developments in Business Simulation and Experiential Learning, Vol. 13, 195-203. Reprinted in the Bernie Keys Library, 9th edition [Available from http://ABSEL.org] Maiorca, Joseph (1997). “How to construct behaviorally anchored rating scales (BARS) for employee evaluations.” Supervision, 58 (8), 15-18. Malik, S. Dolly, Barbara J. Howard, and Kenneth O. Morse (1997). “Business plans, case studies, and total enterprise simulations: A natural co-existence.” Developments in Business Simulation and Experiential Learning, Vol. 24, 158-163. Reprinted in the Bernie Keys Library, 9th edition [Available from http://ABSEL.org] Moskal, Barbara M. (2000). “Scoring rubrics: What, when and how? Practical Assessment, Research & Evaluation, Vol. 7 (3). Retrieved November 15, 2008 from http://PAREonline.net/getvn.asp?v=7&n=3 Schreier, James W. and John L. Komives (1977). “Assessment of entrepreneurial skills using simulations.” Developments in Business Simulation and Experiential Learning, Vol. 3, 53-60. Reprinted in the Bernie Keys Library, 9th edition [Available from http://ABSEL.org] 150 | Developments in Business Simulation and Experiential Learning, Volume 36, 2009 http://www.acbsp.org/p/cm/Id/fid=81 Smith, Patricia C. & Lorne M. Kendall (1963). “Retranslation of expectations: An approach to the construction of unambiguous anchors for rating scales.” Journal of Applied Psychology, 47, 149-155. Szczrbaki, David, Frank Duserick, Amy Rummel, J. Howard, and F. Viggiani (2000). “Active learning in a professional undergraduate curriculum.” Developments in Business Simulation and Experiential Learning, Vol. 27, 272-278. Reprinted in the Bernie Keys Library, 9th edition [Available from http://ABSEL.org] The Association to Advance Collegiate Schools of Business (2008). Eligibility procedures and accreditation standards for business accreditation. Retrieved November 12, 2008, from http://www.aacsb.edu/accreditation/process/documents/AA CSB_STANDARDS_Revised_Jan08.pdf Trice, Ashton D. (2000). A handbook of classroom assessment. New York: Longman. United States Small Business Association (2008). Small business planner. Retrieved November 12, 2008, from http://www.sba.gov/smallbusinessplanner/index.html 151 | Developments in Business Simulation and Experiential Learning, Volume 36, 2009 http://www.aacsb.edu/accreditation/process/documents/AACSB_STANDARDS_Revised_Jan08.pdf http://www.aacsb.edu/accreditation/process/documents/AACSB_STANDARDS_Revised_Jan08.pdf EXHIBIT I BOARD MEMBER EVALUATION OF GROUP REPORTS AND PRESENTATIONS Firm Firm Name ___________________________________________ Number _____ Please circle one number for each question. Neither Agree No Basis Strongly Strongly nor for A. Report was Agree Agree Disagree Disagree Disagree Judgment 1. Well Written 10 9 8 7 6 5 4 3 2 1 0 2. Well Conceived 10 9 8 7 6 5 4 3 2 1 0 3. Clearly Organized 10 9 8 7 6 5 4 3 2 1 0 4. Complete 10 9 8 7 6 5 4 3 2 1 0 5. Apparently Accurate 10 9 8 7 6 5 4 3 2 1 0 6. Helpful and/or Convincing 10 9 8 7 6 5 4 3 2 1 0 7. Generally Satisfactory for its Purpose 10 9 8 7 6 5 4 3 2 1 0 B. The Group Presentation: 1. Had all Members Participate 10 9 8 7 6 5 4 3 2 1 0 2. Had all Members Participate Equally 10 9 8 7 6 5 4 3 2 1 0 3. Presented the Material in an Orderly Fashion 10 9 8 7 6 5 4 3 2 1 0 4. Seemed Prepared 10 9 8 7 6 5 4 3 2 1 0 C. Individual presentation ratings are on the reverse side of this form. D. Comments on the report and/or group presentation can be written below. 152 | Developments in Business Simulation and Experiential Learning, Volume 36, 2009 EXHIBIT II BOARD MEMBER EVALUATION OF BUSINESS PLAN Board Members – For each Business Plan that you review, evaluate the document based on the categories noted below. Use a rating scale of 4 (Highest) to 1 (Lowest). Place the score for each Category out to the right. No need to total all your scores. Firm Name _________________________________ Firm Number ________ CATEGORY 4 3 2 1 SCORE Organization Information is very organized with well- con- structed para- graphs/sub- headings; para- graphs are pre- sented in a logical order. Information is organized with many well-con-structed para-graphs/sub- head-ings; many para-graphs presented in a logical order. Information is organized in places with adequate para- graphs/sub-head-ings; some para-graphs presented out of order. Information is unorganized and paragraphs/sub- headings poorly constructed; many paragraphs presented out of order. Section Construction All sections of the business plan include introductory sentences, ex- planations or details, and concluding statements. Most sections of the business plan include intro-ductory senten- ces, explanations or details, and concluding statements. A few sections of the business plan do not include intro-ductory senten-ces, explanations or details, and concluding statements. Most sections of the business plan do not include introductory sen-tences, explana- tions or details, and concluding statements. Amount of Information Breadth of information covers all areas that should be addressed in a business plan. Breadth of information covers most of the areas that should be addressed in a business plan. Breadth of information covers only some of the areas that should be addressed in a business plan. Breadth of information covers few to none of the areas that should be addressed in a business plan. Quality of Information All supporting details and financials are accurate and consistent in the business plan. Most supporting details and financials are accurate and consistent in the business plan. Many supporting details and fi-nancials are inac-curate and incon-sistent in the business plan. Too few support-ing details and financials to determine the quality of the information. Mechanics 0-3 gram-matical, spell-ing, or punc- tuation errors. 4-6 grammatical, spelling, or punctuation errors. 7-10 grammati-cal, spelling, or punctuation errors. More than 10 grammatical, spelling, or punctuation errors. Strategic Direction Strategies/Ob- jectives are consistent with the Mission of the firm and supported by current and/or forecasted financials. Strategies/Objectives are gen-erally consistent with the Mission of the firm and often supported by current and/or forecasted financials. Strategies/Objectives are incon-sistent with the Mission of the firm and often not supported by current and/or forecasted financials. Insufficient presentation of Mission/Strate- gies/Objectives and/or inade-quate presenta-tion of current and/or forecasted financials. Graphics/ Tables Styles and content of graphics/tables are appropriate and accurate. Styles and con-tent of graphics/ tables are gen-erally appropri- ate and accurate. Styles and con-tent of graphics/ tables are in many cases in- appropriate and/ or inaccurate. Graphics/tables are too few or too poorly pre-sented to add value to the business plan. 153 | Developments in Business Simulation and Experiential Learning, Volume 36, 2009 Additional Comments: If you would like, feel free to offer some comments regarding any of the categories above or any other item related to the Business Plan. ____________________________________________________________________________________________________________ ____________________________________________________________________________________________________________ ____________________________________________________________________________________________________________ ____________________________________________________________________________________________________________ ____________________________________________________________________________________________________________ ____________________________________________________________________________________________________________ ____________________________________________________________________________________________________________ ____________________________________________________________________________________________________________ ____________________________________________________________________________________________________________ ____________________________________________________________________________________________________________ Thank you very much for taking the time to review this document. 154 | Developments in Business Simulation and Experiential Learning, Volume 36, 2009 Table of Contents Volume 36, 2009 Protecting Academic Integrity: Student Assessment In The Online Environment Pedagogical Shift: Teaching Report Writing For Accountants Online Increasing Student Learning In An Investment Management Course Through The Innovative Use Of Experiential Learning Pedagogy A Moral Development Unit For Business Courses Dividing Up Grandma’s Things: A Multifaceted Exercise In Critical Thinking Application Of Haekel’s Thesis To ABSEL Development Linking Stories With On-Line Threaded Discussions For Critical Thinking In Management Curriculum Why Have We Neglected Vicarious Experiential Learning? A Triadic Multi-Disciplinary Approach To Enhancing The Efficacy Of Experiential Learning Movement Toward Increased Student Roles In The Design Of Experiential Exercises Ready-Mix Concrete Company: An Experiential Exercise In Management Theory Using Portfolio Theory In A General Management Simulation An Experimental Analysis Of Advertising Strategies And Advertising The Relationship Between Goal Orientation And Simulation Performance With Attitude Change And Perceived Learning Beyond The Profitable-Product Death Spiral: Managing Product Mix In An Environment Of Constrained Resources Online Marketing Control With The Strategic Business Unit Analysis Package Beat The Market Game Creating Decision Support Systems In Business Simulation Games The Use Of Computer-Assisted, Interactive Role-Play Simulation In China Asia Marketing: An International Business Game An Empirical Test Of “Behavioral Immersion” In Experiential Learning Existing And Emerging Business Simulation-Game Design Movements Computerized Business Simulations: A Systems Dynamics Process Service Launch Evaluating Business Plans In A Simulation Environment Teaching Software Development By Means Of A Classroom Game: The Software Development Game Group Decision Experiments Using Business Game - Problem Solving With Conflict Luna: A Role Play Game For Learning Incoterms 2000 Entrepreneurship: A Game Of Risk And Reward Phase I -- The Search For Opportunity The Wee Game: A Pre-Game Learning Inhibitors In Business Simulations And Games New Version Of An Old Simulation Helps Build A Perfect Capstone Course Project Competitor: A Simulation Game For Project Management With 2 Models And 2 Modes Mitigating The Winner’s Curse In The Auction Market Of A Computer-Assisted Business Gaming Simulation Evaluation Of Collaborative Filtering By Agent-Based Simulation Considering Market Environment Marketing Simulation Game Decision Making Experience And Its Impact On Indecisiveness Among Introductory Marketing Students Individual And Organizational Learning In A Top Management Game In Pursuit Of Stockholder Value: Reinforcing Core Concepts In A Business Strategy Simulation With A “Shadow” Stock Market Competition The Simplicity Paradox: Another Look At Complexity In Design Of Simulations And Experiential Exercises In Search Of The Ethnocentric Consumer: Experiencing “Laddering” Research In International Advertising The Paradise Islands Revisited: Trouble In Paradise Developing And Assessing Student Information Literacy Competency Do First Mover Advantages Exist In Competitive Board Games: The Importance Of Zugzwang The Natural Debriefing Approach: A Case In A Simple Business Game Pursued For Perfect Communications New Product Development Simulation Dominance In On Line Business Games Competitions Cooperative Business Game With Framing Effect Experiential Exercises In The Online Environment Enhancing Web-Based Simulations With Game Elements For Increased Engagement The Ginseng Game Experience Business Using a Simple Business Simulation