Marketing Simulation Results as Embedded Forms of Program Assessment MARKETING SIMULATION RESULTS AS EMBEDDED FORMS OF PROGRAM ASSESSMENT David V. Rudd Lebanon Valley College Rudd@lvc.edu Bradley W. Brooks Queens University of Charlotte BrooksB@queens.edu Timothy E. Burson Queens University of Charlotte BursonT@queens.edu ABSTRACT Marketing simulations have been gaining increasing entry into marketing curriculum because they facilitate experiential learning that enhances both quantitative and qualitative skills in marketing decision-making. Previous research has examined the impact of student characteristics, simulation execution, and ancillary pedagogical activities on student learning outcomes. This study, however, calls for an additional stream of research to examine benefits of marketing simulations that may be gaining significant attention in actual practice. The advent of web-based simulations allows publishers to gather data on the performance of the broad range of individual, teams, and schools that participate in their simulation at any one time. Web-based simulation publishers are recognizing that tracking output from simulations could provide benchmarks for program assessment purposes. With the benchmark data so readily available, much research is needed to evaluate how appropriately equipped simulations actually are for this purpose. INTRODUCTION Marketing simulations are becoming an increasingly popular tool among marketing educators for a variety of reasons. As tools that bring students closer to the complex nature of real world marketing decision making, simulations offer professors an alternative method to more traditional approaches such lectures, readings, videos, cases, and experiential learning projects with live clients. Marketing educators are increasingly turning to simulations to facilitate an understanding of marketing that extends well beyond conceptual knowledge. A true understanding of marketing requires the student to develop skills integrating multiple variables while facing significant unknown risks and simultaneously forecasting anticipated results. Such estimated results then must be incorporated into a systematic approach for decision making among mutually exclusive potential alternative actions. Memorizing concepts is simply not sufficient for this level of learning. Zych (1997) sums it well when he points out that, “A recurring concern for educators is how to immerse students in the learning process, rather than have them be passive receptors of theory or knowledge expounded by the teacher” (p. 51). Electronic simulations, therefore, have been extensively studied in regards to their use as an experiential tool for engaging students in the application of concepts beyond the textbook and the classroom. LITERATURE ON “ASSESSING” SIMULATIONS A review of the materials in the Proceedings of the annual conferences of the Association for Business Simulation and Experiential Learning (ABSEL) includes several retrospectives on the research in the field of simulations and experiential learning over the 33 year history of the Association. These include reviews of the contributions of ABSEL research to the understanding of the use of simulations and experiential learning in the 1970s (Graf 1999), in the 1980s (Kelly and Brice 1999), and in the 1990s (Butler 1999); two broad-based content analyses of the ABSEL proceedings (Howard and Strang 2001; 2003); and an expansion of the ABSEL content analysis to include works published in selected journals on the same topics (Strang 2007). The bulk of this research focuses on five topics: 1. The design of various simulations and their “realism.”. 2. Assessment of individual student learning through simulations. Developments in Business Simulation and Experiential Learning, Volume 35, 2008 225 3. Variations in the delivery of the execution in a myriad of specific situations and the impact of those variations on student learning. 4. Student reactions to the simulation experience and the ancillary pedagogies employed. 5. The learning effectiveness of the simulation as measured by individual learning. In the vast majority of cases, the term assessment in this context involves evaluation of individual or student learning. The term Assessment appeared infrequently in the period up to 2003. “Assessment,” as a term in either the title or body of an article, ranked twenty-second among the forty-two most common terms in the Howard content analysis of 2003. In the years since only 13 of 282 papers (3.9%) published in the ABSEL Proceedings contained the term “assessment” with only three apparently linked to program assessment. Howard, Markulis, and Strang (1999) acknowledged that “the AACBS has determined that outcome assessment is an integral—albeit somewhat controversial—part of the continuous improvement process for business schools” (p. 39). They tested the impact of experiential learning on the Major Field Achievement Test (MFAT) published by Educational Testing Service which was and continues to be a significant tool in program assessment. Acknowledging the limits of their sample, they stated that in testing such as this, that there was a need for “a larger, more heterogeneous group of students [who] should better represent the full spectrum of academic preparation.” (p. 39) The advent of web-based simulations allows the gathering of just these kinds of broad spectrum samples. Web-based simulations that are operated centrally (i.e., wherein the publisher processes each set of decisions via the web and reports back the results to the faculty, industry, team, and individuals playing the game) allow the efficient gathering of massive amounts of data on the outcomes of all similar simulations. This gives rise to the potential for the use of comparative data to see how whole programs compare in terms of simulation performance and whether or not simulation performance improves over time. Hornyak, Peach and Snyder (2007), for example, extended Hornyak, Peach, Bowen, Moes and Wheeler’s (2006), in attempting to develop a rubric using simulations to assess “student learning objectives.” Additionally, Hall and Ko (2006) developed a model using simulations to assess the learning assurance process in an Executive MBA program. This could be particularly useful, as students in EMBA programs are sometimes evaluated more subjectively than those in a more traditional MBA program. BENEFITS OF MARKETING SIMULATIONS QUANTITATIVE FORECASTING AND DECISION- MAKING Previous research on marketing-related simulations indicates that simulations are an effective tool in leading students to utilize quantitative data within decision-making. Simulations offer an experiential learning opportunity based on multiple integrated variables within complex, risky situations (Burns and Gentry 1992). As such, this type of experiential learning “simulates” actual marketing decisions complete with reiterative feedback. Herche and Fox (1994) demonstrated that such an interactive process significantly improved undergraduate marketing students’ forecasting and decision-making skills in approaching complex decisions with multifarious variables. QUALITATIVE SKILLS Extensive literature also demonstrates that professors can and do add a myriad of additional aspects to the game administration including periodic results analysis and reporting, competitive analysis, and presentation of final results. Simulations can be quite beneficial in developing the students’ “soft skills” that employers regularly demand and expect from recent graduates. Playing simulation games, for example, enhances the student’s abilities to integrate various decision making aspects of marketing and then to develop a strategy linked to the characteristics of the customers, the dynamics of the markets, and the competitive dynamics of the “industry” in which each game is played. In addition, Lamont (2001) contends that playing the game as a group (small or large) enhances students’ teamwork and collaboration skills. FLEXIBILITY IN MEETING COURSE OBJECTIVES Of late, purveyors of such simulations have responded to the increasing market demands for simulations with versions that can be tailored to the level of the course; executed efficiently in web-based formats; and are flexible in their administration. Key among these flexibilities is allowing the simulation to be “played” by individual students, small groups of students, entire classes, or entire schools. Further flexibility is provided by allowing the “competition” against the computerized market, the professor, other individuals, other small groups, other classes, or other schools. PROGRAM ASSESSMENT – THE NEXT FRONTIER Simulations may offer another potential benefit, however, which has been largely unrecognized within the research literature. As accrediting agencies have pushed for increasing assessment of learning outcomes, business schools are continually searching for new and better ways to accomplish their requirements. Marketing simulations could be used as embedded forms of program assessment, which would give administrative value to a tool already Developments in Business Simulation and Experiential Learning, Volume 35, 2008 226 proven useful in the classroom. The purposes of this study, therefore, are to propose the relationship between assessment and simulations and to call for additional research in this area. INCREASING ASSESSMENT NEEDS FROM ACCREDITING BODIES Continuous assessment of learning outcomes is a must for business schools. Accrediting bodies, whether regional, national or international, require certain standards of assessment. As stated on its website, AACSB International, for example, demands that the “definition of learning expectations and assurance that graduates achieve learning expectations are key features of any academic program” (www.AACSB.com 2007 p. 59). AACSB goes on to require that “learning goals should be set and revised at a level that encourages continuous improvement in educational programs” (www.AACSB.com 2007 p. 59). In previous years, many schools used a standardized test, such as the MFAT, as their primary assessment instrument. The test, which business students would typically take late in their senior year, would be a comprehensive vehicle, incorporating questions from the various business disciplines. The school would receive individual and aggregate scores, and be able to compare the performance of their students to that of other participating schools. The test would require minimal effort from the school, and the consistency of the exam would allow easy year-to-year comparisons, both internally and externally. Banta (2005), however, points out two major problems in this type of assessment vehicle, the first having to do with comparisons. The questions are generic by nature, not specific to what has been taught at the school. A liberal arts school, for example, might have its student results compared to students at research universities or community colleges, comparisons which are not terribly useful. The other problem has to do with incentives; students are required to take the exam, but there is typically no “grade” associated with the exam, thus no consequence to an indifferent effort. An ancillary approach might have been an exit survey, which allowed administrators to craft questions that were program specific, but allowed no comparisons between students at other institutions. Accrediting bodies appear to be putting increasing emphasis on assessment and Schamber and Mahoney (2006) note that universities have thus become much more concerned with outcomes assessment in recent years. This has led to a more concerted effort on the part of business schools to come up with advanced and improved approaches to assessment. Some schools have had success involving faculty on a greater level. (See, for example, Riordan 2006 and Bennion and Harris 2005.) As experts in their curriculum areas, faculty understand what skills their students need to have as exiting graduates, and should, therefore, be better equipped to assess than administrators. Faculty involvement also carries the additional advantage of forcing the instructors to think more “globally” about learning outcomes of the entire program rather than just assessing their individual classes. Schools have also found success with assessment vehicles that involve “benchmarking,” where a school compares its results with similar institutions. In drawing external comparisons, benchmarking addresses one of the criticisms of the standardized exam. Laufgraben et al. (2004) note that benchmarking also allows a school to identify the top performers, to share information with specific institutions, and to affect systematic changes. Benchmarking can work on a macro level, with consideration of an overall program, or on the micro level of an individual class. It appears that some of the more effective and innovative assessment vehicles produced in recent years have involved faculty in the development process, have related directly to the college’s program, and have allowed meaningful comparisons with other institutions. POTENTIAL AS AN ASSESSMENT TOOL . The major benefits for the use of simulations as assessment tools are that the omnibus benchmark data is available at virtually no additional cost; the data comes as an integral part of activities that are already scheduled into the course structure; and (in most cases) the analysis and reporting of the data is relatively easy. In general, simulations have been used within the structure of a course to assess individual student and group performance in a “real world” analysis and decision making environment. With the advent of web-based simulations, the game operators have the ability to capture decision-by-decision results for a large number of players, arrange that data into convenient and easily conveyed performance metrics, and track relative performance of players from literally around the world. Consider a couple of examples from two of the more popular simulations currently on the market. One particular simulation provides specific details of the relative performance of the firms within each industry as expected, but also provide omnibus benchmark comparisons on a number of dimensions using a three-level performance metric. 1. The top firm in each industry compared with the mean of the top firms in each industry world-wide. 2. The average firm in each industry compared with the mean of the average firms in each industry world-wide. 3. The lowest firm in each industry with the mean lowest firm world-wide. Among the measures are: 1. Overall firm performance as measured by a ratio with specific standard (benchmarked) performance rating characterizing “good” performances as well as top performances. 2. Financial performance as measured by: a. Earnings Per Share b. Return on Average Equity c. Stock Price Developments in Business Simulation and Experiential Learning, Volume 35, 2008 227 http://www.aacsb.com/ http://www.aacsb.com/ d. Credit Rating 3. Marketing effectiveness as measured by Image Rating This particular simulation draws its omnibus benchmark data from 1400 groups in 900 schools world-wide with a total student count well above 9,000. A different, but similarly popular, simulation provides these same data parameters consistent with its simulation in reporting omnibus global benchmark data. In addition, this particular simulation provides a series of optional on-line surveys and quizzes that measure individual student skills in the following areas: 1. Leadership 2. Collaboration and Teamwork 3. Financial Analysis 4. Operations Management 5. Marketing Management 6. Human Resources Management 7. Strategic Analysis and Planning Individual scores are provided and the simulation provides an overall “Class average” for each category. The results are reported in percentile rank against a very large, global sample with over 30,000 students in 25 countries. The authors concede Brooks, Burson, and Rudd’s (2006) point that the game conditions, game administration, and ancillary activities can play a large part in the “results” of any single simulation game. Game conditions include who is playing, whom they are playing against, the role that information plays in the game, and the level of complexity of the simulation. Game administration includes the role the professor plays in managing the game, the use or non-use of dynamic controls to manage the market, and the number and size of the groups competing in any one game administration. Ancillary activities include “practice decisions” prior to the actual start of the game, periodic reports to the professor concerning the strategy underlying the decisions, publishing formal performance reports, and expecting formal presentations of final results and interpretation of those results. However, unless faculty make radical changes in their individual approach to the conduct of a simulation in a particular course, the trend in the various performance metrics could still serve as an effective program assessment tool. Furthermore, even if individual faculty make significant and radical changes in game conditions, game administration, and/or ancillary activities from one term to the next or one year to the next, knowledge of the impact of these changes would still allow the performance metrics to be used for program assessment. Either case supports added research into the impact of these factors (and others) on program performance metrics. Progress in customizing benchmarks has been made by other assessment methodologies (such as the Educational Benchmarking Institutes (EBI) exit survey for undergraduate business and accounting majors). It may well be that simulation publishers will soon build appropriate subsets like the EBI’s Select Six and Carnegie Classification options. In the meantime, there is still value in using the omnibus benchmarks that are available as the basis for assessment. There are three arguments to support the value of comparing firm performance to the omnibus benchmarks: 1. Consistent use of the simulation in the same manner should still allow the professor to examine trends in his/her own group and firm performance over time. 2. If the professor makes changes in the use of the simulation and experiences changes in firm performance, measuring changes in performance versus the omnibus benchmarks can serve as some indication as to whether the performance change was due to student issues or not. For example, should a professor choose to slow the growth of a particular segment, a significant decrease in firm performance versus the omnibus benchmarks could result. An examination of the changes made could indicate that too many firms crowded into a market with inadequate total growth and inadequate segment growth to support competition in each segment. Such circumstances could result in a mass commoditization of the industry and the resultant expected decline in margins and profits for all players regardless of strategy. 3. Use of the omnibus standards can allow the professor to examine the extent to which firms are using end-game strategies, independent of the actual game conditions to optimize the anticipated measures that the professor will gauge in measuring “success”. After the simulation ends, the professor could repeat the decisions from the end point (i.e. continue the strategy for multiple periods) and judge the robustness of the end strategy. REQUIREMENTS AS AN ASSESSMENT TOOL These benefits dovetail well with the demand for additional learning outcomes assessments to compliment current methodologies. The authors propose that, in order to be effective at measuring assessment, at the very least simulation results benchmarked to a broader base of players would be required to: 1. Serve as a core measure of how well different players integrate the various disciplines involved in marketing decision making. 2. Serve as a surrogate measure of effective team work skills. 3. Provide professors with rudimentary measures of how the performance of students in their programs compares with others. 4. Provide programs with secondary measures of student learning in the marketing discipline. 5. Provide programs with secondary measures of students understanding of the impact of marketing decisions on corporate financial performance. Developments in Business Simulation and Experiential Learning, Volume 35, 2008 228 All of these are measures of the higher levels of Bloom’s Taxonomy of Learning (application, analysis, synthesis, and evaluation) which have proven to be difficult to assess. Cadotte (1995) points out that Miyamoto Musashi’s classic strategy tome, A Book of Five Rings, states that mastery of strategy only occurs through constant training and tireless practice. Similarly, Senge (1990) states that training, not study alone, is what changes a person. Cadotte (1995) also adds that “sophisticated simulations offer students the opportunity to practice a number of important skills, including strategic planning and thinking; management strategy; leadership, teamwork, and interpersonal skills; budgeting and cash flow management; and understanding and delivering customer value” (p. 11). Having useful benchmarks (omnibus and customized) of student firm performance would greatly enhance the ability to assess performance against this complex set of critical skills. To facilitate the identification of key variables in the move toward simulation benchmark subsets, the authors are proposing a research program that would survey simulation game operators (professors) about the administration of their simulations; survey student perceptions of learning on a series of measures; and link these results with the actual simulation results and the benchmarks derived from these results. The over-riding objective is to determine how well the current versions of complex marketing simulations can also simultaneously serve as embedded assessment measures of program performance on selected disciplinary topics and on sought after soft skills. Doing so should have significant implications throughout business programs as many programs have already begun utilizing simulations as assessment tools and almost all business schools are continuously seeking effective assessment approaches. REFERENCES www.AACSB.com (2007), “Eligibility Procedures and Standards for Business Accreditation,” AACSB International, revised: January 31, 2007, 59. Banta, Trudy W. (2005) “How Much Have We Learned?” BizEd, (September/October), 35-38. Bennion, Donald H. and Michael Harris (2005), “Creating an Assessment Culture at Eastern Michigan University: A Decade of Progress,” Assessment Update, (March/April), Vol. 17, No. 2, 7-9. Brooks, Bradley W., Timothy E. Burson, and David V. Rudd (2006), “Addressing Current Research Gaps and Directions in Educational Marketing Simulations,” Journal for the Advancement of Marketing Education, 9 (Winter), 43-49. Burns, Alvin C. and James W. Gentry (1992), “Computer Simulation Games in Marketing: Past, Present, and Future,” Marketing Education Review, 2 (Spring), 3-13. Butler, John, Jr. (1999) “ABSEL’s Contributions to experiential Exercises in the 1990’s,” Developments in Business Simulation and Experiential Learning 26, 16- 24. Cadotte, Ernest R. (1995), “Business Simulations: The Next Step in Management Training,” Graduate Management Admissions Council, Autumn, 8-19. Graf, Lee A. (1999) “ABSEL’s Contributions to Experiential Learning/Exercises: The Decade of the 1970’s,” 26, 23-31. Hall, Owen P. Jr. and Kenneth Ko (2006), “Learning Assurance Using Business Simulations Applications to Executive Management Education,” Developments in Business Simulation and Experiential Learning, 33, 1- 6. Hornyak, Martin J., E. Brian Peach, Adam Bowen, William Moes, and Robert Wheeler (2006), “Examining Program Management in Business Simulations: Student and Faculty Views,” ,” Developments in Business Simulation and Experiential Learning, 33, 107-117. Hornyak, Martin J., E. Brian Peach, and Stephen Snyder (2007), “Assessment and Simulations: Measuring the Academic Learning Compacts Within,” Developments in Business Simulation and Experiential Learning, 34, 215-222. Howard, Barbara, Peter Markulis, and Daniel Strang (1999), “A Study of the ETS General Field Test as an AACSB Assessment Tool and the Impact of Experiential Exercises and Simulation on Learning,’ Developments in Business Simulation and Experiential Learning, 26, 33-39. Howard, Barbara and Daniel R. Strang (2001), “The Way We Talk,” Developments in Business Simulation and Experiential Learning. 28, 111-116 Howard, Barbara and Daniel R. Strang (2003) “The Way We Talk, Take II,” Developments in Business Simulation and Experiential Learning, 30, 133-145 Herche, Joel and Marjorie Fox (1994), “An Evaluation of Improved Decision Making in a Competitive Experiential Environment,” Marketing Education Review, Vol. 4 (Spring), 29-34. Kelly, Lane and William David Brice (1999) “The Contributions of ABSEL During the 1980’s, Developments in Business Simulation and Experiential Learning 26, 9-15. Lamont, Lawrence M. (2001), “Enhancing Student and Team Learning with Interactive Marketing Simulations,” Marketing Education Review, Vol. 11 (Spring), 45-55. Laufgraben, Jodi Levine, with Joseph A. Pica and Randy L. Swing (2004), “How Benchmarking Can Help Us Improve What We Do,” About Campus, (January- February), 4-11. Riordan, Catherine (2005), “Encouraging Faculty to Play Key Role in Assessment,” Academic Leader, Vol. 21 (May), 6-8. Schamber, Jon F. and Sandra L. Mahoney (2006), “Assessing and Improving the Quality of Group Critical Thinking Exhibited in the Final Projects of Collaborative Learning Groups, The Journal of General Education, Vol. 55, No. 2, 102-126. Developments in Business Simulation and Experiential Learning, Volume 35, 2008 229 http://www.aacsb.com/ Senge, Peter M. (1990), The Fifth Discipline: The Art and Practice of the Learning Organization (New York, Bantam, Doubleday, and Dale). Strang, Daniel (2007) “ANSEL Research – One Additional Perspective on Where We are and Where We have Come From,” Developments in Business Simulation and Experiential Learning, 34, 69-75. Zych, John M. (1997), “Adding Case Materials to Reinforce the Realism in Marketing Simulations,” Marketing Education Review, Vol. 7 (Spring), 51-61. Developments in Business Simulation and Experiential Learning, Volume 35, 2008 230 Table of Contents Volume 35, 2008 Linking Team Covenants To Peer Assessment Of Simulation And Experiential Performance Using the Balance Scorecard Approach: A Group Exercise IndoAmerican Enterprises Class Size and Game Design Implementation Of Effective Experiential Learning Environments Simulation Sensemaking: The BusinessWeek Approach To Effective Debriefing A Lesson in Hide-And-Go-Seek: A Team Building Game Thoughts On How To Motivate Students Experientially Experiential Learning Is Not Just Experiential Teaching: Measurement of Student Skill Acquisition via Assessment Centers ABSEL Redux: Reflections after a 25 Year Hiatus College Student's Expectations of Technology- Enhanced Classrooms: Comparing 1996 and 2006 The Business Student Satisfaction Inventory (BSSI): Development and Validation of a Global Measure of Student Satisfaction The "Big Picture Question" Project: Explorations in Teaching Creativity within a Force-Field Research Framework "Viva Voce": Oral Exams as a Teaching & Learning Experience Internships And Occupational Socialization: What Are Students Learning? Does Learning Occur in One-Shot, Non-Cooperative Games? Beliefs and Behavior, an Ancient Perspective and Modern Application Developing Enterprise Culture Among the Students Through Intercollegiate Competitions: A Case of Student Enterprise Competition (SEC) 2007 Student Views of Management Skills and Their Future Careers after Using Business Simulations Back to the Future: Gender Differences In Self-Ratings of Team Performance Criteria A Case for Experiential Learning: Using Central Europe as a Classroom Modeling Strategic Opportunities in Product-Mix Strategy: A Customer- Versus Product-Oriented Perspective Assessment in the Modern Large High-Tech Classroom Target Profit Pricing With the Web-Based Breakeven Analysis Package Are the Business Simulations We Play Too Complex? Shared Experience as Incentive for Horizontal Integration in Business Simulations Affinity Propagation: A Clustering Algorithm for Computer-Assisted Business Simulations and Experiential Exercises Marketing Simulation Results as Embedded Forms of Program Assessment Human and Agent Playing the "Beer Game" Issues in Porting a LAN-based Total Enterprise Simulation Game to a Web-based Environment Partners or Competitors? A B2B Simulation Evaluation Model of the Global Performance of a Management Simulation for the Academic Environment Applying Bloom's Revised Taxonomy In Business Games Do Price Strategies Work in Business Simulations? Early Japanese Gaming Simulation Efforts Using Par Players to Enhance Learning in Business Simulations Corporate Cartooning: The Art of Computerized Business Simulation Design Should Business Game Players Choose Their Teammates: A Study with Pedagogical Implications Goal Orientation and Simulation Performance Design and Demonstration of an Online Managerial Economics Game with Automated Coaching For Learning and Graded Exercises for Assessment Are Good Strategy Decisions Consistently Good? A Real-Time Investigation Active Learning 2.0 or Wiki is not a 4-letter Word Comparing Student Learning in Online and Classroom Formats of the Same Course How Do We Get To Tomorrow? The Path to Online Learning Similar Media Attributes Lead to Similar Learning Outcomes Facilitating Business Gaming Simulation Modeling