Assessing the Role of Assessment in Business Simulations Page 124 - Developments in Business Simulation and Experiential Learning, volume 42, 2015 ABSTRACT This study looks at the role that simulations play in assess- ment. Schools of Business are under increasing pressure to demonstrate that they have in place protocols and policies that provide an assurance of learning (AoL) in their courses and their programs. Business simulations have a potentially power- ful role in facilitating the assurance of learning process. This study examines that role by first listing the more popular simu- lations used in business schools and then reviewing the AoL capabilities of these simulations. Forty nine simulations were scrutinized and an attempt was made to determine to what ex- tent use of the simulations and the materials accompanying these simulations might be a viable component of an AoL pro- gram. Information for the study was obtained from simulation websites as well as from contact persons associated with or representing various simulations. Results indicate that while many simulation providers offer AoL modules as part of their simulation packages, they do not always correspond neatly to the learning goals of business schools’ programs or courses. INTRODUCTION Simulations are a fantastic way to integrate many of the concepts and skills students have learned from their collective coursework. Simulations create an experiential learning envi- ronment where students have an opportunity to demonstrate their understanding of learning objectives which business schools have deemed important. Further, simulations provide a potentially powerful tool for faculty to assess and document student learning. As Anderson and Lawton note, “Since their inception in the late-1950s, business games have been recog- nized as being problem-solving exercises” (2004). Accreditation associations have made assessment of student learning a priority for the certification of business schools. Gov- ernmental bodies, as well as college accrediting agencies, re- quire business programs to define learning goals and objectives at both the college and course level (Bollag, 2006; Mundhenk, 2005). These learning objectives must be measured and as- sessed using various tools; such as, rubrics, tests, simulations, presentations, etc. Some have even proposed the use of class- room clickers for instantaneous assessment (Markulis & Strang, 2008). As the need for assessment of learning goals have grown, the companies which provide pedagogical materials, such as textbooks and simulations, have been incorporating assessment modules and protocols into their products. This study has two goals. First, to compile a list of popular simulations, and second, to list the assessment components that these simulations offer. In terms of the second goal, the authors rely principally on what business simulations say about them- selves and about how their simulations can be used to address assessment. A question raised, but unanswered is: Have simu- lation providers taken the time to carefully craft an assessment strategy that will meet the accreditation needs and standards of individual business schools? ASSESSMENT & SIMULATIONS “Business schools are now quite familiar with the AACSB mantra: Assessment! The AACSB believes that assessment leads to--or at least promotes--“better” learning on the part of students, (Martell & Calderon, 2005, 2). Assessment (often re- ferred to as Assurance of Learning of AoL) is defined as “an ongoing process aimed at understanding and improving student learning,” (Angelo, 1995, 10). Since 2002, the AACSB has sponsored numerous assessment seminars, as do other accredit- ing organizations, so that schools can learn how to integrate meaningful AoL programs into their schools. According to Martell and Calderon, most business schools have completed the initial steps and some type of assessment program is in place (2005, 21). What schools of business are now wrestling with is how to alter or augment their present curricula in order to take advantage of assessment results, referred to in the as- sessment literature as “closing the loop” (Martell & Calderon, 2005, 8).” More recently, the AACSB outlines the assessment process in a paper entitled “Assurance of Learning Standards” (2013). ASSESSING THE ROLE OF ASSESSMENT IN BUSINESS SIMULATIONS Peter M. Markulis SUNY Geneseo Markulis@Geneseo.Edu Michael Nugent SUNY Stony Brook Michael.Nugent@Stonybrook.Edu Daniel R. Strang SUNY Geneseo Strang@Geneseo.Edu mailto:Markulis@Geneseo.Edu mailto:Strang@Geneseo.Edu Page 125 - Developments in Business Simulation and Experiential Learning, volume 42, 2015 The outcomes assessment process should include: 1. Definition of student learning goals and objectives 2. Alignment of curricula with the adopted goals 3. Identification of instruments and measures to assess learning 4. Collection, analyzing, and dissemination of assessment information 5. Using assessment information for continuous improve- ment including documentation that the assessment pro- cess is being carried out in a systematic, ongoing basis. Another form of the above steps can be stated as: 1. What will our students learn in our program? What are our expectations? 2. How will they learn it? 3. How will we know they have learned it or not? 4. What will we do if they have not learned it? Simulations are a popular mode of instruction for many business programs and courses. But simulations are often used outside the classroom and conducted using teams. In a FAQ section, the AACSB answers two very important questions for using simulations for assessment. “Can assessment take place outside of the business school? Yes, but the learning goals that are being assessed should be relevant for business and account- ing students and established (agreed upon) by the facul- ty.” (2013). We interpret this to mean that faculty can only use simulation assessment modules if there is alignment of the course and or schools learning goals. A further question is: “Can the collective work of student teams be used for as- sessment? Collective work from a student team does not provide a basis to assess individual student performance and outcomes, except where teamwork is a learning goal. In that case, the col- lective work of the team may provide a basis for assessing per- formance as a team member.” This is an important issue. Facul- ty must make sure that the assessment module of any simulation is individually-based, unless they are assessing teamwork (AACSB, 2013). STUDENT ASSESSMENT LITERATURE REVIEW Accreditation should not be the sole reason for assessment in the classroom. There are many papers written which demon- strate a strong benefit from the assessment of student learning. We have highlighted a few key points below from some of these works. "Using effective assessment techniques can improve an instructor’s understanding of student needs and support learner- centered classrooms." (Vonderwell & Boboc, 2013) This state- ment is the baseline for most of the literature and research on student assessment. Additional support for this concept comes from a paper by Crisp & Ward entitled “The development of a formative scenar- io-based computer assisted assessment tool.” Their research showed a connection between continuous feedback and en- hancement of student engagement, improved achievement and increased motivation to learn.” (2008) Palomba and Banta in their paper “Assessment Essentials” define the outcomes assessment process as: The systematic col- lection, review, and use of information about educational pro- grams undertaken for the purpose of improving student learning and development (1999). To be sure, an important aspect gleaned through the litera- ture review is the difference between assessment of learning and assessment for learning. W. Popham, writes, “Assessment of learning is focused primarily on assigning grades as the prin- cipal indicator of student performance, while assessment for learning is when the students’ status with respect to educational variables of interest is determined” (1999). This can be inter- preted to mean that assessment for learning is encouraging stu- dents with feedback to help them to better understand what they have learned and what they still have yet to learn, and providing them a path to achieve the desired learning objectives. Faculty should consider assessing student learning for improving course content and student learning. Simulation developers and users are quick to tout the bene- fits of using simulations for assessment purposes. “Among the first to note that games [simulations] might be useful in assess- ment were Keys and Wolfe, who wrote that "management games will play a more significant role in management develop- ment and assessment efforts in business schools as part of the move toward competency-based outcome measurement" (1990, 324). “Simulations as assessment tools provide the advantage of measuring dynamic performance rather than simply static knowledge. This characteristic also makes them superb peda- gogical devices. The difficult decision is arriving at the proper criteria and criteria levels to use as hurdles.” (Fritzsche, 1997). “In terms of testing or assessment of learning, quantita- tive performance in the simulation games is typically used to measure or infer the degree of learning derived from the game (Anderson, Cannon, Malik, & Thavikulwat, 1998). METHODOLOGY The authors started by trying to compile a list of the busi- ness simulations being used by schools of business. As it turned out, this was not a trivial task. First, several ABSEL members were asked to contribute the names of simulations with which they were familiar. Then a lengthy internet search was conducted which resulted in a reasonably extensive list of simulations as well as the contact information relating to the simulations (Table 1). The authors then send emails to many of the simulation providers with a list of questions pertaining to the simulation's assessment capabilities (see Appendix A). In some cases, the authors received a reply from an official contact person (Appendix B), but in other cases, no reply was forth- coming. In a few cases, the authors and the contact person ex- changed several emails regarding the assessment questions. Table 1 lists 49 entries. We say entries and not simulations because there is an important proviso to this picture. Some of the websites are actually umbrella sites, which when contacted, list many simulations as part of their offerings but under a sort of “umbrella-type” website. For example, Innovative Learning Solutions is the name of an umbrella organization which then lists the trademarked name of Marketplace Business Simula- tions which itself has 23 simulations. Further, Marketplace Business Simulations has its own website. Further complicating matters, some of the simulations listed on umbrella websites Page 126 - Developments in Business Simulation and Experiential Learning, volume 42, 2015 TABLE 1 BUSINESS SIMULATIONS & WEBSITES BizCafe --https://www.interpretive.com, Interpretive Simulations BOSS -- Blue Ocean Strategy Simulation, web.stratxsimulations.com BSG -- Business Strategy Game https://www.bsg-online.com (McGraw-Hill) BSS -- Business Strategy Simulations –http://www.business-smart.com Capsim -- Capsim Business Simulations, www.capsim.com Celemi -- http://celemi.com/simulations-services/business-simulations/ Cesim -- Global Strategy, http://www.cesim.com/ COMPETE -- A Dynamic Marketing Simulation, Richard D. Irwin, Inc. DECIDE -- http://www.webdecidesim.com/ EthicsGame -- https://www.ethicsgame.com European School of Management and Technology -- https://www.esmt.org/marga Executive Challenge -- http://www.enspire.com/executive-challenge/ GBG -- Global Business Game, http://onlinegbg.com/ Global Tycoon -- http://www.intellisim.com/software.html Gold Simulations --http://www.goldsimulations.com/ GoVenture -- http://www.goventure.net Harvard Business Publishing for Educators -- http://hbsp.harvard.edu/list/simulations HTW -- www.howthemarketworks.com Investopedia -- www.investopedia.com/simulator Interpretive Simulations -- https://www.interpretive.com Innovative Learning -- http://ilsworld.com KnowledgeMatter, Inc. -- http://www.knowledgematters.com LAPTOP -- A Marketing Simulation, Richard D. Irwin, Inc. Marketplace Live -- http://www.marketplace-live.com Marketplace Sim. – Innovative Business Solutions, http://www.marketplace-simulation.com/ Merlin -- http://www.mcgraw-hill.com.au/html/9780072946581.html Mike's Bikes -- http://www.smartsims.com/ MMS -- The Marketing Management Simulation, The Simulation Source Marketplace, https://www.interpretive.com Oaktree -- http://www.oaktreesim.com/ OB Sim -- http://obsim2.geneseo.edu/ ProSim www.knowledgematters.com SMS -- The Sales Management Simulation, South-Western Publishing Company SimVenture, simventure.co.uk Simulations Harvard Business, hbsp.harvard.edu/list/simulations SmartSim --http://www.smartsims.com/ Smart Stocks -- www.smartstocks.com Stocktrak -- www.stocktrak.com StockMkt – Stock Market Game, http://www.smgww.org/ StockMS – Stock Market Simulation, http://www.nationalsms.com/ SMS -- Strategic Marketing simulation, http://www.marketplace-live.com/ StratSimMg-- StratSimManagement: The Management Strategy Simulation, https://www.interpretive.com, Interpretive Simulations. StratSimMk -- StratSimMarketing: The Marketing Strategy Simulation, https://www.interpretive.com, StratX -- http://web.stratxsimulations.com/ Tata-- http://www.tatainteractive.com/business-simulations.html The Business Strategy Game -- https://www.bsg-online.com Towson University -- http://pages.towson.edu/precha/GEO/index.htm Virtual Leader -- http://www.simulearn.net/ ZBS -- Zoom Business Simulation, www.jupiterinteractive.net http://web.stratxsimulations.com http://www.capsim.com http://www.cesim.com/ http://www.webdecidesim.com/ https://www.ethicsgame.com http://onlinegbg.com/ http://www.goldsimulations.com/ http://www.howthemarketworks.com http://www.investopedia.com/simulator http://www.marketplace-simulation.com/ http://www.mcgraw-hill.com.au/html/9780072946581.html http://www.smartsims.com/ http://www.oaktreesim.com/ http://www.knowledgematters.com http://simventure.co.uk http://hbsp.harvard.edu/list/simulations http://www.smartsims.com/ http://www.stocktrak.com http://www.smgww.org/ http://web.stratxsimulations.com/ http://www.tatainteractive.com/business-simulations.html http://pages.towson.edu/precha/GEO/index.htm http://www.jupiterinteractive.net Page 127 - Developments in Business Simulation and Experiential Learning, volume 42, 2015 have their own separate websites as well. This obviously makes compiling a complete, non-duplicated list rather difficult. In order for us to completely capture all of this information, we would have had to construct some type of fairly elaborate tree/ branch diagram, which we felt was beyond the scope of this paper. In some instances, the promoters of the simulations address the issue of assessment in the materials that they place on their websites. In other cases, no assessment information is provided on the website. To fill in these gaps as well as to gather addi- tional assessment information, the authors used the contact in- formation on the website to reach an official representative of the simulation. As mentioned earlier, in some cases this was successful, in other cases, it was not. Thus, this approach pro- vided information that was far more limited in scope than the authors had hoped for when they began this project. That said, there are some insights that can be gained from the re- sults. Table 2 presents the results on this second phase of anal- ysis. Although these results do not represent an exhaustive list of the business simulations currently in use, they may provide some real insights in terms of directions and dimensions in which simulations might be expected to help address the need for business school assessment. Table 2 details what the includ- ed simulations have stated (or their contact persons have stated) in terms of how their simulation (or simulations) address the issue of assessment. It should be noted that the “yes” designa- tion simply means that the website (or contact person) has stat- ed that the simulation in question does have an assessment mod- ule (or modules). On the other hand, it should not be taken that just because the website (or contact person) did not address or mention assessment, that the simulation does not contain an assessment module. It simply means that we were unable to obtain the information. Below are some of the major findings (with some commen- tary) from TABLE 2.  The authors found only one simulation that had a separate charge for their assessment modules (CAPSIM)  Several simulations maintained assessment databases whereby comparisons could be made between one’s own student’s assessment results and the results of all students using the simulation's assessment modules.  Most assessment modules seemed to be in the form of a series of multiple-choice questions pertaining to what the simulations purported to “teach.” In other words, fact- based or knowledge-based questions.  Two simulations (The EthicsGame & OBSim) reported that they use written essays with rubrics to conduct assessments for both teams and individuals or both. OBSim allows for the instructor to tailor the rubric to meet course or program- matic assessment needs.  We do not have an exact count on this, but many simula- tions are starting to develop single-user versions, perhaps due to the fact that assessment requires individual assess- ments. For example, in Thavikulwat’s GEO game (http:// pages.towson.edu/precha/GEO/index.htm), individual users compete against the computer.  We have not seen any Accreditation agencies officially endorse the assessment modules of any simulation. OBSERVATIONS OF TWO PREDOMINANT SIMULATION ASSESSMENT TOOLS Capstone’s “Comp-Xm” and Zoom Simulation’s “Individual Assessment“ are two examples of simulation as- sessment tools. One of the authors adopted the role that students take and to complete both assessment tools to better understand their assessment potential and also, to specifically see if they met AoL standard 8, which covers areas of teaching and learn- ing assessment requirements. Standard 8 states, “The school uses well-documented, systematic processes for determining and revising degree program learning goals; designing, deliver- ing, and improving degree program curricula to achieve learn- ing goals; and demonstrating that degree program learning goals have been met.” (AACSB, 2013) For the assessment modules to be successful in meeting AACSB standard 8, the learning goals must be measured and documented. As stated in AACSB report, “Eligibility Proce- dures and Accreditation Standards for Business Accreditation” “Learning goals state the educational expectations for each de- gree program.” (AACSB, 2013) After students complete Capsim’s Capstone simulation-- usually as part of a student team--they can work on an individu- al simulation assessment tool called Comp-XM. This is a sepa- rate module that requires students (or institutions) to pay extra for. The Comp-XM starts out as a simulation; students play the role of the CEO of a new company, the Andrews Corpora- tion. Students must complete four rounds of decisions. The simulation and its decision points are identical to the decisions they made while playing the team version of the Capstone simu- lation. They play as an individual student against the computer. The Comp-XM simulation is basically a scaled down version of Capstone. The second section of CompXM is a series of quizzes, called “Board Queries.” The “Board Queries” are web-based quizzes, true/false and multiple choice, that relate directly to the results of the simulation. The scenario sets up the student as the CEO of the company who reports to the Board of Directors. This Board will ask five sets of questions, or “Board Queries”, that are based on the results of the four rounds of the individual play. The “Board Queries” ask specific questions about the fi- nancial results of the company. The test bank questions in- clude: a break-even analysis, effects on capital budgeting, finan- cial ratios, effects of changes in assets, liabilities and equity. The “Board Queries” also ask specific questions about product profit margins and revenues from the simulation results. Faculty are able to select the questions that students will work on, giving faculty a chance to align course learning goals with Comp-XM board queries. These questions will incorporate data from the student’s simulation results into the quantitative questions, which can reduce the ability of students to cheat. The Comp-XM is an integrated evaluation tool that focuses on assessing the student’s basic business knowledge. Faculty might have a challenge aligning their program or course learn- ing goals with the Comp-XM’s “Board Queries” test bank. In our opinion, Comp-XM does a good job evaluating how well students know the capstone simulation and the basics of finan- cial analysis. Comp-XM is a convenient tool, however, faculty could http://pages.towson.edu/precha/GEO/index.htm Page 128 - Developments in Business Simulation and Experiential Learning, volume 42, 2015 TABLE 2 SIMULATIONS & ASSESSMENTS Capsim has several simulations. Yes (Assessment Modules sold separately for $14.99) Celemi has several simulations. Yes, for most of our simulations. The purpose of any simulation is to let you experiment with reality – without ruining your business. Our business simulations are not just games; they are self-contained learning experiences that recreate real workplace or operational environments, and give your people the opportunity to test new skills and knowledge – and make mistakes – in a fun, safe environment. Cesim has several simulations – Yes, All Cesim business management simulation games are built on a powerful, stable and highly scalable online platform, which allows instructors to run their courses from one centralized interface. Compete: A Dynamic Marketing Simulation—One simulation. EthicsGame has several simulations. Yes, 2 assessment modules require written essays. Executive Challenge. Yes Global Tycoon -- http://www.intellisim.com/software.html GoldSimulations, has several simulations. Yes, Exceptional gaming simulation that students can interactively learn the microeco- nomic cause/effect relationships and learning outcomes. Very engaging and effective game simulation for students in high school to college. GoVenture has several simulations. Yes, GoVenture includes software simulations and applications, mobile games, board games, and soon a massively multiplayer online game. GoVenture programs are designed for youth and adults, and for self-directed or facili- tated learning. They can be used on their own or as components to enhance other courses, learning, and entertainment experiences. Harvard Business Publishing for Educators. Multiple simulations. Yes, new online simulations from Harvard Business Pub- lishing use real-world contexts to reinforce student learning. How the Market Works—one simulation. Innovative Learning Solutions. 23 simulations. 10,300 students have used the assessments (both objective questions and MCs) Interpretive Simulations has several simulations. JupiterInteractive—one simulation. Yes. Learning assessment tools for review The Zoom Business Simulation developers understand that learning must be assessed through performance. Therefore, this simula- tion documents what students can do with their learning. Assessing students’ performance is built directly into the simulation. Each student can monitor their progress through the simulation, and see how well they are meeting the learning goals set forth in the Zoom simulation through our learning assessment tools. Instructors can review each student's progress as well. Benchmarked assessment data can be downloaded directly to excel and used to meet most accreditation assessment requirements. KnowledgeMatter, Inc.—one simulation. Yes. Restaurant is a series of simulation-based assignments and projects that give your students hands-on experience running a restaurant. ProSim - Restaurant consists of ten assignments and two projects that are meant to work with your current curriculum. Marketplace-Livehas several simulations. Yes. Oaktree Simulations, multiple. Yes. OBSim, multiple versions of the same simulation. Yes SimVenture—one simulation. Yes. Since SimVenture collects user data in different formats, teachers and trainers have various op- tions to assess performance. When it comes to 'what' is measured teachers and trainers typically want users to demonstrate what's been learnt from using the resource. The software's data analysis tool allows for quick collection and analysis of user performance data. Smartsims Business Simulations has several simulations. Yes STRATX Simulations has several simulations. Yes. Str atX’ simulations not only have great pedagogical value, but they also create a fantastic competitive spirit and class stimulation. Whether used as a business game for an inter business schools competition or as a social responsibility challenge, they prove to bring memorable business experiences. TATA Interactive Systems has several simulations. Yes. Simulations offer learners the ability to learn in a realistic environ- ment where they can apply knowledge and skills without the fear of real-world implications. The Business Strategy Game—single simulation. Yes. Two assessment reports are generated for course instructors at the end of the simulation. The first assessment report measures how the class as a whole performed in the simulation versus all of the classes that used The Business Strategy Game within the last 12 months. Each class is compared to the global high, low and average on 6 varia- bles. Towson University—single simulation, yes (oriented toward single users). Virtual Leader—yes Page 129 - Developments in Business Simulation and Experiential Learning, volume 42, 2015 easily duplicate the assessment themselves by creating an exam that would ask specific questions about the simulation decisions as they relate to the learning goals of the course. At the end of the day, an exam is not necessarily an assurance of learning unless it specifically measures the program or course learning goals, as explained in AACSB Standard 8. (AACSB 2013) The Zoom Simulation assessment is similar to the Comp- XM. Students must complete an individual, self-paced simula- tion against computer players. Here again, the simulation and its decision points are identical to the decisions they made while playing the team version of the simulation. What is unique about this assessment tool is the reflective observations questions, which ask students to express their un- derstanding of key aspects of the simulation and business knowledge. Many questions focus on company performance, asking students to detail their strategy and how they will im- prove in the following round. Here is an example reflective ob- servation questions from the sales department page, “Describe your new strategy for redesigning your economy class vehicle. How will this help your company's performance? What factors influenced your calculation of the sales forecast for the econo- my class vehicle? Was your design and forecast successful last round?” (jupiterinteractive.net) Student players will find the reflective observation ques- tions challenging, as the questions try to connect student deci- sions with critical thinking about what they have done in the previous round, as well as what changes they will make to im- prove next rounds results. As a student player, these questions changed the simulation from a numbers game to more of a stra- tegic thinking game. It forces students to describe their under- standing. Instructors will find this be a very informative way to understand what the student was thinking and what they had learned. After each round students get an “assessment of learning goals” report which rates the level of understanding, measures of performance against peers, and provides suggestion for im- provement. This is a good way to have students identify with the simulation learning goals and how well they are learning them. Just like Comp-XM, faculty might have a challenge aligning their program or course learning goals into the list of Zoom’s learning goals. One useful feature the Zoom Simulation provides faculty is the ability to download a report that summarizes the assessment of student learning. The report outlines the simulation learning objectives, assessment measures, benchmarks, analysis of re- sults, and suggested actions for faculty to improve re- sults. Before one downloads the report, benchmark percentage of performance can be set. Zoom calls this a “Percentile Rank” which measures student performance using averages from past results stored in their database. “Percentile Rank compares your performance against all other teams worldwide. If your Percen- tile Rank is 65%, then your team performed better than 65% of all other teams for that round.” (jupiterinteractive.net) Both simulation assessment tools do a good job assessing how well the student understands the inner workings of the sim- ulation and financial analysis. In addition, having students com- plete an individual simulation assures that the nonparticipating team members actually do some work. However AACSB is more interested in the learning goals of the degree program. As stated in AACSB report “Eligibility Procedures and Accredita- tion Standards for Business Accreditation” “For assurance of learning purposes, AACSB accreditation is concerned with broad, program-level focused learning goals for each degree program, rather than detailed learning goals by course or topic, which must be the responsibility of individual faculty mem- bers,” (AACSB, 2013). Faculty must make sure that the pro- gram learning goals are aligned with the simulation learning goals. This is the main drawback of relying on a simulation to assess broad, programmatic learning goals. Since the simula- tion companies do not know your school’s specific degree learning goals, it’s unlikely for them to be in alignment. Faculty would be required to do some additional work to ensure the assessment of the simulation achieves some of the degree pro- gram requirements. TABLE 3 LIST OF ZOOM SIMULATION LEARNING GOALS Understanding the importance of sales forecasting, Marketing, and their impact upon growing revenues. Understanding shareholders' required rates of return, managing capital, and overall creation of wealth. Understanding of profit generation by managing costs and sales prices. Understanding of how to optimize operating profits and create an efficient marketing budget. Understanding how to create and maximize shareholders' wealth. Asset turnover measures a firm's efficiency at using its assets in generating sales or revenue. Shows an understanding of profits compared to total assets. Understanding of how to increase shareholders return for their investment in the company. An overall assessment of learning goals and the management of the company in all areas. Page 130 - Developments in Business Simulation and Experiential Learning, volume 42, 2015 CONCLUSIONS The authors conclude with several observations.  some websites represent the actual name of the simulation-- others do not. Some websites represent so-called “umbrella” sites offering an array of simulations for both academic and professional use. Some simulations and/or websites also represent the name of the company and/or the name of the simulation. All in all, this makes it difficult for faculty to determine what exactly to look for and how to look for it.  Some simulations use an elaborate executive mentoring process to assess student learning using simulations. While extensive and appropriate for assessment purposes, this process (or processes) is not part of the simulation per se. An example is the program that was implemented at Uni- versity of Tennessee at Knoxville’s School of Busi- ness. This program was reported by Ernie Cadotte in a recent BizEd article (2013). To operate this kind of pro- gram elsewhere would be quite expensive and require con- siderable buy-in from faculty to operate this kind of pro- gram elsewhere.  There have been some attempts to develop rubrics simula- tions to be used for (general) assessment purposes. A pa- per by Hornyak, Peach and Snyder (2007), for example, attempted to develop a rubric using simulations to assess “student learning objectives.” Additionally, Hall and Ko (2006) developed a model using simulations to assess the learning assurance process in an Executive MBA program. For institutions and instructors who are seeking a quick way in which to conduct assessment using simulations, the re- sults of this study do not provide an easy answer. A study con- ducted by Pat Neely and Jan Tucker of Ashford University re- port on an extensive process, which involved several commit- tees, required considerable time and effort, to evaluate the AoL of various simulations. The work started by setting course and learning goals and then tried to match these with the assessment modules of various simulations. The study was exhaustive in that the committee extensively reviewed many simulations, with some of the providers actually being interviewed. They con- clude: Simulations allow students to interact with complex systems and ideas but assessing the actual learning that takes place can be challenging. Finding an effective instructional model [viz., a simulation] which supports both hard (technical or procedural) and soft (people, communication) skills can be challenging....our find- ings indicate that simulations are not designed for the sole purpose of assessing the competencies of business students. At the present time, off-the-shelf business simulations fall short in the measurement of student learning at the summative assessment level. (Neely & Tucker, 2012) We are left with the issue of determining whether the as- sessment modules contained in simulations meet or match the learning goals of the particular course in which they are used and/or whether they address the broader assessment needs of a school of business’ programmatic learning goals. Individual faculty members and business school AoL committees will have to devote considerable time and effort to determine for them- selves if simulation assessments--as they now stand--provide useful assessment methods for course and programmatic learn- ing goals. REFERENCES AACSB. (2013). Assurance of Learning Standards: An Inter- pretation. Tampa, Fla. AACSB International. AACSB. (2013). Eligibility Procedures and Accreditation Standards for Business Accreditation. Tampa, Fla. AACSB International. Anderson, P.H. and L. Lawton (2004) “Simulation Exercises and Problem-Based Learning: Is There a Fit?” Devel- opments In Business Simulation & Experiential Learn- ing, 31, 68-73. Reprinted in The Bernie Keys Library, 14th edition. Anderson, Phil, Cannon, Malik, Dolly, and Thavikulwat, Pre- cha (1998). “Games as Instruments of Assessment: A Framework for Evaluation, Developments In Business Simulation & Experiential Learning, 25, 31- 37. Reprinted in The Bernie Keys Library, 14th edition. Anderson, Phil, et. al. (1998). “Games as Instruments of Assess- ment: A Framework for Evaluation, Developments In Business Simulation & Experiential Learning, 25, 31- 37. Reprinted in The Bernie Keys Library, 14th edition. Angelo, T. A. (1995). Reassessing Assessment. AAHE Bulletin, 47 (8), 7-13. Bollag, B. (2006). “Fears of possible federal learning stand- ards grow as liberal-arts accreditor is penal- ized,” Chronicle of Higher Education. Cadotte, Ernie (2014) “Metrics For Mentors”, BizEd, May/ June. Crisp, V., & Ward, C. (2008). The Development of a Formative Scenario-Based Computer Assisted Assessment Tool in Psychology for Teachers: The PePCAA Project. Computers & Education, 50(4), 1509-1526. Drost, Donald and Chaney, John (2001). “Core Course Out- comes Assessment and Program Improvement Using An Integrative Business Plan – An Empirical Evalua- tion,” Developments In Business Simulation & Experi- ential Learning, 28, 42-48. Reprinted in The Bernie Keys Library, 14th edition. Feinstein, Andrew, & Cannon, Hugh. (2002). Constructs of simulation evaluation. Simulation & Gaming, 33(4): 425-440. Fristzsche, David, and Cotter, Dick (1997). “The Business Poli- cy Game: An International Simulation—An Assess- ment Tool,” Developments In Business Simulation & Experiential Learning, 24, 25-30. Reprinted in The Bernie Keys Library, 14th edition. Page 131 - Developments in Business Simulation and Experiential Learning, volume 42, 2015 Gold, Stephen (2008). “Design and Demonstration of an Online Managerial Economics Game with Automated Coach- ing for Learning and Graded Exercises for Assess- ment,” Developments In Business Simulation & Expe- riential Learning, 35,336-339. Reprinted in The Ber- nie Keys Library, 14th edition. Hall, Owen and Ko, Kenneth (2006). “Learning Assurance Us- ing Business Simulations Applications to Executive Management Education,” Developments In Business Simulation & Experiential Learning, 33,1- 6. Reprinted in The Bernie Keys Library, 14th edition. Hornyak, Martin, Peach, E. Brian, and Snyder, Stephen (2007). “Assessment and Simulations: Measuring the Academ- ic Learning Compacts Within,” Developments In Busi- ness Simulation & Experiential Learning, 34, 215- 222. Reprinted in The Bernie Keys Library, 14th edi- tion. Markulis, Peter and Strang, Daniel (2008). “Assessment in the Modern Large High-Tech Classroom,” Developments In Business Simulation & Experiential Learning, 35,191-197. Reprinted in The Bernie Keys Library, 14th edition. Markulis, Peter (2007). “From Case Presentation to Case Facili- tation: How Assessment Changed the Capstone Course,” Developments In Business Simulation & Ex- periential Learning, 34, 85-93. Reprinted in The Ber- nie Keys Library, 14th edition Martell, Kathryn and Calderon, Thomas G. (2005) “Assessment in Business School: What It Is, Where We Are, and Where We Need to go Now,” in K. Martell and T. Cal- deron (eds.), Assessment of Student Learning in Busi- ness Schools: Best Practices Each Step of the Way. The Association for Institutional Research and AACSB International Volume 1: Number 2, pp. 1- 26. Tallahassee, Fl.: Association for Institutional Re- search Mundhenk, R. T. (2005). “Assessment in the context of accred- itation. In K. Martell and T. Calderon (eds.), Assess- ment of Student Learning in Business Schools: Best Practices Each Step of the Way. The Association for Institutional Research and AACSB International Vol- ume 1: Number 2, pp. 27-42. Tallahassee, Fl.: Associa- tion for Institutional Research. Neely, Oam & Tucker, Jan (2012) “Using Business simulations As Authentic Assessment Tools” American Journal of Business Education July/August, Vol. 5, (4), pp. 449- 456. Palomba, C.A., and T. W. Banta. (1999). Assessment Essen- tials. San Francisco, CA., Josey-Bass. Popham, W. (1999). Classroom Assessment: What Teachers Need To Know. Second Edition. Prindle, Charles 7 Michel, Mitri (2007). “Assessment Practices in AACSB-Accredited Business schools” Journal for Education for Business (Mar/April, 82 (40), pp. 202- 211. Vik, Gretchen, Micklich, Douglas and Forte, Monique (2005). “How Simulations and Experiential Learning Fit As We Comply With Legislative and AACSB Assessment Guidelines: How to Develop Academically Sound Courses That Also Meet Stakeholder Needs,” Develop- ments In Business Simulation & Experiential Learning, 32, 53-60. Reprinted in The Bernie Keys Library, 14th edition. Vonderwell, S., & Boboc, M. (2013). Promoting Formative Assessment in Online Teaching and Learning. Techtrends: Linking Research & Practice To Improve Learning, 57(4), 22-27. doi:10.1007/s11528-013-0673- x Page 132 - Developments in Business Simulation and Experiential Learning, volume 42, 2015 APPENDIX A CONTACT PERSON QUESTIONS These questions represent the first round. If we received a response, we generally followed up with some clarification questions or additional questions on assessment 1. In what course or subject area is your simulation primarily used? 2. Does your simulation have a formal assessment tool, mod- ule or component(s)? 3. What or which learning goal(s) does your simulation as- sess? How many learning goals? 4. How are learning goals assessed? 5. Is there an additional charge for the assessment tool, mod- ule or component(s)? 6. How many colleges currently use your simulation? 7. Is the simulation’s assessment tool or module influenced by any accreditation body? Has it been approved by an ac- crediting agency? 8. In your view, how critical is assessment to the faculty who choose your simulation? 9. In the past two years how many faculty adopters did you have? How many student participants? We would appreciate the opportunity to talk to someone in your organization who is knowledgeable in this area. We will be publishing our research in ABSEL (The Association of Busi- ness Simulations & Experiential Learning) in March 1015. If you are unable to provide this information who in your organi- zation might contact in this regard. APPENDIX B CONTACT PERSONS The following persons responded to our email questions (see Appendix 1). We did not include the names of the simula- tions and/or respective companies which did not respond.  Sean Mullins & Gary Lewis (ILS)  Peter Harrington (www.simventure.co.uk)  Maureen Ginley (Knowledge Matters, Inc.)  Adele Khakimova (Interpretive Simulations)  Brook McFarlane (Smartsims Business Simulations)  Lynda Jones (www.jupiterinteractive.net)  Eric Smith (CAPSIM)  Harvey Gold (GOLD simulations)  Tony Faria (COMPETE) http://www.simventure.co.uk http://www.jupiterinteractive.net