Using Assessment To Measure The Effectiveness of a Financial Simulation Page 187 - Developments in Business Simulation and Experiential Learning, Volume 46, 2019 ABSTRACT This study examines how useful a financial simulation is at reinforcing financial learning goals as measured by an assessment. Programs within many College of Businesses are under increasing pressure to demonstrate that students are learning within their courses, assessment is one method of documenting learning. Most accreditation certifications require colleges to provide an assurance of learning using assessment techniques at the course and program levels. Business simulations play a potentially significant role in reinforcing course learning goals which are measured by assessments and document the assurance of learning process. Results indicate an improved comprehension of financial learning objectives for students after completing a business simulation with financial content. INTRODUCTION Business Simulations which utilize financial analysis and contain financial measurements in the performance review can be useful in the reinforcement of fundamental financial concepts. The college-level course, Principles of finance, will introduce these financial concepts within the coursework. Business simulations create an experiential learning environment where students have an opportunity to put their financial knowledge to work. Business simulations provide financial based feedback based on student inputs and performance compared to peer groups. This performance output can be an artifact of financial learning objectives which business schools need to document with assessment tools. Simulations provide a potentially powerful application for instructors to improve student problem solving and comprehension of key financial measures. As Anderson and Lawton note, "Since their inception in the late-1950s, business games have been recognized as being problem-solving exercises" (Lawton 2004). Business school accreditation associations have made it a priority to document key learning goals with an appropriate assessment method. Collecting the evidence of student learning has become the primary focus for the certification of business schools. "Governmental bodies as well as college accrediting agencies requiring business programs to define learning goals and objectives, at both the college and course level." (Bollag, 2006; Mundhenk, 2005; Hazeldine and Munilla, 2004). Assessments are used to measure the progress students make in comprehending course learning objectives. Instructors have used different approaches too assessing student learning such as comprehensive exams, simulations, writing rubrics, and presentations. Some instructors have even experimented with the use of classroom clickers (an electronic data collection tool students hold in class to provide feedback) for instantaneous assessment (Markulis & Strang, 2008). The pressure for business schools to achieve accreditation from The Association to Advance Collegiate Schools of Business (AACSB) has grown with the overall awareness AACSB by students, parents, and business. AACSB's website invites prospective parents of business students to learn about why their child should choose an AACSB accredited business school with this statement, "Helping prepare your child for a fiercely competitive, global workforce may be one of your toughest parenting challenges. Your child is one in a million to you. Make sure they are one in a million to employers, too. You're here because you are checking out a school's accreditation. You're doing the right thing because accreditation is the most important checkpoint when helping your child choose a business school. You might say selecting a business school is similar to purchasing a home. Since it is such a large investment, you would not purchase a home unless it had been inspected, right? AACSB Accreditation is a lot like a home inspection but for business schools. It helps to ensure that a business school has qualified faculty, rigorous curriculum that will teach skills that employers require, and proper assessment of learning processes in place." (AACSB 2018) This pressure has increased the total number of schools working towards accreditation, directly resulting in the growth of instructors who need to make assessments of students learning. Pedagogical support companies have seen the demand from faculty grow for solutions to measure student assessment and have provided assessment modules in materials such as textbooks, homework managers, and simulations. This study has two goals. First, do simulation reinforce and improve the understanding of financial learning objectives, as measured by current assessment methods. Second, using a survey to assess students opinions of how much the simulation has helped them understand the financial learning goals. SIMULATION & ASSESSMENT LITERATURE REVIEW We have reviewed and summarised some of the past research that has examined the potential for using simulations as an Using Assessment To Measure The Effectiveness of a Financial Simulation Michael Nugent SUNY Stony Brook Michael.Nugent@Stonybrook.edu Stoyan Stoyanov SUNY Stony Brook stoyan.stoyanov@stonybrook.edu Page 188 - Developments in Business Simulation and Experiential Learning, Volume 46, 2019 adequate assessment of student learning. For over 30 years simulations have been used in college classes to reinforce core course concepts. Evaluation of students knowledge has become a required step in schools of business achieving accreditation. “In 2003 the AACSB developed assurance of learning, under which each school sets its own learning goals suitable to the characteristics, circumstances, and mission of the school. AACSB to continue expanding its list of accredited schools. Between1919-1988, 260 business schools earned a business school specific accreditation; in the last 25 years that number has increased 160%.” (Hunt 2015) As a result, faculty members have been seeking methods to assess student progress; some have investigated the potential of simulations to provide assessment results. The goal is to understand how well students are comprehending their course learning goals. A few papers were written that describe and document the benefits from the assessment of student learning as well as the potential for simulations to reinforce learning objectives. Let us start by outlining what the outcomes of assessment are. Using AACSB's definition of assessment as detailed in the paper entitled "Assurance of Learning Standards" (2018) The outcomes assessment process should include: 1. Definition of student learning goals and objectives 2. Alignment of curricula with the adopted goals 3. Identification of instruments and measures to assess learning 4. The collection, analyzing, and dissemination of assessment information 5. Using assessment information for continuous improvement including documentation that the assessment process is being carried out in a systematic, ongoing basis. We found one recent study that examined the direct connection between assessment and simulations. However, it focused mostly on using the simulation as an assessment measure, not whether the students were directly learning from the simulation activities. The paper's findings are still relevant to this study, "Its difficult to determine whether the assessment modules contained in simulations meet or match the learning goals of the particular course or whether they address the broader assessment needs of a school of business' programmatic learning goals. Individual faculty members and business school committees will have to devote considerable time and effort to determine for themselves if simulation assessments--as they now stand--provide useful assessment methods for the course and its learning goals".(Markulis, Nugent, & Strang, Daniel 2015) Why link assessment of student learning and business simulations? The universal adoption of simulations as an experiential learning tool demands it. There have been a few papers to outline the increasing use of simulations within business schools. Angles Fito-Bertran states it well in his 2014 paper entitled "The effect of competences on learning results an educational experience with a business simulator" Within the paper Fito-Bertran states "Business games constitute a relevant method in management training. Their use and application have grown progressively, especially in business schools, universities and professional associations all over the world. The main reason for their popularity is that business games are seen as learning tools that allow students to acquire certain skills and competencies above and beyond the skills fostered by other online or face-to-face methodologies." This paper conducted a study that helped to identify and assess which competencies can achieve the best learning outcomes, within a simulation. Simulations remain popular with business faculty, and many papers have discussed their impact on student skills. Meihua Qian and Karen Clark examined "the most recent literature regarding game-based learning and identified 29 studies which targeted 21st-century skills as outcomes. The findings suggest that a game-based learning approach might be effective in facilitating students' 21st-century skill development. The paper also provides valuable insights for researchers, game designers, and educators in issues related to educational game design and implementation in general."(Qian, Clark 2016) Because of the popularity of simulations, it has become necessary to study the link between the assessment of student learning with simulations as part of accreditation and using continuous improvement to building successful learning environments for students. Most of the past literature on assessment focus on how assessment can lead to a better understanding of how well students are absorbing the course materials. "Using effective assessment techniques can improve an instructor’s understanding of student needs and support learner-centered classrooms." (Vonderwell & Boboc, 2013) The paper by Crisp & Ward entitled “The development of a formative scenario-based computer-assisted assessment tool.” Reiterates the common findings of how assessment leads to better learning, their research focused on how continuous feedback and the enhancement of student engagement, can produce increased both student performance and their motivation to learn.” (2008) Many published papers examine the assessment process as a "systematic collection, review, and use of information about educational programs undertaken to improve student learning and development.” (Palomba, Banta 1999). A critical aspect of our research is determining if a simulation can reinforce financial learning goals using an assessment measure to document the learning. Which is different from measuring the assessment of learning, this difference is detailed in a paper by W. Popham. "Assessment of learning is focused primarily on assigning grades as the principal indicator of student performance, while assessment for learning is when the students' status concerning educational variables of interest is determined" (Popham1999). Upon reading W. Popham's paper, we interpret that simulations most often fall into the assessment for learning category. Most financial or business simulations use the change in financial ratios as feedback to help students understand how well they have performed, the difference in financial ratio results are the result of student decisions within the simulation. The other student firms performances make up an industry average which can benchmark their progress. The Zoom simulation we worked with had an evaluation tool to help students understand their performance, and providing them suggestions on how to achieve better results hopefully leading them to comprehend the financial learning objectives of the course. Instructors can review these results to get a sense of how a student learning is improving including the understanding of course learning goal. Another more recent paper entitled "Applying learning analytics to students' interaction in business simulation games. The usefulness of learning analytics to know what students really learn." Discussed how learning with simulations involves uncertainty, Page 189 - Developments in Business Simulation and Experiential Learning, Volume 46, 2019 and the effects of students discussions about the uncertainty within the simulation game can a reinforce comprehension. The experiential aspects of the simulation can make the assignment more engaging and possible improve recall, of course, specific learning goals. "With regard to the implications for educators, in the specific case of business simulation games, this study established that to improve the learning results of students, teachers should pay special attention to the uncertainty involved in the game. Most of the learning problems of students when participating in business simulation games are caused by how they face uncertainty." Perhaps this uncertainty is what makes simulations more effective at reinforcing business concepts. Their finding "revealed that the most frequent contents in the students' online discussion forums were related, firstly, to the parameters and features of the business simulation game, and, secondly, to elements that fostered the students' learning process, while small talk or regular conversation did not appear to be relevant. In addition, the contents with predictive power over learning results were related to uncertainty, time, interaction, communication and collaboration, although none of these elements influenced teacher assessment of student learning." (Hernandez-Lara 2018) Simulation publishers have quickly seized upon this assessment trend to offer instructors tools for using simulations as assessment solutions. This study will not use the tools provided by the simulation, but rather an assessment exam, already approved and utilized as part of the course assessment, to determine how well a student comprehends the course material after they have had a chance to participate in the Zoom business simulation. A study conducted by Pat Neely and Jan Tucker of Ashford University report: "Simulations allow students to interact with complex systems and ideas but assessing the actual learning that takes place can be challenging. Finding an effective simulation which supports both technical and communication skills can be challenging. Our findings indicate that simulations are not designed for the sole purpose of assessing the competencies of business students. At present, off-the-shelf business simulations fall short in the measurement of student learning at the summative assessment level." (Neely & Tucker, 2012) This paper is not examining the simulation as a tool for assessment but rather if the use of a financially based simulation can improve a student understands, of course, learning goals, after completing the simulation. The improvement in student learning would be measured and documented by the current assessment methodology for the course. HYPOTHESIZES This research paper will investigate the following hypothesis for each of the twenty questions on the assessment exam: The post-assessment group will have a higher percentage score of correct answers than the pre-assessment group. SIMULATION For this study, students worked with the Zoom Business Simulation. This simulation is comparable to other popular business simulations companies, including Capsim's, Marketplace, and Smartsims, who have produced similar products such as Capstone, Marketplace pro, and Mikes Bikes. The Zoom Business Simulation enables students to compete against each other as they make high-level management decisions. During the gameplay, students design automobiles, create a marketing budget, plan production schedules, invest in plant and equipment, and review capital budgets and financial statements. Improvement of financial performance is vital in achieving a high score, in the Zoom simulation, the performance measure is overview points. Overview points are measured by year over year improvements of financial ratios. During each simulation round, students make a series of business decisions with the ultimate goal of maximizing shareholders wealth. After each round, the Zoom simulation provides automated feedback, of financial performance, chart, and reports on performance in various operational measures. SIMULATION AND COURSE LEARNING GOALS We measured the Principles of Finance course learning goals using an assessment exam; the course learning goals are a subset of the stated learning outcomes of the Zoom Business Simulation. For this study to be relevant, the course-specific learning goals must be in aligned with the simulation learning goals. We felt that the learning environment created by the simulation experience directly correlates to the learning goals from the Principles of Finance course. Course Specific Learning Outcomes as stated in the course Syllabus: Demonstrate an ability to analyze financial ratios related to a firm’s financial performance. (Financial Ratios) Demonstrating knowledge of how a firm generates profits by managing costs and sales prices. (Operational and Gross Profitability) Demonstrating knowledge of how to manage liabilities, cash, inventory, and optimize operating profits. (Debt, Liquidity & Activity) Demonstrating knowledge of how to maximize shareholders' wealth. (Net Profit Margin) Interpreting total asset turnover as a measure of a firm's efficiency at using its assets in generating sales. (Asset Turnover) Demonstrating knowledge of profits compared to total assets. (Asset Efficiency) Making decisions that will result in increased shareholders return and stock price. (ROE and Market Value. Zoom Simulation Learning outcomes as stated on their website: Understanding the importance of sales forecasting, and their impact on growing revenues. Understanding shareholders' required rates of return, managing capital, and the creation of wealth. Understanding of profit generation by managing costs and sales prices. Understanding how to optimize operating profits and create an effective marketing budget. Understanding how to create and maximize shareholders' wealth. Asset turnover measures a firm's efficiency at using its assets in generating sales or revenue. Shows an understanding of profits compared to total assets. Knowledge of how to increase shareholders return on their investment in the company. Page 190 - Developments in Business Simulation and Experiential Learning, Volume 46, 2019 EXHIBIT 1 SUCCESS RATE PRE AND POST TEST Question Success Rate P-value Success Rate P-value Pre-Test 2017 Post- Test 2017 Pre-Test 2018 Post- Test 2018 1 The ________ ratios are primarily used as measures of return. 0.71 0.69 0.77 0.66 0.78 0.03 2 The ________ of a business firm is measured by its ability to satisfy its short-term obligations as they come due. 0.65 0.84 0.00 0.52 0.63 0.09 3 ________ may indicate a firm is experiencing stock-outs and lost sales 0.79 0.81 0.74 0.73 0.80 0.18 4 The ________ ratio indicates the efficiency with which a firm uses its assets to generate sales. 0.60 0.91 0.00 0.41 0.50 0.14 5 A firm with a total asset turnover lower than in- dustry standard may have ________. 0.69 0.80 0.09 0.71 0.77 0.28 6 ________ ratio measures the proportion of total assets financed by the firm's creditors 0.55 0.93 0.00 0.50 0.57 0.24 7 ________ indicates the percentage of each sales dollar remaining after the firm has paid for its goods. 0.52 0.88 0.00 0.47 0.50 0.68 8 ________ measures the return earned on the com- mon stockholders' investment in the firm. 0.69 0.89 0.00 0.67 0.88 0.00 9 ________ measures the percentage of each sales dollar remaining after all costs and expenses, in- cluding interest, taxes, and preferred stock divi- dends, have been deducted. 0.63 0.85 0.00 0.68 0.65 0.60 10 ________ measures the percentage of profit earned on each sales dollar before interest and taxes but after all costs and expenses. 0.51 0.89 0.00 0.43 0.55 0.04 11 Time-series analysis is often used to ________ 0.55 0.87 0.00 0.51 0.58 0.31 12 A firm's total asset turnover increased from 0.75 to 0.90. Which of the following is true about the given data? 0.64 0.93 0.00 0.62 0.66 0.48 13 A firm has a current ratio of 1; in order to im- prove its liquidity ratios, this firm might ________. 0.47 0.88 0.00 0.34 0.36 0.63 14 When assessing the fixed-payment coverage ra- tio, ________. 0.49 0.91 0.00 0.34 0.47 0.05 15 Cross-sectional ratio analysis is used to ________. 0.65 0.88 0.00 0.62 0.72 0.09 16 ________ is a term used to describe the magnifi- cation of risk and return introduced through the use of fixed-cost financing, such as preferred stock and debt. 0.54 0.77 0.00 0.40 0.48 0.21 17 ________ measures the overall effectiveness of management in generating profits with its availa- ble assets. 0.54 0.88 0.00 0.55 0.56 0.80 18 A firm with sales of $1,000,000, net profits after taxes of $30,000, total assets of $1,500,000, and common stockholders' investment of $750,000 has a return on equity of ________. 0.43 0.83 0.00 0.27 0.34 0.20 Page 191 - Developments in Business Simulation and Experiential Learning, Volume 46, 2019 METHODOLOGY This study collected data from three class sections of Principles of Finance taught at Stony Brook University's College of Business. The first section of 186 students occurred during the fall semester of 2017, the second of 199 students, and the third section of 57 students occurred during the spring semester of 2018. In all class sections, course learning content was presented during lectures and reinforced with computational homework assignments. Each class formed random Pre-Test and Post-Test groups of students. The Pre-Test groups were given a pre-assessment exam before the start of the simulation. The Post-Test groups completed the pre-assessment exam at the end of the simulation. The pre and post assessment exam contained identical questions. The assessment exam represented 5% of the course grade. Our statistical analysis has two parts. The first part consists of an analysis of all questions on a stand-alone basis. Potentially, this analysis would allow identifying questions that students find difficult. More specifically, we compute the success rate of each question individually in the Pre-Test and the Post-Test groups, and we test if the success rate of each question is higher in the Post-Test group for each year. The success rate is measured by the average number of successful answers within each group. Statistically, testing the hypothesis involves comparing the averages of two samples with unknown standard deviations and, thus, the standard t-test applies. We assume that the observations in each sample are independent and identically distributed. In our study, the success rate can be modeled by a binomial distribution, which is asymptotically normal when the number of trials is high enough. Because the number of trials equals the number of students in the sample and we have a relatively large sample of students, the asymptotic distribution of the test statistic is expected to work well. The second part of the statistical analysis compares the success rate of groups of questions across the Pre- and Post-Test groups and allows drawing a general conclusion. This analysis may not be possible by comparing questions individually. For example, suppose that the previous analysis identifies significant improvement in the success rate in 60% of the questions asked, however, the corresponding p-values are close to the critical level. Can we accept the hypothesis that the simulation is overall useful? The same problem appears if we need to group questions by a learning goal. EXHIBIT 3 PRE-TEST GROUP IN 2017 AND 2018 COMBINED 19 ________ ratio measures a firm's ability to pay contractual interest payments. 0.48 0.88 0.00 0.29 0.36 0.28 20 Which of the following is true of benchmarking? 0.57 0.84 0.00 0.53 0.64 0.07 EXHIBIT 2 THE P-VALUES OF THE TWO-SAMPLE T2 TEST Year 2017 2018 p-values 0 0.005 Page 192 - Developments in Business Simulation and Experiential Learning, Volume 46, 2019 One possible strategy is to average the success rate of all questions with the group in the Pre-Test and Post-Test and then, again, to compare two averages. This strategy, however, is methodologically correct only if it is reasonable to assume that the answers to the different questions within the group are independent which is unlikely if there is evidence of learning. For example, if all students who answered Question 1 correctly also answered Question 2 correctly in the group, then the answers will be correlated, and the asymptotic distribution of the test statistic which assumes independence may be incorrect. To deal with this issue, we use a statistic which tests if a vector of success rates in the Pre-Test group equals a vector of success rates in the Post-Test group. The underlying assumption is that the Pre-Test and the Post-Test answers are drawn from a multivariate distribution with the same covariance structure. The statistic used is the two-sample T2 statistic1. The two-sample test was applied to all questions; additionally, it is also possible to apply to any subgroup. DISCUSSION OF RESULTS The statistical results in the first part of the analysis are presented in exhibit 1. Each row of the table corresponds to a question which is given in the second column. The success rates in the Pre-Test and the Post-Test groups of each year are presented in columns three, four, six, and seven. The p-values of the t-test that the success rate in the Post-Test group equals than that of the Pre-Test group are included in columns five and eight. If the p-value is below 5% (exhibit 1), we reject the hypothesis that the two success rates are equal at 95% confidence level. In both years, we notice that the success rate for each question in the Post-Test sample is higher than that in the Pre-Test sample with the only exception of Question 9 in 2018. However, in 2017 the gaps are relatively larger leading to more questions with statistically significant differences in the success rate. Even though the differences are smaller on a relative basis, we accept the hypothesis that the simulation adds value. The two-sample test results are presented in exhibit 2. Finally, we take a closer look at the correlation of the successful answers across questions. As a result of learning, one would expect to see some elevation in correlations. The reason is that students who have mastered the material are more likely to answer most of the questions correctly; that is, if we fix the student, then question answers should not look like a sample of independent observations. The correlations are represented graphically in two panels in exhibit 3. Positive correlations are encoded by a red color map, and negative correlations are encoded by a blue color map. Even though not a statistical test strictly speaking, visually the correlations in exhibit 4 appear somewhat elevated compared to those in exhibit 3 which supports the fact that the simulation improves the likelihood of answering questions correctly. Individual question success rate are displayed in the Pre-Test and Post-Test samples in 2017 and 2018 together with the corresponding p-values. The p-values are computed under the null hypothesis that the success rates are equal. The p-values in bold indicate statistical significance at 95% confidence level. EXHIBIT 4 POST-TEST GROUP IN 2017 AND 2018 COMBINED. 1 See for example Chapter 6 of Johnson and Wichern (2012) Page 193 - Developments in Business Simulation and Experiential Learning, Volume 46, 2019 Exhibits 3 and 4 display a graphical representation of the correlations of the answers across questions. Exhibit 3 includes the correlations in the Pre-Test group for both years 2017 and 2018 combined, and Exhibits 4 includes the correlations in the Post-Test group for the years combined. STUDENT SIMULATION SURVEY Interestingly the students surveyed after completing the simulation have overwhelming agreed that the simulation improved their understanding of corporate finance, financial ratios and how companies maximize wealth. One hundred and seven students responded to the three-question survey, which used a Likert scale, Strongly Agree, Agree, Neither Agree nor Disagree, Disagree. 53.3% of students agreed, and 26.2% of students strongly agreed with question one, “Completing the Zoom Simulation helped to improve my overall understanding of corporate finance.” Students responded 50.4% agreed and 24.3% strongly agreed to questions two, “Completing the Zoom Simulation helped to develop my overall knowledge of financial ratios.” 46.7% of students agreed, and 33.6% of students strongly agreed with question three, “Completing the Zoom Simulation helped to improve my overall understanding of how companies maximize wealth.” CONCLUSIONS Pedagogy incorporating active learning, such as simulations, are more commonplace in the college classroom and as demand for active learning experiences from student increases, researching the impact of simulations on learning outcomes becomes more relevant. Financial simulations have the potential to enhance a student’s comprehension of the fundamentals of finance. Research exploring the effectiveness of financial simulations and the best practices for deploying financial simulations within the curriculum can have a significant impact on student learning. The results of this research will be valuable to both the finance faculty and the documentation of learning based assessments. Gaining new knowledge of how financial stimulations can improve the comprehension of financial concepts is critical to the success of future students. The findings of this research, which demonstrated that completing a financial simulation has a significant impact on correctly answering financial-based questions. Demonstrates the usefulness of Financial simulations to enhance student comprehension of financial concepts as measured by an assessment exam. These results will be beneficial to universities and finance EXHIBIT 5 ZOOM SIMULATION SURVEY DATA Question 1 Count % Total Agree Total Disagree Agree 57 53.3% 79.4% 10.3% Disagree 6 5.6% Neither Agree nor Disa- gree 11 10.3% Strongly Agree 28 26.2% Strongly Disagree 5 4.7% Question 2 Count % Total Agree Total Disagree Agree 54 50.5% 74.8% 11.2% Disagree 7 6.5% Neither Agree nor Disa- gree 15 14.0% Strongly Agree 26 24.3% Strongly Disagree 5 4.7% Question 3 Count % Total Agree Total Disagree Agree 50 46.7% 80.4% 8.4% Disagree 5 4.7% Neither Agree nor Disa- gree 12 11.2% Strongly Agree 36 33.6% Strongly Disagree 4 3.7% Page 194 - Developments in Business Simulation and Experiential Learning, Volume 46, 2019 AACSB. (2018). Assurance of Learning Standards: An Interpretation. Tampa, Fla. AACSB International. AACSB. (2018). Eligibility Procedures and Accreditation Standards for Business Accreditation. Tampa, Fla. AACSB International. AACSB. (2018). Why Your Child Should Choose an AACSB- Accredited Business School. Tampa, Fla. AACSB International. Anderson, P.H. and L. Lawton (2004) “Simulation Exercises and Problem-Based Learning: Is There a Fit?” Developments In Business Simulation & Experiential Learning, 31, 68-73. Reprinted in The Bernie Keys Library, 14th edition. Anderson, Phil, Cannon, Malik, Dolly, and Thavikulwat, Precha (1998). “Games as Instruments of Assessment: A Framework for Evaluation, Developments In Business Simulation & Experiential Learning, 25, 31-37. Reprinted in The Bernie Keys Library, 14th edition. Anderson, Phil, et al. (1998). “Games as Instruments of Assessment: A Framework for Evaluation, Developments In Business Simulation & Experiential Learning, 25, 31-37. Reprinted in The Bernie Keys Library, 14th edition. Angelo, T. A. (1995). Reassessing Assessment. AAHE Bulletin, 47 (8), 7-13. Bollag, B. (2006). “Fears of possible federal learning standards grow as liberal-arts accreditor is penalized,” Chronicle of Higher Education Cadotte, Ernie (2014) “Metrics For Mentors,” BizEd, May/ June. Crisp, V., & Ward, C. (2008). The Development of a Formative Scenario-Based Computer Assisted Assessment Tool in Psychology for Teachers: The PePCAA Project. Computers & Education, 50(4), 1509-1526. Drost, Donald and Chaney, John (2001). “Core Course Outcomes Assessment and Program Improvement Using An Integrative Business Plan – An Empirical Evaluation,” Developments In Business Simulation & Experiential Learning, 28, 42-48. Reprinted in The Bernie Keys Library, 14th edition. Feinstein, Andrew, & Cannon, Hugh. (2002). Constructs of simulation evaluation. Simulation & Gaming, 33(4): 425-440. Fitó-Bertran, À., Hernández-Lara, A. B., & López, E. S. (2015). The effect of competences on learning results an educational experience with a business simulator. Computers In Human Behavior, 51(Part B), 910-914. doi:10.1016/j.chb.2014.11.003 Fristzsche, David, and Cotter, Dick (1997). “The Business Policy Game: An International Simulation—An Assessment Tool,” Developments In Business Simulation & Experiential Learning, 24, 25-30. Reprinted in The Bernie Keys Library, 14th edition. Gold, Stephen (2008). “Design and Demonstration of an Online Managerial Economics Game with Automated Coaching for Learning and Graded Exercises for Assessment,” Developments In Business Simulation & Experiential Learning, 35,336-339. Reprinted in The Bernie Keys Library, 14th edition. Hall, Owen, and Ko, Kenneth (2006). “Learning Assurance Using Business Simulations Applications to Executive Management Education,” Developments In Business Simulation & Experiential Learning, 33,1-6. Reprinted in The Bernie Keys Library, 14th edition. Harcharik, Kathleen, and Hwang, Drew, “Developing a Framework for Assessing the Undergraduate Business Core” (1998), presented at the AACSB Continuous Improvement Symposium. Dallas, Texas, September 13-15, 1998. Hazeldine, M. F. & Munilla, M. S. (2004). The 2003 AACSB accreditation standards and implications for business faculty: A short note," Journal of Business Education, 80, 29-35. Hernández-Lara, A. B., Perera-Lluna, A., & Serradell-López, E. (2018). Full-length article: Applying learning analytics to students' interaction in business simulation games. The usefulness of learning analytics to know what students really learn. Computers In Human Behavior, doi:10.1016/j.chb.2018.03.001 Hornyak, Martin, Peach, E. Brian, Bowen, Adam, Moses, William and Wheeler, Roberta (2006). “Examining Program Management in Business Simulations: Student and Faculty Views,” Developments In Business Simulation & Experiential Learning, 33, 107- 117. Reprinted in The Bernie Keys Library, 14th edition. Hornyak, Martin, Peach, E. Brian, and Snyder, Stephen (2007). “Assessment and Simulations: Measuring the Academic Learning Compacts Within,” Developments In Business Simulation & Experiential Learning, 34, 215-222. Reprinted in The Bernie Keys Library, 14th edition. Hunt, S. C. (2015). Research on the Value of AACSB Business Accreditation in Selected Areas: A Review and Synthesis. American Journal Of Business Education, 8 (1), 23-30. Johnson, R. A. and Wichern, D. W. (2012) Applied Multivariate Statistical Analysis, 6ed. Phi Learning Private Limited. REFERENCES faculty who incorporate simulations into classroom curriculum and use examinations as the basis of assessment of learning., LIMITATIONS This study only looked at one financial simulation; different simulations may have different results on student assessment results. The study looked at only three sections of finance, taught at different points in time, by the two instructors. Results were not compared to other groups of students, at different schools. Homogeneity comparisons are not included for participating students. Page 195 - Developments in Business Simulation and Experiential Learning, Volume 46, 2019 Markulis, Peter and Strang, Daniel (2008). “Assessment in the Modern Large High-Tech Classroom,” Developments In Business Simulation & Experiential Learning, 35,191-197. Reprinted in The Bernie Keys Library, 14th edition. Markulis, Peter (2007). “From Case Presentation to Case Facilitation: How Assessment Changed the Capstone Course,” Developments In Business Simulation & Experiential Learning, 34, 85-93. Reprinted in The Bernie Keys Library, 14th edition Markulis, Peter, Nugent, Michael and Strang, Daniel (2015) “Assessing The Role Of Assessment In Business Simulations” Developments In Business Simulation & Experiential Learning, 42 volume. Martell, Kathryn and Calderon, Thomas G. (2005) “Assessment in Business School: What It Is, Where We Are, and Where We Need to go Now,” in K. Martell and T. Calderon (eds.), Assessment of Student Learning in Business Schools: Best Practices Each Step of the Way. The Association for Institutional Research and AACSB International Volume 1: Number 2, pp. 1-26. Tallahassee, Fl.: Association for Institutional Research Mundhenk, R. T. (2005). “Assessment in the context of accreditation. In K. Martell and T. Calderon (eds.), Assessment of Student Learning in Business Schools: Best Practices Each Step of the Way. The Association for Institutional Research and AACSB International Volume 1: Number 2, pp. 27-42. Tallahassee, Fl.: Association for Institutional Research. Neely, Oam & Tucker, Jan (2012) “Using Business simulations As Authentic Assessment Tools” American Journal of Business Education July/August, Vol. 5, (4), pp. 449- 456 Palomba, C.A., and T. W. Banta. (1999). Assessment Essentials. San Francisco, CA., Josey-Bass. Popham, W. (1999). Classroom Assessment: What Teachers Need To Know. Second Edition. Prindle, Charles 7 Michel, Mitri (2007). “Assessment Practices in AACSB-Accredited Business schools” Journal for Education for Business (Mar/April, 82 (40), pp. 202- 211. Qian, M., & Clark, K. R. (2016). Full-length article: Game- based Learning and 21st-century skills: A review of recent research. Computers In Human Behavior, 6350- 58. doi:10.1016/j.chb.2016.05.023 Vik, Gretchen, Micklich, Douglas and Forte, Monique (2005). “How Simulations and Experiential Learning Fit As We Comply With Legislative and AACSB Assessment Guidelines: How to Develop Academically Sound Courses That Also Meet Stakeholder Needs,” Developments In Business Simulation & Experiential Learning, 32, 53-60. Reprinted in The Bernie Keys Library, 14th edition. Vonderwell, S., & Boboc, M. (2013). Promoting Formative Assessment in Online Teaching and Learning. Techtrends: Linking Research & Practice To Improve Learning, 57(4), 22-27. doi:10.1007/s11528-013-0673- x Page 196 - Developments in Business Simulation and Experiential Learning, Volume 46, 2019 APPENDIX A ASSESSMENT EXAM 1) The ________ ratios are primarily used as measures of return. A) debt B) liquidity C) activity D) profitability 2) The ________ of a business firm is measured by its ability to satisfy its short-term obligations as they come due. A) debt B) activity C) profitability D) liquidity 3) ________ may indicate a firm is experiencing stock-outs and lost sales. A) Quick B) Average collection period C) Average payment period D) Inventory turnover ratio 4) The ________ ratio indicates the efficiency with which a firm uses its assets to generate sales. A) inventory turnover B) quick C) current asset turnover D) total asset turnover 5) A firm with a total asset turnover lower than industry standard may have ________. A) excessive debt B) insufficient sales C) insufficient fixed assets D) excessive interest costs 6) ________ ratio measures the proportion of total assets financed by the firm's creditors. A) Total asset turnover B) Current C) Inventory turnover D) Debt 7) ________ indicates the percentage of each sales dollar remaining after the firm has paid for its goods. A) Gross profit margin B) Net profit margin C) Earnings available to common shareholders D) Operating profit margin 8) ________ measures the return earned on the common stockholders' investment in the firm. A) Return on equity B) Net profit margin C) Price/earnings ratio D) Return on total assets 9) ________ measures the percentage of each sales dollar remaining after all costs and expenses, including interest, taxes, and pre- ferred stock dividends, have been deducted. A) Operating profit margin B) Earnings available to common shareholders C) Gross profit margin D) Net profit margin 10) ________ measures the percentage of profit earned on each sales dollar before interest and taxes but after all costs and expenses. A) Earnings available to common shareholders B) Operating profit margin C) Net profit margin D) Gross profit margin 11) Time-series analysis is often used to ________. A) correct errors of judgment B) standardize results C) assess developing trends D) evaluate the value of a firm or its assets 12) A firm's total asset turnover increased from 0.75 to 0.90. Which of the following is true about the given data? A) Its assets have been efficiently used to derive the minimum level of gross profit. B) Its assets have been efficiently used to derive the optimum level of net income. C) Its assets have been efficiently used to derive the optimum level of sales. D) Its assets have been efficiently used to derive the minimum level of net income. Page 197 - Developments in Business Simulation and Experiential Learning, Volume 46, 2019 13) A firm has a current ratio of 1; in order to improve its liquidity ratios, this firm might ________. A) increase inventory, thereby increasing current assets and the current and quick ratios. B) decrease current liabilities by utilizing more long-term debt, thereby increasing the current and quick ratios. C) improve its collection practices and pay accounts payable, thereby decreasing current liabilities and decreasing the cur- rent and quick ratios. D) improve its collection practices by providing extended credit policy. 14) When assessing the fixed-payment coverage ratio, ________. A) the higher its value, lesser is its reliability to pay up the debts B) the lower its value, the higher is the firm's financial leverage C) preferred stock dividend payments can be disregarded D) the lower its value the riskier is the firm 15) Cross-sectional ratio analysis is used to ________. A) isolate the causes of problems B) measure the relative performance of a firm with its peers C) provide conclusive evidence of the existence of a problem D) correct expected problems in operations 16) ________ is a term used to describe the magnification of risk and return introduced through the use of fixed-cost financing, such as preferred stock and debt. A) Fixed-payment coverage B) Financial leverage C) Operating leverage D) Benchmarking 17) ________ measures the overall effectiveness of management in generating profits with its available assets. A) Return on equity B) Total asset turnover C) Return on total assets D) Price/earnings ratio 18) A firm with sales of $1,000,000, net profits after taxes of $30,000, total assets of $1,500,000, and common stockholders' invest- ment of $750,000 has a return on equity of ________. A) 4 percent B) 3 percent C) 20 percent D) 15 percent 19) ________ ratio measures a firm's ability to pay contractual interest payments. A) Times interest earned B) Debt C) Average payment period D) Fixed-payment coverage 20) Which of the following is true of benchmarking? A) It is an analysis in which a firm's ratio values are analyzed to project the fundamental values of the assets for upcoming years or business cycle. B) It is an analysis in which a firm's financial performance over time is evaluated using financial ratio analysis. C) It is a financial statement analysis technique which combines cross-sectional and time-series analyses. D) It is an analysis in which a firm's ratio values are compared with those of a key competitor or with a group of competi- tors that it wishes to emulate. Page 198 - Developments in Business Simulation and Experiential Learning, Volume 46, 2019 APPENDIX B ASSESSMENT EXAM QUESTION TO LEARNING GOAL MAP 1. Liquidity & Activity 2. Financial Ratios 3. Liquidity & Activity 4. Asset Turnover 5. Liquidity & Activity 6. Debt 7. Gross Profitability 8. ROE & Market Value 9. Net Profit Margin 10. Operating Profitability 11. Financial Ratios 12. Debt 13. Liquidity & Activity 14. Debt 15. Financial Ratios 16. ROE & Market Value 17. Asset Efficiency 18. ROE & Market Value 19. Debt 20. Financial Ratios