Practice Makes Perfect: The Impact of Longitudinal Quizzing on Computer Simulation Group Performance Page 165 - Developments in Business Simulation and Experiential Learning, volume 41, 2014 PRACTICE MAKES PERFECT: THE IMPACT OF LONGITUDINAL QUIZZING ON COMPUTER SIMULATION GROUP PERFORMANCE Brandon Kilburn University of Tennessee Martin bkilburn@utm.edu Ashley Kilburn University of Tennessee Martin akilburn@utm.edu ABSTRACT Computer simulations used in the business classroom have demonstrated positive outcomes: learning reinforcement (Dweck, 1986), exposure to real-world decision-making scenarios, increased decision-making speed, and extended information retention times (Bolt, 1993). More specifically, using supplemental assessment tools with computer simulations has been documented emphasizing the importance of oral or written presentations incorporating simulation variables (Alpert, 1995), pairing content with case studies (Zych, 1997), exams/quizzes pertaining to the simulation parameters (Brooks et al. 2006) on learning. Interestingly, however, there is limited research on longitudinal assessment as supplemental assessment and its impact on student performance in simulations on a group-level. This study examines the effect of longitudinal change across two sequential, related, yet non-identical group simulation-related quizzes on group simulation performance ranking. Findings from a sample of 10 different CAPSIM administrations examining 55 groups (over 275 students) suggest that a change in group quiz scores is a significant predictor of group performance ranking in the CAPSIM© business simulation. The group is the unit of analysis. Pedagogical implications discuss the role of learning and longitudinal assessment within groups. STUDENT GROUP SIMULATIONS Online computer simulations are internet-based games which introduce a more realistic learning experience than pure theory or even case studies (DiMeglio, 2008). In an effort to enrich the classroom experience for students in higher education, competitions, namely computer simulations, have been used heavily (Cantor, 1995). Computer simulations have been used in various business disciplines: marketing, accounting (Polimeni, Burke and Benyaminy, 2009), organizational science (Hill, Bartol, Tesluk and Langa, 2009), political science and international relations (Meleshevich & Tamashiro, 2008). The use of computer simulations as learning tools has been main stream since the mid-late 1970’s (Sprouls, 1962; Trieschmann, 1976) due to the ability of students to learn through practice as opposed to the traditional hands-off approach. Umble, Umble & Artz (2008) describe the benefits of team-based competitions as learning tools and their positive effects on student learning process, student motivation to learn, greater retention of knowledge, a more comprehensive and integrative understanding of course material, among other benefits. Positive outcomes from competition include learning reinforcement (Dweck, 1986), exposure to real-world decision-making scenarios, increased decision-making speed as well as longer information retention times (Bolt, 1993). Academic research has also recognized the ability for students to evaluate information, weigh alternatives, and to make decisions in a virtual environment (Di Meglio, 2008). Kilburn, Kilburn and Faught (2010) previously examined pre-competition student assessment scores, group size, and average group GPA to assess their predictability of final group rankings within the simulation. CAPSIM© The computer simulation used in this study will be CAPSIM©. CAPSIM© is a cross-functional team-based competitive computer simulation wherein students are tasked with managing a $100 million company over a simulated time period of up to 8 years (Saulnier, 2009). Over 500 business colleges and universities across the globe utilize the CAPSIM© Capstone Business Simulation (Saulnier, 2009). Within their simulated company, student groups are responsible for decision making in 4 interrelated functional areas: research and development, marketing, Page 166 - Developments in Business Simulation and Experiential Learning, volume 41, 2014 production and finance. The design of the simulation prohibits the use of outcome variables (e.g., profitability, ROI, ROA, etc.) as a unit of analysis for comparison between industries due to the uniqueness of each industry. To allow for diversity among groups, group members were selected by placing different majors within each group. These majors included accounting, marketing, management, economics, and management information systems. Also, gender and race were assigned at random to the groups to allow for additional diversity. All teams were assigned with four, five, or six members. Results of CAPSIM© consider the financial performance of each group with multiple dimensions of business decision-making: R & D, Marketing, Production, HR and Finance, relative to their competitors (CAPSIM, 2010). After approximately 3 weeks of stringent training, the CAPSIM© competition simulation is typically spans a period of 7 weeks during a semester. The data used for this study included semesters with 7 and 8 week simulation competitions. Group performance within the simulation is determined by a weighted relative score calculation which is generated automatically by the simulation (ranging from 0-100). This weighted score is determined by the following variables and corresponding weights: Market Share 12%; Stock Price 12%; Market Capacity 16%; Return on Equity 12%; Return on Sales 12%; Return on Assets 12%; Asset Returns 12%. EVOLUTION OF SIMULATION GROUPS Rue and Byars (2007) suggest that much of the time group performance is better than the average group member; however, the term “much” leaves the door open for opposing viewpoints and analysis. Thus, using a group to make decisions has several advantages and disadvantages. Two of the advantages of groups are: greater pool of knowledge and different perspectives. For these advantages to be effective there must be diversity in the group. Disadvantages include domination by one or more persons in the group, groupthink (Maier, 1967), and dissention among the group (Gentry, 1980). In a study by Laughlin, Hatch, Silver, and Boh (2006), it was found that groups of three to five people perform better than individuals when attempting to solve complex problems. Further, Schumann, Anderson, Scott and Lawton (2008) noted the importance of improvement over time in properly assessing a student’s simulation learning experience. A more thorough understanding of group evolvement can provide indicators as to a group’s expected performance outcomes. Lemberger and Clemens (2012) examine changes across time that impact student performance. The authors cite improved social/ communication and organizational skills as outcomes of student groups over time. Further, Vij and Sharma (2013) find that, among college-age business student groups, training and education have significant relationship to student Entrepreurial Drive. Student group change through learning has been examined thoroughly (Lewis and Grosser, 2012; Naudé, 2012; Spencer, Brown, Griffin and Abdullah, 2008) and overwhelmingly supports the idea of intergroup communication, time and shared experiences as drivers of group success. Naudé (2012) thoroughly examines the role of social learning theory in service learning groups. In this work, Naudé (2012) specifically proposes that student group interrelationships are greatly impacted by prior knowledge, but can be altered based on new experiences within the group. Likewise, Lewis and Grosser (2012) highlight the role of resistance to change in student group failure. Here Lewis and Grosser (2012) point out the importance of intergroup communication, trust and motivation in overcoming resistance to change and increased group performance and effectiveness. With works such as these, the basis for asserting that learning through social interaction, modeling, communication or past experience leading to positive outcomes is made. In order to detect these positive outcomes for student groups such as learning, evolving, or maturing during a simulation, instructors can monitor their improvement. One such way is through sequential quizzing where student groups take highly related, yet different quizzes regarding the content and nature of the simulation. The study presented here is based on 4 years of accumulated data across 10 sections of a capstone business course. Originally, the instructor’s goal was to help the students get a more comprehensive understanding of the parameters regarding the CAPSIM© business simulation. Students are introduced to the simulation through an intense lecture review. They are also required to participate in practice/rehearsal rounds over the first few weeks of the course. After the groups have begun participation in the practice rounds, the instructor administers Quiz 1. Quiz 2 typically follows approximately two class periods later at the time the groups are completing their practice rounds in the simulation. Quiz 1 and Quiz 2 each pertain to the specifications/ parameters of CAPSIM©. The sequential quizzes address the exact same simulation material while utilizing different sets of questions. After administering the quizzes numerous times, the instructor began to notice that student groups who ultimately ranked high in final performance in the simulation (1st or 2nd) also seemed to improve their quiz score from Quiz 1 to Quiz 2. Alternatively, groups that didn’t improve or decreased their score from Quiz 1 to Quiz 2 seemed to perform worse in the simulation. Therefore, the following study specifically addresses whether or not change in group quiz scores (an implied predictor of learning) will assist in predicting group simulation performance. As students undertake assessments in functioning work groups, they are likely to incorporate their increased Page 167 - Developments in Business Simulation and Experiential Learning, volume 41, 2014 functionality and outcomes of social learning among the group into their output. Ideally, Quiz 2 scores would be higher than Quiz 1 scores in cases of functioning groups wherein learning is taking place. Likewise, groups with worsening scores might reflect their lack of functionality/ poor intergroup relations. Ultimately, no matter the improvement or worsening of scores, their change across time and experience should be a significant predictor of overall performance. Worsening groups will likely perform worse in the simulation relative to those healthier better-functioning groups who would perform better. This study addresses one hypothesis: the role of quiz score change (improvement/worsening) as a predictor of group performance. The study will assess the significance, strength, and direction of change in quiz scores on the ability to predict performance in a computer simulation; therefore, the following hypothesis is provided: H1: Change in quiz score (Q2-Q1) is a significant predictor of computer simulation performance ranking. RESEARCH METHOD DATA COLLECTION Data was collected from graduating College of Business seniors at a Southeast university across 10 spring, summer, and fall college semesters for years 2010-2013. The data used for this study included semesters with 7 and 8 week simulation competitions. Within most semesters multiple simulation were being conducted across numerous sections of the same course. Across this time period, 10 different completed CAPSIM© simulations were available for data collection. Within each simulation, there were 5- 6 groups competing against each another. These 10 simulations comprised of over 275 students yielded results for 55 student groups (approximately 5 students per group) which were ranked according to their final standing from 1- 5 or 1-6 depending on the number of groups competing. Student groups are given two quizzes during the semester (Quiz 1 & Quiz 2) to assess their knowledge of the simulation. Although similar in content, the quizzes were distinct. The quiz scores were assessed longitudinally by subtracting Quiz 1 score from that of Quiz 2. Final simulation performance ranking was derived from CAPSIM© computer simulation ending reports which provided the weighted relative score. Final competitive ranking (ranging from 1-6) represents the student groups’ relative performance (1st place=1, - 6th place=6) and is based on the final relative performance score. The final relative performance score was determined by the following performance variables and corresponding weights: Market Share 12%; Stock Price 12%; Market Capacity 16%; Return on Equity 12%; Return on Sales 12%; Return on Assets 12%; Asset Returns 12%. RESULTS Data was analyzed using SPSS Statistics 17.0. H1 was assessed through regression analysis. It is important to note that the best-performing groups were ranked in 1st place. Therefore, a negative correlation was expected. Findings suggest that change in quiz score is a significant predictor of group simulation performance (Correlation= - .285; sig.= .035; F-Statistic=4.695; t-value: -2.167) (See Table 1). The R2 (.081) indicates that approximately 8.1% of the variation in group simulation performance ranking can be explained by the change in group quiz score. Perhaps, as Boscia and Turner (2008) suggested, while groups work to improve their Quiz 2 score and are generally learning more about the simulation, they are building potency and consensus within the group, thus decreasing conflict and increasing decision-making TABLE 1 REGRESSION ANALYSIS Model R R Square Adjusted R Square Std. Error of the Estimate Durbin-Watson 1 .285a .081 .064 1.77308 1.706 Model Sum of Squares df Mean Square F Sig. 1 Regression 14.760 1 14.760 4.695 .035b Residual 166.621 53 3.144 Total 181.382 54 Model Unstandardized Coefficients Standardized Coefficients t Sig. B Std. Error Beta 1 (Constant) 3.428 .239 14.335 .000 change -.044 .020 -.285 -2.167 .035 Page 168 - Developments in Business Simulation and Experiential Learning, volume 41, 2014 efficiency. DISCUSSION AND PEDAGOGICAL IMPLICATIONS Brooks, Burson and Rudd (2006) call for further research on how supplemental assignments to computer simulations can benefit student learning. This study analyzes the impact of sequential quizzes on student learning. The assessment of student groups participating in online simulations can be more helpful if it is multi-faceted and not solely reliant on the simulation-generated ranking. The investigation into student group learning provides insight into pedagogical strategy of utilizing both student groups and online simulations in the classroom. Student groups have been found to be more productive than the individual in their ability to introduce diversity in thought and understanding of material. The use of simulations in the classroom have also proved positive: learning reinforcement (Dweck, 1986), exposure to real-world decision-making scenarios, increased decision-making speed, and extended information retention times (Bolt, 1993). Students are also aided in a computer simulation’s ability to make real-time decisions (Di Meglio, 2008). Results from this study suggest that longitudinal assessment of groups can lead to greater simulation performance. Learning, both simulation content and best practices of social interaction, lends to greater group performance. Longitudinal multi-quiz assessment provides the group with an opportunity to improve their original quiz score. The results indicate that there is a significant linkage between group simulation performance and change in quiz scores. Approximately 8.1% of the variation in group simulation performance ranking can be explained by the change in group quiz score. Therefore, instructors can leverage the test-retest approach in order to help student groups enhance learning in simulations. Assigning multiple quizzes/exams, projects, or presentations across the life of the simulation can benefit a group’s performance. The inverse relationship between quiz change and performance ranking indicates that positive change (quiz score increase) will result in a lower-numbered simulation ranking (1st place as opposed to 2nd). Encouraging students to adequately prepare for quizzes by informing them that their quiz score improvement can help predict their simulation outcome will have two repercussions: (1) motivate the groups to adequately prepare and (2) enhance the simulation performance of all groups. Finally, based on the findings of this study, instructors may use the sequential quiz performance as an indicator of “troubled” groups which can allow the instructor to direct extra attention and effort towards these groups. This extra attention can serve as a preemptive measure to help these groups to gain a balance and understanding of the simulation so that they can maximize learning, which is the ultimate goal of any classroom simulation. REFERENCES Alpert, Frank (1995), “Using Executive Briefings in Marketing Simulations,” Marketing Education Review, 5 (1), 25–32. Bolt, J. (1993). Ten years of change in executive education. Training & Development, 47 (8), 43-44. Boscia, M. and Turner, L. (2008), Linking Team Covenants To Peer Assessment Of Simulation And Experiential Performance, Proceedings, ABSEL, 37, pp. 1-7. Brooks, Bradley W., Timothy E. Burson, and David V. Rudd (2006), “Addressing Current Research Gaps and Directions in Educational Marketing Simulations,” Journal for Advancement of Marketing Education, 9 (Winter), 43–49. Cantor, J. (1995). ASHE-ERIC Higher Education Report series 95-7. Experiential Learning in Higher Education: Linking Classroom and Community, 24 (7). CAPSIM (2010). Retrieved April 10, 2010 from http:// www.capsim.com/visiting_profs/ undergraduate_education/index.cfm? menu_type=undergraduate&header_type=prof Di Meglio, F. (2008). Virtual workplaces in the classroom. Business Week Online, January, p23. Dweck, C. S. (1986). Motivational processes affecting learning. American Psychologist, 41, 1040-1048. Gentry, J. (1980), The Effects Of Group Size On Attitudes Toward The Simulation, Proceedings, ABSEL, 7, pp. 165-168. Hill, S., Bartol, K., Tesluk, P., and Langa, G. (2009). Organizational context and face-to-face interaction: influences on the development of trust and collaborative behaviors in computer-mediated groups, Organizational Behavior & Human Decision Processes, 108 (2), pp. 187-201. Kilburn, B., Kilburn, A. and Faught, S. (2010), Kilburn, B., Kilburn, A. and Faught, S. Web- Based Simulation Team Success: The Role of Pre- Competition Assessment, Group Size and Group Average GPA, Journal of Learning in Higher Education, Vol. 6(2), pp. 89-94. Laughlin, P., Hatch, E., Silver, J. and Boh, L. (2006). Groups perform better than the best individuals on letters-to-numbers problems: effects of group size, AddedAcademic Journal Journal of Personality & Social Psychology, 90:4, pp. 644 -651. Lemberger, M. & Clemens, E. (2012), Connectedness and Self-Regulation as Constructs of the Student Success Skills Program in Inner-City African American Elementary School, Journal of Counseling & Development, Vol. 90, pp. 450-458. Lewis, A. and Grosser, M. (2012), The Change Game: An Experiential Exercise Demonstrating Barriers to Change, Journal of Management Education, Vol. 36 (5), pp. 669-697. http://web.ebscohost.com/bsi/resultsadvanced?vid=11&hid=110&sid=069a5685-e858-4c6c-9007-634c748c3510%40sessionmgr104&bquery=(AU+(Laughlin))+and+(AU+(Hatch))&bdata=JmRiPWJ1aCZjbGkwPURUMSZjbHYwPTIwMDYwMS0yMDA2MTImdHlwZT0xJnNpdGU9YnNpLWxpdmU%3d Page 169 - Developments in Business Simulation and Experiential Learning, volume 41, 2014 Maier, N. (1967). Assets and liabilities in group problem- solving: The need for an integrative function. Psychological Review, 74, pp. 239–249. Meleshevich, A., & Tamashiro, H. (2008). Learning to learn: Learning to win: How to succeed in the simulated world of model NATO. PS, Political Science & Politics, 41 (4), 865-870. Naudé, C. (2012), At the Cultural Crossroads: Intergroup Psychology Among Students in a Service-Learning Programme, Current Psychology, Vol. 31, pp. 221– 245 Polimeni, R., Burke, J. & Benyaminy, D. (2009). Using Computer Simulations to Recruit and Train Generation Y Accountants, The CPA Journal, May, pp. 64-68. Rue, L. & Byars, L. (2007). Management: Theory and Application. (12th ed). Irwin, Homewood, Ill. Saulnier, D. (2009). Business Simulations: An Interview with Capsim's Dan Smith, Experiential eLearning, retrieved February 14, 2010 from http:// saulnier.typepad.com/learning_technology/2009/03/ business-simulations-an-interview-with-capsims-dan- smith.html. Schumann, P., Anderson, P., Scott, T. and Lawton, L. (2008), A Framework For Evaluating Simulations As Educational Tools, Proceedings, ABSEL, 28, pp. 215- 220. Spencer, M., Brown, M., Griffin, S. and Abdullah, S. (2008), Outcome Evaluation of the Intergroup Project, Small Group Research, Vol. 39(1), pp. 82-103. Sprouls, R. (1962). A role of computer simulation in accounting education. Accounting Review, 37 (3), pp. 515-520. Umble, E., Ubmle, M., & Artz, K. (2008), Enhancing Undergraduates’ Capabilities Through Team-Based Competitions: The Edward Jones Challenge, Decision Sciences Journal of Innovative Education, 6:1, pp. 1- 27. Vij, S. and Sharma, P. (2013), Does Entrepreneurial Education Enhance the Entrepreneurial Drive of Business Students?, Journal of Entrepreneurship Development, Vol. 10 Issue 2, p65-82. Zych, John M. (1997), “Adding Case Materials to Reinforce the Realism in Marketing Simulations,” Marketing Education Review, 7 (Spring), 51–61. http://saulnier.typepad.com/learning_technology/2009/03/business-simulations-an-interview-with-capsims-dan-smith.html http://saulnier.typepad.com/learning_technology/2009/03/business-simulations-an-interview-with-capsims-dan-smith.html http://saulnier.typepad.com/learning_technology/2009/03/business-simulations-an-interview-with-capsims-dan-smith.html http://saulnier.typepad.com/learning_technology/2009/03/business-simulations-an-interview-with-capsims-dan-smith.html