Digital Game Building as Assessment: A Study of Secondary Students’ Experience Page 74 - Developments in Business Simulation and Experiential Learning, volume 40, 2013 ABSTRACT This paper explores the opportunities afforded to secondary students in their mathematics learning through game building as an assessment tool. More specifically: 1). Does the overall experience of game design and building as an assessment tool improve students’ mathematics achievements as demonstrated in their test scores? 2). What emotion prevails among students in the game building process? This study uses mixed methods for collecting data, and focuses on behavioral and cognitive outcomes. A total of 40 secondary students participated in the study either in the treatment or the control group. Data collected included pre- and post-tests, unit tests, and pre- and post-surveys. The results showed that the treatment group experienced significant achievement gains, although there was no difference between the treatment and control groups. INTRODUCTION Rapid technological development is transforming our economy, culture, and society, and “redefining what and how and with whom we learn” (Dede, 2008, p.80). Scholars (Ito, 2009; Jenkins, et al, 2006) argue that we are moving towards a participatory culture that, according to Dede (2008), is changing our views of knowledge, expertise and learning, resulting in a seismic shaking of the foundation of formal education. Consequently, access to this participatory culture is growing increasingly more important and has, in many ways, becomes a new hidden curriculum. This is potentially creating new gaps between youths who will succeed and who will fail in their future educational and career pursuits (Jenkins, et al, 2006). This calls for new ways of technology integration into school curricula and stresses the need to empower students with knowledge for the emerging participatory culture. To answer such call, digital game (hereafter game) building is proposed to address this challenge, for it active knowledge creation instead of passive information consumption (Li, 2010, Games & Squire, 2011). It also provides the opportunity for students to participate in their learning through embodiment and action to cognition (Kafai, 2006; Papert, 1993). By integrating game design in classes, students can learn with increased engagement and enhanced understanding of mathematical concepts. This is done through rich creative thinking and problem-solving situations whereby students experience a variety of learning opportunities. This paper explores the opportunities afforded to secondary students in their mathematics learning through game building. Specifically, it attempts to answer the following questions: 1. Does the overall experience of game design and building as an assessment tool improve students’ mathematics achievements as demonstrated in their test scores? a. Does student achievement improve from pre-game building to post game building? Does the game-building group achieve improved mathematics learning compared to the traditional methods group? 2. What emotion prevails among students in the game building process? THEORETICAL PERSPECTIVE This paper is rooted in the theoretical framework of ‘enactivism’ when applied to educational technologies. Enactivism is proposed for the application of instructional design and technologies because it provides a more encompassing philosophical stance than other viewpoints (Li, Clark et al, 2010). While objectivism and constructivism argue that the world is dichotomous (sensory behaviour is separate from cognition), enactivism rejects this dualism (Li, Clark et al, 2010). Enactivism argues that sensory behaviour and cognition are inseparable, it focuses on the importance of embodiment and action to cognition (Holton, 2010; Thompson, 2006). In enactivism, thinking and cognition are grounded in action (Holton, 2010) or, learning occurs through action. Agents (learners) are changed through the action of learning (Proulx, 2004). Furthermore, learners interact with their environment. Through these interactions, learners come to understand features held or offered by objects and events (Davis, Sumara et al. 2008). Further, enactivism places emphasis on knowledge co-authoring; in a participatory culture this makes enactivism a suitable framework to design learning. Digital game design and building provides participants with opportunities that promote knowledge co-authoring and embodied learning. The game design and building process allows emphasis to be placed on doing; a core characteristic of enactivism. In this study, students build their own digital games as a way to assess their mathematics learning. The assumption is that through the design process, the final product will be influenced by the students’ cognitive process and thereby embody their learning experience. The research goals of this study were exploratory and descriptive rather than conclusive. It is also the intention of this study to extend the enactivist perspective through description of theoretically relevant practice within the present context. Digital Game Building as Assessment: A Study of Secondary Students’ Experience Qing Li Towson University li@towson.edu file:///C:/Users/asmith/Documents/ABSEL/2013/unedited/AssessmentGameHighSchoolABSEL2013_0122final.doc#ENREF_1#ENREF_1 file:///C:/Users/asmith/Documents/ABSEL/2013/unedited/AssessmentGameHighSchoolABSEL2013_0122final.doc#ENREF_3#ENREF_3 Page 75 - Developments in Business Simulation and Experiential Learning, volume 40, 2013 LITERATURE REVIEW Digital game-based learning is a promising area of research and debate as it promotes contextualized learning, creates and harnesses motivation in learning capacities and, encourages curiosity (Gee, 2003; Shaffer, Squire, Halverson & Gee, 2004; Paras & Bizzocchi, 2005; Prensky, 2006; Kirriemuir & McFarlane, 2004). Growing evidence supports experience-based learning and it can be argued that a video game is a set of experiences that a player participates in from a particular perspective (Gee, 2008). Beyond playing games, the experience of designing a computer game for mathematics has been shown to be effective among elementary students (Li, 2010; Kaifa, 1996). The design process pushes the designer to focus key ideas from various aspects of their knowledge. Game design was shown to foster communication, collaboration, and developed understanding among designers (Kafai, 1998). Further to design, the building of computer games (game building) empowers students to become creators rather than passive consumers of games (or solely game players) (Li, 2010). Game building provides students the opportunity to work as scientists by creating models that simulate real world phenomenon, investigate, and critique each others’ models. Through this process, students are able to modify and evaluate their own mental models and deepen understanding (Colella, 2001; Colella, Klopfer, and Resnick, 2001). Programming simulations and games has been shown to foster skills in problem solving and creativity (Klopfer, 2009). Clear indicators of learning have been shown through the learning-through-game-building method: at the elementary level for math and science (Li, 2010) and the secondary level for physics (Klopfer, 2009). Game-building has been found to increase engagement, enhance understanding of mathematics concepts, and introduce students to computer programming as a modality for creative thinking and problem solving (Li, 2010). Game-building puts design into the students’ hands, such student-controlled activities can create intrinsic motivation, presenting fertile ground for student engagement and learning (Shernoff et al., 2003). Intrinsic motivation is defined as the enjoyment of doing an activity which is “rewarding in and of itself...apart from its end product or any extrinsic good that might result from the activity”: a concept that fuelled “flow theory” (Nakamura & Csikszentmihalyi, 2002, p.89). Emotions and reactions such as joy, confusion, anger, apathy, frustration and boredom influence a person’s learning experiences and affect flow. Chan and Ahern (1999) further posit that flow describes a state where a person’s experiences are positively maximized when a balance is reached between challenge and skill. Frustration, although deemed a negative emotion, presents an affective indicator of current challenges and obstacles, of which the person can feed off of before apathy, anger, confusion or boredom sets in (Nakamura & Csikszentmihalyi, 2002). Furthermore, it can be used as a measurement to gauge when challenge presents itself; therefore, allowing moderators of an activity (i.e. teachers, game systems, game designers) the opportunity to adjust levels or teach a skill sufficient to overcome the immediate obstacle (Gilleade & Dix, 2004) Due to students’ on-going relationship with technology, learning through game building has encouraged researchers to perform empirical studies in the subject areas of mathematics (Harel, 1991; Kafai & Carter Ching, 1996; Noss & Hoyles, 2006), computer science (Korte, Anderson, Pain & Good, 2007) and science (Kafai, Carter Ching & Marshall, 1997; Li, 2010). In these studies, students’ game-building experiences were analyzed and interpreted in order to find patterns, processes, and connections between building and learning. Kafai and Carter Ching found that game making was relevant to children’s ‘real world’ as well as that it provided them the opportunity to explore math concepts in other ‘real worlds’ through the process of game design (Kafai & Carter Ching, 1996). Additionally, in working with children aged 6 to 8, Noss and Hoyles found that the construction of mathematical models through game design allowed for the expression of math ideas through models to facilitate reflection and discussion. This reflection and discussion in turn allowed for the construction of rich understanding of math (Noss & Hoyles, 2006). Where this ‘real world’ experience and metacognition may not occur through traditional methods, the design of models and games seemed to allow for this to occur. Kafai and Carter Ching had 26 elementary students aged 10 to 12 years design and implement interactive multimedia resources in science for younger students. It was found that constructing multimedia applications provided the students context to learn about and with technology. The creation of content and animation required deeper thinking than just the creation of an interactive quiz . The creation of content and animation proved to be beneficial in the area of technology and science for students when completed in tandem with science material (Kafai, Ching & Marshall, 1997) Through the creation of digital games by 21 students aged 7 to 11, evidence was found that students increased their understanding of the subject matter of their game (math, science, and technology). The student experience in game-building suggested that the game building experience can enhance both learning of the game design process and their subject matter. Additionally, problem solving skills were enhanced among the participants (Li, 2010). While an increasing number of research studies have explored the educational benefits of game play, less attention has been paid to the investigation of the benefits of game building. Specifically, little literature exists that focus on assessment through game building and designing in secondary mathematics. With clear indicators shown through game building among elementary students (Li, 2010; Klopfer, 2009; Harel 1991; Kafai & Carter Ching, 1996; Noss & Hoyles, 2006; Kafai, Carter Ching & Marshall, 1997;) and undergraduate students (Anderson, Pain & Good, 2007) merit is given to study the effect game building has on learning outcomes in secondary mathematics students. METHODS This study uses mixed methods for collecting data, and focuses on behavioral and cognitive outcomes. The primary Page 76 - Developments in Business Simulation and Experiential Learning, volume 40, 2013 approach to this study is an experimental model with a case study nested within it (Creswell, 2003). The mixed method approach advances “one method...nested within another method to provide insight into different levels or units of analysis”, thereby drawing on all possibilities (Tashakkori & Teddlie, as cited in Creswell, 2003). Furthermore, this study’s concurrent structure in collecting data simultaneously provided a broader perspective on the phenomena with the qualitative data helping to describe aspects the quantitative data cannot address (Creswell, 2003). PARTICIPANTS AND COURSES The study was conducted in a high school in Canada. A total of 40 secondary students participated in the study either in the treatment or the control group. The treatment group consisted of 27 students (7 males and 20 females) who built games for the specific math units. The control group was comprised of 13 students (3 males and 10 females) who learned mathematics in the traditional method. The game-based learning project included five essential components: Kodu tutorials, game planning, game building, game completion, and game play (see Table 1 for details). First, students were required to complete game building tutorials in Kodu or the building platform of their choice. The Kodu tutorials introduced students to programming in Kodu and provided sample ideas for their mathematics games. Then, during the ‘game planning’ phase, students developed a written game proposal focusing on particular math concepts. The proposal required a game storyline, target audience, and a student generated list of mathematical components to be included in their games (see Appendix 1 for details). The proposals were approved by a teacher to ensure that the students intended to include the required mathematics elements. The project occurred over a period of six weeks at the school. DATA AND INSTRUMENTS The study used both quantitative and qualitative data to determine the effect of game building on student learning. Data was collected at various stages during the project as outlined in Table 2. Quantitative data was collected in three forms depending on the participant group. Treatment participants took pre- and post-game tests before and after their game building. They also took the unit test after building their games. The pre-game and post-game tests were modified forms of questions from the end-of-unit tests. Control data was collected from the unit tests normally given to all students. The test items for the unit tests, the pregame tests and the postgame tests were drawn from the same test item bank. This way we made sure that the question format and level of difficulty remained consistent between the unit test and pre-game/post-game evaluations. All evaluations and unit tests were paper based and completed by individual participants. Qualitative data were collected via open-ended survey questions administered to the students in the treatment group. Surveys were administered prior, during, and after game building. Surveys included both open-ended narrative questions and multiple-choice questions. The exit survey focused on student learning experiences with technology and their impressions of its impact on their math learning. Independent or dependent t-tests, depending on the specific problem, were used to answer the first research question. The second research question was answered mostly by descriptive statistics. SUMMARY OF RESULTS ACHIEVEMENT The first research question examined the impact of the game building experience on secondary student achievement. This was explored from two perspectives: 1) treatment group pre and post-tests comparison, and 2) control and treatment achievement comparison. The first perspective looked at whether student achievement scores improved from pre-game building to post-game building. To evaluate the impact of the game building experience on achievement, a paired t test on the mean difference between evaluation results from pre and post game building was performed. The mean difference was .07 with a standard deviation of 0.17 (n=25). The resulting test statistic was t = 2.09, p=0.0233, suggesting that the population mean difference was greater than zero (alpha=.05) and that the mean test results of post-evaluation were greater than the pre-evaluation. That is, students’ achieved statistically significant higher scores in the post- test than in the pre-test. Table 1 Summary of Project Activities and Project Deliverables Activity Project Deliverable Kodu Tutorials None Game Planning Game Proposal (Teacher approval required) Game Building Kodu Game File Game Completion Game Summary Game Play Peer Evaluation Page 77 - Developments in Business Simulation and Experiential Learning, volume 40, 2013 The second perspective looked at student achievement by comparing the game builders’ test scores to the test scores of the students in the traditional environment. This comparison of treatment and control groups was done for the overall test results. For the control group (n=14), the mean was 57.71% and the standard deviation was 26.32%. For the treatment group (n=15), the mean was 61.27% and the standard deviation was 21.75%. Since the sample sizes were borderline large, it was decided to perform both the parametric and non-parametric tests to determine if there was a difference. An F-test was first conducted to determine if the population variances were equal. The test resulted in a test statistic of F=1.46, p=0.49 indicating the variances were equal. The independent Student t-test and the Mann-Whitney test both evaluated whether the mean test results of the treatment group were different from the control group. The independent t-test was t =-0.40, p=0.6943. For the Mann- Whitney test, the resulting test statistic was U=90, which resulted in a p-value of 0.47. Therefore, the data suggested that the population means were not different. The consistent results from both the Mann-Whitney and the independent t-test indicated that the population means were the same. That is, regardless of our assumptions of normality, there was no significant difference between the test scores of the game builders and the traditional learners. EMOTIONS Next question attempted to understand the students’ emotions throughout their experience of game building. Specifically it focused on the following emotions: happy, excited, smart, proud, confused, frustrated, bored, and annoyed. Table 3 summarizes the frequency of these emotions during each phase of the learning-as-game building process. Student emotions were also looked at from the perspective of the project as a whole. To do this evaluation, the frequency of responses for each emotion was added from each phase of the project. For example, the number of responses for ‘excited’ was added for all the phases: building, testing, modifying, and finishing. In general, excited was the most frequent response with happy and proud closely following. All the positive emotions had more frequent response than any of the negative emotions, indicating that the positive emotions were the prevailing emotions for the project as a whole. The overwhelming emotions of happy, proud, excited, and smart seem to stem from the completion and sharing of a working product. Out of the negative emotions students seemed to be more frustrated than they were confused or annoyed. As to be expected with the individual phase comparison results, bored was the least frequent response overall. The analysis of the data showed that ‘finishing the game’ was the phase where none of the negative emotions were identified. Rather, in this phase, all of the positive emotions, particularly the feeling of happy, proud and excited, were strongly demonstrated. In addition, ‘bored’ was the least frequent emotion identified by students in every phase of game building. Further analysis of the students’ qualitative narratives indicated that those students that chose the emotion ‘bored’ did so in relation to tedious tasks required for modifications in the game. DISSCUSSION This study demonstrates a strong correlation between game building’s effect on the mental and emotional experiences of secondary students’ math learning. The treatment group has experienced significant achievement gains, although there is no difference between the treatment and control groups. It is important to note that although game building was used as a way to assess students’ learning, the test data for this study was collected from traditional paper tests. From this, we can surmise that game -building as assessment provides a meaningful way to help students’ learning, and the results is at least as effective as traditional learning from the conventional achievement point of view. The fact that these students expressed overwhelmingly positive emotions to the game building process, suggests that they have learned mathematics with great passion, and they achieved the same results on paper evaluations as those students who learned in the traditional Table 2 Data Collected and Timeline Timeline Data Collected (Treatment group) Data Collected (Control Students) Pre-game building  Pre-game Test  Pre Survey  Pre-game Test During Game building  Interim Survey After Game Building  Post-Game Test  Unit test  Post Survey  Post-Game Test  Unit test Page 78 - Developments in Business Simulation and Experiential Learning, volume 40, 2013 method. When students experience positive associations with their experience we have reason to believe that this will lead to improved long-term memory and retrieval, and enhance creative problem solving (Bichelmeyer, et al. 2009; Erez and Isen 2002; Isen and Reeve 2005). The experience of building games empowers students to become creators rather than passive consumers of games and knowledge, preparing them for the 21st century’s participatory culture. REFERENCES Bichelmeyer, B., J. Marken, et al. (2009). Fostering Affective Development Outcomes: Emotional Intelligence. Instructional-Design Theories and Models. C. Reigeluth and A. Carr-Chellman. NY, Taylor and Francis, Publishers. III: 199-224. Creswell, J. W. (2003). Research design: Qualitative, quantitative, and mixed methods approaches (2nd ed.). Thousand Oaks, CA: Sage. Cobb, P., Yacke, l. E., & Wood, T. (1992). Interaction and learning in mathematics classroom situations. Educational Studies in Mathematics, 23(1), 99-122. Dede, C. (2008). A seismic shift in epistemology. Educause Review, 34(3), 80-81. Erez, A. and A. M. Isen (2002). The influence of positive affect on the components of expectancy motivation. Journal of Applied Psychology. 87(6),1055-1067. Gee, J. P. (2003). What video games have to teach us about learning and literacy (2nd ed.). New York: Palgrave/ Macmillan. Harel, I. (1991). Children designers: Interdisciplinary constructions for learning and knowing mathematics in a computer-rich school. Norwood, NJ: Ablex Publishing. Isen, A. M. and J. Reeve (2005). The influence of positive affect on intrinsic and extrinsic motivation: Facilitating enjoyment of play, responsible work behavior, and self -control. Motivation and Emotion, 29(4). 297-325. Ito, M. (Ed.). (2009). Hanging Out, Messing Around, and Geeking Out: Kids Living and Learning with New Media. Cambridge: MIT Press. Jenkins, H., Purushotma, R., Weigel, M., Clinton, K., & Robison, A. J. (2006). Confronting the challenges of participatory culture: Media education for the 21st century (part two). Nordic Journal of Literacy, 2, 97– 113. Kafai, Y. (2006). Playing and making games for learning: Instructionist and constructionist perspectives for game studies. Games and Culture, 1(1), 36-40. Kafai, Y., & Carter Ching, C. (1996). Meaningful contexts for mathematical learning: the potential of game making activities. Proceedings of the International Conference on Learning Sciences (ICLS ‘96), Bloomington, IN, USA. Kafai, Y., Carter Ching, C., & Marshall, S. (1997). Children as designers of educational multimedia software. Computers & Education, 29(2-3), 117-126. Kafai, Y., Franke, M., Ching, C., & Shih, J. (1998) International Journal of Computers for Mathematical Learning 3, 149–184. Klopfer, E., Scheintaub, H., Huang, W., Wendel, D., & Roque, R. (2009) The simulation cycle: combining games, simulations, engineering, and science using StarLogo TNG. E-Learning, 6(1), 71-96. Li, Q. (2010). Digital game building: Learning in a participatory culture. Educational Research, 52(4), 427 -443. Noss, R., & Hoyles, C. (2006). Exploring mathematics through construction and collaboration. In R. K. Sawyer (Ed.), The Cambridge Handbook of the Learning Sciences (pp. 389-408). New York: Cambridge University Press. Otrel-Cass, K., Forret, M. & Taylor, M. (2009).Opportunities and challenges in technology rich classrooms: Using the Scratch software. SET:Research Information for Teachers, 1, 49–54. Papert, S. (1993). The children's machine: Rethinking school in the age of computers. NY: Basic Books. Provenzo, E. F. (1991). Video kids: Making sense of Nintendo. Cambridge, Mass.: Harvard University Press. Wolfe, P. and R. Brandt (1998). What do we know from brain research? Educational Leadership, 56(3): 8-13. Table 3 Student Emotions in Game Building Process Emotion Building Testing Modifying Finishing Total Happy 7 14 7 20 48 Excited 14 11 8 17 50 Smart 12 7 5 13 37 Proud 13 9 2 20 44 Confused 15 7 8 0 30 Frustrated 14 10 10 0 34 Bored 4 1 6 0 11 Annoyed 7 7 6 0 20 http://www.sciencedirect.com.ezproxy.lib.ucalgary.ca/science/journal/03601315 http://www.sciencedirect.com.ezproxy.lib.ucalgary.ca/science?_ob=PublicationURL&_tockey=%23TOC%235956%231997%23999709997%2312193%23FLP%23&_cdi=5956&_pubType=J&view=c&_auth=y&_acct=C000051253&_version=1&_urlVersion=0&_userid=1067480&md5=3305193205ed726e4ba5ac0 Table of Contents Volume 40, 2013 Leo Investor SERIOUS PLAY: ON-LINE PEPULATOR SERIOUS PLAY:ON-LINE GAME "MANAGE-ART" The Game Within the Simulation Game the Research Method Concept and Project With Game Design Implementation Managing Human Resources Simulation Rising Stars: A Study of Creativity as Experiental Learning in County Government Why do Gen Y Students Study Abroad? The Relationship Between Individual Growth and the Intent to Study Abroad Our Favorite Assignment: The In-Basket Activity Mirror-Mirror: How Can We Better Understand Our Consumption Behavior? Reflective Thinking, Writing and Imagery as Tools to Better Understand Abstract Marketing Concepts Increasing Intuitive Decision Making Speed and Accuracy by Further Understanding Intuitive Decision Making Using Emotional Means Group Exams: Are they Relevant and Reliable as a learning Tool? Integrative Learning: Exploring Opportunities in Business Simulations Simulated Tabletop Exercise for Risk Management - Anti Bio-Terrorism Multi Scenario Simulated Tabletop Exercise Simulated Tabletop Exercise for Electric Power Saving Management in a Small Organization Merging the Case Method and Simulation in Management Education: Is it Possible? Experiential Learning in Accelerated Human Resource Management Courses Digital Game Building as Assessment: A Study of Secondary Students' Experience Using an Investment Fund Simulation Integrated with a Business Game Creating Simulation Conditions Based On Real World Data Simulation Games in Training New Management Methods A Summary Overview of Cultural Differences in Higher Education Innovations And Future Directions In Education: Case Review For Best Online Teaching Practices An Implicit Measure of Forecasting Accuracy Immediate Feedback and Assessment Technique (IF-AT) testing forms: An overview of the tool and uses Teaching Secondary Mathematics: Pre-service Teachers' Digital Game Design, Pedagogy and 21st Century Skills Follow The Leader II Business Simulations: From Punch Cards to Web-Based University Engagement: Good Neighbors Budget Allocation Exercise Customizing Business Simulations: An Exploration and a Hierarchy A Review of the Simulation Research in the Academy of Management Journal: Suggestions for Strengthening the Research Conducted by ABSEL Members ABSEL Reflections: 40 Years Of Excellence, Now Going Forward How Many Options do Multiple-Choice Questions Really Have? The Management/Accounting Simulation Now 100% Web-based Design, Server Side Programming, and Implementation A Large-Scale Game Test of Early-Determined Game Finishes Large-Scale Business Games for Assurance of Learning Purposes The Gamification Of Education OB Simulation: A Hands-On Demonstration Converting Simulations for the Online Environment: The New Ginseng Game Repositioning Brands with the Web-Based Product Positioning Map Graphics Package Implications of Regulatory Focus Theory for Simulation and Experiential Learning Riskware: A Game for Teaching Software Project Risk Management Teamwork for Decision-Making Through Games: The Case of On-Line "Manage-Art" Virtual Pepulator, a Model for Teaching Negotiation Small-Scale Business Games for Assurance of Learning Purposes Incorporating Intellectual Property Issues into a Business Simulation© Nationalization and Privatization in a Computer-Assisted Business Game An Investigation of the Relationship of Plan Quality, Forecast Accuracy and Earnings Performance Under Equal and Unequal Starting Positions in Marketing Positions in Marketing Simulations Measuring the Performance Ranking Curve in Marketing Simulation Games The Effectiveness of SDM Method in Business Simulation Game 40 years of Brazilian S&G - Analysis and Perspectives Accounting For Externalities: Harnessing The "Face-In-The-Mirror" Phenomenon The Role Of Simulations In Organizational Learning: Building Individual Absorptive Capacity Assessing Project Management as an Academic Learning Outcome (ALO) At the Inflection Point: Designing Army Assessments "Economics in Practice" A Simulation Game for High School Students Teaching the Basics of Economics and Entrepreneurship Fueling the Force: Exploring Leader Priorities Implementing Mental Models: Extending Insight and Whole Person Learning Using Business Simulations To Introduce Business Concepts