Microsoft Word - 11- Watts Barker Layout 133 Transversal: International Journal for the Historiography of Science 2018 (5): 133-145 ISSN 2526-2270 www.historiographyofscience.org © The Authors 2018 – This is an open access article IDTC Dossier Methods and Cognitive Modelling in the History and Philosophy of Science–&–Education Meeting Galileo: Testing the Effectiveness of an Immersive Video Game to Teach History and Philosophy of Science to Undergraduates Logan L. Watts1 Peter Barker2 Abstract: Can video games teach students about the history and philosophy of science? This paper reports the results of a study investigating the effects of playing an educational video game on students’ knowledge of Galileo’s life and times, the nature of scientific evidence, and Aristotle’s and Galileo’s views of the cosmos. In the game, students were immersed in a computer simulation of 16th century Venice where they interacted with an avatar of Galileo and other characters. Over a period of two weeks, 71 undergraduates were exposed to lectures about Galileo and the Copernican revolution in a traditional classroom setting. However, only half of the students (i.e., experimental group) also played the game. The other half (i.e., control group) were only exposed to lectures. The knowledge of both groups was assessed at the beginning (i.e., pre-test) and end (i.e., post-test) of the two-week period. The results demonstrated objective improvements in knowledge for the experimental group while the control group showed virtually no change. Implications of these findings for teaching and learning the history and philosophy of science are discussed. Keywords: Gamification; Educational Video Games; Cognitive Models; History and Philosophy of Science Education; Galileo Galilei Received: 25 January 2018. Reviewed 9 September 2018. Accepted: 28 September 2018. DOI: http://dx.doi.org/10.24117/2526-2270.2018.i5.10 This work is licensed under a Creative Commons Attribution 4.0 International License. ____________________________________________________________________________ Gamification, or the integration of gaming elements in non-game contexts (Deterding et al., 2011), is emerging as a “hot topic” in higher education. However, empirical research on the subject to date has mostly focused on the impact of gamification on students’ motivational 1 Dr. Logan L. Watts [Orcid: 0000-0001-7629-0188] is an Assistant Professor in the Department of Psychology at the City University of New York – Baruch College & The Graduate Center. Address: 55 Lexington Ave, Box B8-215, New York, NY 10010, USA. E-mail: logan.Watts@baruch.cuny.edu 2 Dr. Peter Barker [Orcid: 0000-0002-1443-4227] is a Professor in the Department of the History of Science at the University of Oklahoma. Address: Norman, OK, 73019 USA. E-mail barkerp@ou.edu Please address all correspondence to Dr. Peter Barker. Meeting Galileo: Testing the Effectiveness of an Immersive Video Game to Teach History and Philosophy of Science to Undergraduates Logan L. Watts – Peter Barker 134 outcomes (e.g., course reactions, intrinsic motivation) as opposed to cognitive outcomes like improvements in knowledge (Dicheva et al., 2015). In this paper, we describe an empirical study of the cognitive benefits of gamifying traditional lecture courses in a novel subject area – the history and philosophy of science. Game Development The lecture courses investigated in the present study covered a pivotal time period in the history and philosophy of science. The focus of the lectures was on Galileo’s early career and his development of telescopic evidence against the prevailing cosmology of Aristotle and in favor of the new sun-centered cosmology of Copernicus. Galileo’s findings immediately raise issues about verification and falsification in science, as well as the rational response to dogma. At the same time, Galileo was attempting to improve his career by gaining a position at the court of the Grand Duke of Tuscany, as well as encountering Church objections to his support for Copernicus. This social and cultural material is particularly difficult to convey to undergraduates, as it runs counter to implicit contemporary social norms. Thus, the designers of the game hoped that this material would be conveyed more effectively in an immersive game where the students could interact with avatars who acted out different norms within a historically accurate context. Project background. In 2011 Peter Barker of the History of Science program at the University of Oklahoma was approached by entrepreneur Brent Lollis of Parnassus LLC, to join a project using the full resources of video game technology to teach the history and philosophy of science. The central idea of the project was to allow students to learn directly from an avatar of a historical figure, while playing a game that would recreate as accurately as possible their historical setting, including the physical and social environment. By 2011, games like Assassin’s Creed had already set a high bar for quality graphics, and historical realism, admittedly with a strong mix of fantasy (Désilets, Patrice, Raymond and May, first release November 13, 2007). At Lollis’s instigation, Barker visited Tigar Hare Studios in Los Angeles, a company that had provided high-quality graphics for some of the Call of Duty and Lord of the Rings series, among many other projects. As a result, one of the two principals in the studio, Michael Tigar, joined the Parnassus project. Lollis and Barker examined a wide range of historical figures for the pilot project. They decided that Galileo should be the central figure in the first game developed, largely because of his name recognition both inside and outside academe. The prototype game was developed by Lollis, Barker and Tigar. Lollis served as director and producer. Barker and Lollis wrote most of the script, with Barker ensuring historical accuracy. Tigar both supervised and contributed to the construction of the avatars and the game setting. By 2013 a beta version of the game was available for testing, and in Fall 2013 Barker was scheduled to teach two undergraduate classes that would cover Galileo and his times. This seemed to be an ideal opportunity to test the game in a university setting. Barker and Lollis contacted Dr. Michael Mumford at the University of Oklahoma’s Psychology Department. Dr. Mumford invited Logan Watts, then a doctoral candidate, to conduct the research and testing. Barker again contributed historical material in the form of banks of questions that were screened and pilot-tested by Watts. Setting and characters. Galileo Galilei (1534-1642) taught at Pisa from 1588 and Padua from 1592. Padua was part of the Republic of Venice, and Galileo spent much time in the city with contemporary intellectuals. In 1610 Galileo succeeded in his campaign to become a courtier of the Grand Duke Cosimo de Medici (1590-1621) in Florence, in part because of his discovery of the moons of Jupiter, which he named after the Medici family. Beginning in 1611 he traveled throughout Italy, demonstrating telescopes, which he constructed himself, and advocating Copernican cosmology. When the Church found Copernicanism heretical in 1616 Meeting Galileo: Testing the Effectiveness of an Immersive Video Game to Teach History and Philosophy of Science to Undergraduates Logan L. Watts – Peter Barker 135 Galileo suffered no personal consequences, because he was protected by Duke Cosimo and several high-ranking members of the Catholic church.3 One of Galileo’s defenders, Cardinal Maffeo Barberini (1568-1644) became Pope Urban VIII in 1623. He gave Galileo permission to write a new book comparing the Copernican system with the prevailing Aristotelian orthodoxy. The book appeared in 1632, but the political situation had changed and Urban was offended by the content. Galileo was summoned to Rome and tried before the Inquisition, where he was convicted of being “vehemently suspected of heresy”. He was forced to publicly deny his support for Copernicanism and spent the remainder of his life under house arrest at his villa outside Florence (Heilbron, 2012). Galileo’s contributions to science are usually divided into two main parts: his advocacy of Copernicanism based on his discoveries with the telescope, and his work on the science of motion (Cohen 1985, chapter 4 and 5). These subjects are connected by opposition to the received physics and cosmology of Aristotle. Aristotle constructed a spherical universe centered on a spherical stationary earth. The four terrestrial elements (earth, water, air and fire) were confined to the region inside the spherical shell supporting the moon. Beyond the moon were concentric spherical shells supporting other heavenly bodies, bounded by a shell carrying the fixed stars. All these shells were composed of a single substance (aether) with the result that the heavens were perfect, eternal and unchanging, except for circular motions centered on the earth. The moon, for example, was assumed to be a perfect sphere. By contrast, in the region of the four terrestrial elements, the motion was in straight lines and ended rapidly. Also, composite bodies visibly changed over time as they added or lost particular elements (Cohen 1960/1985, 11-23). Galileo’s telescopic discoveries called many of these claims into question, and especially the rigid distinction between the heavens and the earth. Among the standard topics in philosophy of science exemplified by Galileo’s career are paradigm choice, theory choice and falsification, as well as the reliability of observational evidence and scientific instruments (Curd, Cover and Pincock 2012; Kuhn [1962] 2012; Popper 1959, [1963] 1981). For example, although Galileo’s telescopic discoveries showed that many Aristotelian claims were false, they did not establish that Copernicus’s claims were true. He also supported the use of mathematical reasoning to settle issues originally treated in non- mathematical natural philosophy. If, for example, the moon is a perfect sphere, then the line dividing the bright from the dark portion should be a perfect circle. Galileo’s observations showed that the shadow line was interrupted by mountains, valleys and craters. 3 There were many reasons why the Church only acted against Copernicus’s De revolutionibus in 1616. Initially, the unsigned preface by Osiander (Copernicus 1543, fol i V - ii R), suggesting that the book should be read hypothetically, may have caused confusion over the author’s real intent. Quite independent of this, Church leaders like Cardinal Roberto Bellarmine took the viewpoint, from the outset, that Copernicus’s book should be read hypothetically. Apart from Osiander’s preface, however, De revolutionibus uncompromisingly presented heliocentrism as the real structure of the cosmos. Kepler publicly identified the preface’s real author in 1609 (Kepler, tr. Donahue 2015, p. 4 text to n. 28.). After that the possible heresy of the book became unavoidable. Galileo’s telescopic discoveries undermined Aristotle’s and Ptolemy’s geocentric theories, but failed to establish the truth of heliocentrism. The hybrid cosmos unveiled by Tycho Brahe in 1588 retained a stationary central earth and equally explained all Galileo’s evidence. Brahe’s position became a favorite compromise among progressive elements in the Church, for example the Jesuits in Rome (Graney 2015). But it was also possible to accommodate Galileo’s discoveries with very modest adjustments to traditional Aristotelian cosmos (Ariew 2006). The timing of the 1616 proceedings depended primarily on Galileo’s noisy success in Italy after 1610, and attempts to discredit or undermine him by the Dominicans Tomasso Cacini and Niccolo Lorini (Finocchiaro 1989, 134-41). That nothing bad happened to Galileo before 1632-3 depended crucially on his patronage connections with Cardinal Bellarmine, Cardinal Barberini (later Urban VIII), Prince Cesi, Grand Duke Cosimo de Medici and his mother the Grand Duchess (Biagioli 1993). Meeting Galileo: Testing the Effectiveness of an Immersive Video Game to Teach History and Philosophy of Science to Undergraduates Logan L. Watts – Peter Barker 136 While in Venice during 1609 and 1610 Galileo made his first celestial discoveries and produced the book that reported his results as well as attracting the patronage of Duke Cosimo. He also demonstrated the telescope to friends, colleagues and possible patrons, using the platform at the top of the famous Bell Tower (Campanile) in St. Mark’s Square. The action of the Galileo game was therefore set in and around a digital reconstruction of St. Mark’s Square in 1609, including the waterfront on the Grand Canal, all the surrounding buildings, the Bell Tower, the Basilica (Cathedral) of St. Mark, and the façade of the Doge’s Palace. The game began with players finding their way along the Grand Canal to the entrance to the Square, where they would encounter an avatar of Galileo arguing with a Dominican friar. After listening and responding to several exchanges about astronomy, physics and cosmology, players would be encouraged to visit Galileo in a fictional workshop, elsewhere in the Square, or to take a gondola ride that triggered a dream sequence of a conversation with Aristotle, providing basic information about his views on physics and cosmology. At the end of the dream, sequence players were returned to 1609. Next, players who visited the workshop could help Galileo assemble a telescope, and then proceed to the top of the Bell Tower to help Galileo with his observations. It took several trips from the workshop to the Bell Tower to produce a telescope with high enough magnification to see celestial novelties. On each occasion the player would discuss the observations and results with Galileo, leading ultimately to Galileo’s conclusion that the Moon was not a perfect sphere because the shadow line showed mountains, valleys and craters. Although there was a clear direction of play in making and using successively better telescopes, all the sites could be visited repeatedly, and in any order, including a brief side trip to Galileo’s trial in 1633. However, if players visited the Bell Tower without first helping Galileo build a telescope, they would find it empty. As they successfully completed tasks and visited different locales, players accumulated rewards on a toolbar. At the end of the game, they were able to unlock a science-fiction transporter device inside St. Mark’s Basilica, which carried them to a surprise ending that was intended as the final reward for successful players. The non-player characters (NPCs) in the prototype therefore consisted of Galileo, his Dominican opponent, Aristotle, and one other character who sometimes appeared mysteriously to give the players advice. However only Galileo and Aristotle were interactive in this version of the game. Primary tasks/objectives. Based on earlier work applying methods from cognitive psychology to problems in the history and philosophy of science (Andersen, Barker and Chen 2006; Barker 2007), Barker saw the educational goals of the game as encouraging the students to build mental models of Aristotle’s physics and cosmology, the contrasting cosmology of Copernicus, the historical setting of Galileo’s advocacy of Copernicus, and the general philosophical or methodological issues about science relevant to the case. From previous experience, Barker also knew that while students learned Aristotle’s physics and cosmology fairly easily, it was much harder to communicate Galileo’s cultural situation, for example, the role of the Church and the importance of patronage. Barker hoped that the interactive aspects of the game would provide a new and more effective way of communicating all this material, but particularly the cultural aspects. Several items were built into the game to achieve this. For example, before play began, players were asked to choose whether to play as a man or a woman.4 Throughout the game, Galileo would then address them using either male or female honorifics appropriate for a member of Duke Cosimo’s court, and would repeatedly refer to their connection to the Duke’s court and his own hopes of joining it. Similarly, information about the scientific method was not offered directly but embedded in the structure of the game dialogues. Mental model building was encouraged throughout the game by short interactive dialogues. For example, at the site in St. Mark’s Square where Galileo is arguing with a 4 Other possibilities were to be added after the beta version. Meeting Galileo: Testing the Effectiveness of an Immersive Video Game to Teach History and Philosophy of Science to Undergraduates Logan L. Watts – Peter Barker 137 Dominican opponent, the player is invited to join conversations about the nature of the Moon, the nature of stars, the spyglass (the word “telescope” had not been coined in 1609), the motion of the earth, whether Copernicus’s theory is offensive to the Church, whether the Sun can be the center of the planetary motions, as well as general conversations of Aristotle’s cosmos and Copernicus’s cosmos. The dream-sequence meeting with Aristotle encouraged dialogue about each of the four elements, the nature and contents of the heavens and the uniqueness of the cosmos according to Aristotle. The meeting with Aristotle was staged on the Acropolis as it would have appeared in about 325 BCE, with an introduction to the architecture as a bonus when the player and NPC moved around. In the tested version of the game, NPCs spoke directly to the player. The intention was to add speech recognition so that the player could reply verbally, but in the tested version the player’s replies were limited to selecting written prompts. Players were free to visit any site in any order, to complete as many of the tasks offered at the site as they pleased, and to return whenever they liked. However, players were encouraged to visit the Galileo/Dominican scene in St. Mark’s Square first. From here they would receive two sorts of prompts on direction of play. First, the toolbar that was a permanent feature of the screen would light up showing that they had found the first of four major sites (the Galileo/Dominican dialogues; Galileo’s workshop; the Aristotle dream sequence; the Bell Tower). After completing some of the Galileo/Dominican dialogues players would also be prompted by a thumbnail of the next site they should find, appearing briefly. As play continued the players also accumulated rewards on their toolbars (e.g., a key, a book) which would allow them to unlock a surprise reward in St. Mark’s Basilica at the end of the game. Psychological Learning Principles In order to maximize the amount of knowledge that students might acquire from playing the game, three psychological learning principles were considered throughout the game’s design, including active learning, cognitive load, and fidelity. Next, we describe each of these principles, evidence for their effectiveness at promoting learning, and examples of how the game’s design was informed by these principles. Active learning. Active learning does not refer to a single approach to teaching and learning, but rather a principle underlying many instructional strategies such as guided exploration and experimentation. This learning principle suggests that students derive enhanced cognitive and motivational benefits from instructional methods that allow students greater autonomy in the learning process, such as those that provide immersive experiences (Bell and Kozlowski, 2010). Traditional active learning methods include problem- solving activities, team exercises, role-playing simulations, and case studies (Meyers and Jones, 1993), but advancements in technology have opened the door for innovative applications of the active learning principle – including gamification. There is also strong empirical evidence for the effectiveness of active learning methods when compared with traditional methods (e.g., lecture) among students majoring in science, technology, engineering, and mathematics (STEM) fields. For example, in a meta-analysis of 225 studies, Freeman et al. (2014) found that STEM students taught through active learning methods scored 6% higher on knowledge tests compared with students taught through lectures. In addition, those receiving lectures were one-and-a-half times more likely to fail on these knowledge tests. In the present study, the active learning principle was designed into the game prototype by giving the learners extensive autonomy to explore the virtual world, interact with NPCs, and pursue a small number of key objectives (i.e., limited guidance). Cognitive load. People have limited cognitive resources available for processing complex information. The principle of cognitive load refers to recognizing these limitations when designing instructional methods (Sweller, 1988). Mayer and Moreno (2003) Meeting Galileo: Testing the Effectiveness of an Immersive Video Game to Teach History and Philosophy of Science to Undergraduates Logan L. Watts – Peter Barker 138 demonstrated in a series of experiments that segmenting and off-loading are two particularly effective strategies for reducing cognitive load. Segmenting refers to breaking down complex information into smaller, more manageable units. Off-loading refers to limiting unnecessary instances in which both visual and auditory channels are engaged simultaneously. In the present study, both of these strategies were used when designing the prototype to reduce cognitive load. For example, the prototype focused on a narrow window in Galileo’s life framed around his telescopic discoveries. Throughout the game, important concepts were spaced apart and delivered in small chunks (i.e., segments) as students explored and interacted with Galileo’s world. These chunks were designed to fit within the overarching narrative of understanding why Galileo’s telescopic discoveries were so controversial for his time. Off-loading was evident when the Galileo avatar interacted with other characters. When another character responded to Galileo in conversation, no subtitles were presented on the screen. Students would only hear the response verbally. This strategy limited unnecessary, extraneous processing that might have occurred if students were presented with both auditory information and text-based, visual information simultaneously, allowing students to devote more resources towards learning the key concepts. Fidelity. Two types of fidelity have been studied in the teaching and learning literatures: physical and psychological. Physical fidelity refers to the extent to which the instructional method, such as a simulation or game, models the physical environment of the “real world,” while psychological fidelity refers to how much the cognitive processes unfolding during learning align with the cognitive processes that unfold during real-world performance (Cannon-Bowers and Bowers, 2010). Physical fidelity has also been shown to enhance psychological fidelity (Kozlowski and DeShon, 2004). In the present study, the game prototype was designed with both physical and psychological fidelity in mind. Physical fidelity was strong as a result of paying careful attention to the historical accuracy of the game’s setting, characters and costumes. Game designers worked with historians to simulate as closely as possible how St. Mark’s Square might have appeared during Galileo’s life in the 16th century. As shown in Figure 1, Galileo’s avatar was modeled after a historical portrait of Galileo. Finally, psychological fidelity was also strong. Students explored the physical setting, experimented with different tasks (e.g., telescope construction), and engage in dialogue while being consistently addressed with honorifics that reinforced the importance of court culture to Galileo and his contemporaries. Hence, the cognitive processes activated during the game probably aligned well with how these cognitive processes might have unfolded if doing these activities in the real world. CAPTION: Figure 1. Generating a Historically Accurate Avatar of Galileo. Copyright Parnassus LLC. Used by permission. Meeting Galileo: Testing the Effectiveness of an Immersive Video Game to Teach History and Philosophy of Science to Undergraduates Logan L. Watts – Peter Barker 139 Although all three of these psychological learning principles – active learning, cognitive load, and fidelity – were considered in designing the game prototype, it is important to note that in some cases a balance had to be struck between these objectives. An obvious example of these tradeoffs may be observed in the relationship between physical fidelity and cognitive load. Although it would have been historically accurate to present all of the dialogue in the game in 16th century Italian instead of modern English, doing so would have surely interfered with learning in our English-speaking sample of students. In sum, we expected that considering all three learning principles in game design would yield a prototype that benefited student learning. Specifically, our hypothesis was that students who played the game and received lectures (i.e., experimental group) would achieve higher post-test scores on knowledge tests compared with students who only received lectures (i.e., control group). Method Participants. Undergraduate students were recruited by Watts as research subjects in two of Barker’s undergraduate classes. These classes were upper-level courses on the subjects of History of Science from the Greeks to Newton and The Scientific Revolution. Students received a course participation grade in exchange for taking part in the study, and they were offered the opportunity to complete an alternative essay assignment instead. All of the students in both classes, 71 total, chose to participate in the study. Approximately 66% of participants were male, with an average age of 22. Participants reported a variety of major areas of study, but the most common majors represented were the physical sciences (51%) and social sciences (14%). Procedures. The study protocol and materials were approved by the University of Oklahoma’s Institutional Review Board. The study took place during two weeks of the fall 2013 semester on the University of Oklahoma campus. Participants in both classes were randomly assigned to either the experimental group (n = 37) or control group (n = 34) while collecting informed consent forms. All of the participants began the study by completing a pre-test designed to measure their starting level of knowledge. Next, over the course of two weeks, the experimental group played the video game prototype and attended two, 2-hour lectures about Galileo and science in the early 17th century. The control group attended the same lectures but were not allowed to play the game until the study had concluded. Participants played the game in a computer lab on campus, and the average participant played the game for approximately 45 minutes. At the end of the two weeks, both groups completed the post-test. Measures Knowledge. Barker wrote 120 multiple-choice questions targeting four content areas, including Galileo’s life and times (34 questions), the nature of scientific evidence (24 questions), Aristotle’s model of the cosmos (32 questions), and Galileo’s model of the cosmos (30 questions). Each question provided four potential response options, with one correct response option. Knowledge of Galileo’s life and times was tested by questions such as, “What was the name of the city where Galileo used his telescope in the tower?” Knowledge about the nature of scientific evidence was tested by questions such as, “Why did Galileo object to a dogmatic defense of scientific claims?” For questions about Aristotle’s and Galileo’s cosmos, participants were instructed to “answer from Aristotle’s [or Galileo’s] perspective.” For example, participants were asked to answer a question about “the center Meeting Galileo: Testing the Effectiveness of an Immersive Video Game to Teach History and Philosophy of Science to Undergraduates Logan L. Watts – Peter Barker 140 of the cosmos” according to Aristotle and later asked to answer the same question from Galileo’s perspective. Because completing a pre-test can artificially boost participants’ scores on a post-test (i.e., practice effect; Shadish, Cook, and Campbell, 2002), a number of strategies were used to control for this limitation. For example, questions targeting each of the four content areas were randomly assigned to create two versions of a knowledge test (i.e., Test A and Test B). Both test versions consisted of the same number of questions targeting each content area. In addition, the order of administration was counterbalanced so that half of participants completed Test A for the pre-test (and Test B for the post-test) and the other half of participants completed Test B for the pre-test (and Test A for the post-test). Total scores on both versions of the test were strongly correlated (r = .81), providing evidence for the reliability of test scores. Satisfaction and learning reactions. At the end of the knowledge post-test, participants were asked to report their reactions to the game. Using a 5-point scale (i.e., 1 = strongly disagree, 3 = neither disagree nor agree, 5 = strongly agree), participants reported the extent to which they agreed with seven statements about their satisfaction with the game (e.g., “The game was fun”) and five statements about how much they thought they learned by playing the game (e.g., “I have a better understanding of Galileo’s life”). Finally, participants were asked to respond to four open-ended questions about what they liked most and least about the game, the most important thing they learned from playing the game, and how they thought the game could be improved. Results Effects on Knowledge. Pre-test and post-test scores were standardized on a one-hundred- point scale for each content area to improve interpretability. Thus, means (M) and standard deviations (SD) of test scores reported here represent the percentage of questions correct. Cohen’s d effect sizes are also reported, to give an indication of the magnitude of each effect. Cohen’s d provides a standardized estimate (i.e., in standard deviations) of knowledge gains (or losses) in the experimental group relative to knowledge gains (or losses) in the control group. By conventional standards, a Cohen’s d of .20 represents a small effect, .50 a medium- sized effect, and .80 or above a large effect (Cohen, 1992). Using an alpha level of .05, analysis of covariance (ANCOVA) procedures were used to test whether the experimental group (i.e., lecture + game) showed greater gains in knowledge than the control group (i.e., lecture only). Pre-test scores were entered as covariates to control for any differences in starting knowledge between the groups.5 For knowledge of Galileo’s life and times, the experimental group (M = 64%, SD = 14%) demonstrated statistically higher post-test scores than the control group (M = 59%, SD = 18%), F(1, 68) = 4.61, p < .05, d = .60. For knowledge about the nature of scientific evidence, the experimental group (M = 63%, SD = 14%) scored significantly higher on the post-test compared with the control group (M = 50%, SD = 23%), F(1, 68) = 8.36, p < .01, d = .64. For knowledge of Aristotle’s model of the cosmos, the experimental group (M = 72%, SD = 17%) outperformed the control group (M = 57%, SD = 20%) on the post-test, F(1, 68) = 9.44, p < .01, d = .44. For knowledge of Galileo’s model of the cosmos, the experimental group (M = 82%, SD = 13%) scored significantly higher on the post-test compared with the control group (M = 67%, SD = 27%), F(1, 68) = 12.62, p < .01, d = .67. When examining across all four content areas at once, not surprisingly, the experimental group (M = 70%, SD = 10%) outperformed the control group 5 Using analysis of variance (ANOVA) procedures, we also tested whether pre-test scores differed between the experimental and control groups. Across all four content areas, as well as on overall pre- test scores, no statistically significant differences were observed (p > .05). Thus, experimental and control groups showed comparable levels of knowledge at the beginning of the study. Meeting Galileo: Testing the Effectiveness of an Immersive Video Game to Teach History and Philosophy of Science to Undergraduates Logan L. Watts – Peter Barker 141 (M = 58%, SD = 19%) on total scores on the post-test, F(1, 68) = 20.73, p < .001, d = .73. Figure 2 presents a graph to illustrate this final result. This graph also shows that the control group (i.e., lecture only) demonstrated virtually no improvement in knowledge between the pre- test and post-test. In sum, this pattern of results suggests that exposure to the game, when paired with lectures, had moderate to large effects on students’ knowledge of the history and philosophy of science. Figure 2 Total Knowledge Scores for Experimental and Control Groups Satisfaction and Learning Reactions. Participants’ reactions to the game were somewhat mixed. For example, with respect to satisfaction with the game, 66% either agreed or strongly agreed that they liked the game. However, only 37% indicated they would be interested in playing it again. These mixed reactions may be due in part to the technical glitches that affected some participants’ gaming sessions. Because the game was a prototype, several bugs were still being worked out at the time of the study. These bugs occasionally led to the need to exit the game early and re-start from the beginning. Nevertheless, participants’ reactions about how much they believed they learned as a result of playing the game were very positive. For example, 90% of participants either agreed or strongly agreed with the statement, “I have a better understanding of how Galileo’s views were perceived by others during his time.” Table 1 presents the descriptive statistics showing how participants responded to each statement in the reaction questionnaire. 57% 58% 59% 70% 40% 45% 50% 55% 60% 65% 70% 75% 80% Total pre-test score Total post-test score Lecture only Lecture + game Percentage of questions correct Meeting Galileo: Testing the Effectiveness of an Immersive Video Game to Teach History and Philosophy of Science to Undergraduates Logan L. Watts – Peter Barker 142 Table 1 Descriptive Statistics for Reactions to Game Questions Mean SD Percentage agreeing Satisfaction I liked the game. 3.58 .95 66 It was easy to imagine myself as a character in the game. 3.39 1.05 55 The game was challenging. 3.31 .97 52 The game was fun. 3.36 .96 55 The game required my full focus to successfully complete it. 3.52 .95 65 The game was not long enough. 3.17 1.04 37 I would like to play the game again. 3.13 1.00 37 Learning I have a better understanding of Galileo’s life. 3.87 .89 79 I have a better understanding of how Galileo’s views of the cosmos differed from Aristotle’s views. 4.08 .77 89 I know more about Galileo’s scientific achievements. 3.84 .81 79 I have a better understanding of how Galileo’s views were perceived by others during his time. 4.06 .81 90 I have a better understanding of the times in which Galileo lived. 3.73 .93 70 Note. N = 71. Percentage agreeing includes “agree” and “strongly agree” responses. Discussion Before turning to the implications of these results, it is important to note some limitations of the study design. First, the sample was drawn from two undergraduate classes at one university taught by one instructor. As a result, we cannot say whether the results observed in this population and setting would also be observed among other populations and settings, such as students with different ages and backgrounds, or those taught by different instructors at other educational institutions. Second, the experimental group was exposed to both the video game prototype and two weeks of lectures. It is unclear if participants would have gained the same cognitive benefits from playing the game if they had not been Meeting Galileo: Testing the Effectiveness of an Immersive Video Game to Teach History and Philosophy of Science to Undergraduates Logan L. Watts – Peter Barker 143 simultaneously exposed to the lectures. For example, it may be that the lecture provided the framing needed to “make the most” of playing the game. Alternatively, the game may stand on its own as an effective educational intervention. Future research using different designs (e.g., experimental group that only plays the game) will help to address this limitation. Bearing these limitations in mind, the present study yielded a number of noteworthy innovations and contributions. This effort marked the first attempt we are aware of to study gamification in the context of the history and philosophy of science. This multidisciplinary approach was made possible by drawing on a collaborative team of historians, psychologists, entrepreneurs, and software engineers. Moreover, the present findings demonstrate evidence for the educational value of video games – in particular, first-person, immersive, role-playing games. Not only did students self-report that they learned from playing the game, but those in the experimental group also demonstrated objective improvements in knowledge compared with the control group. Finally, as shown in Figure 2, knowledge in the control group did not appear to improve from two weeks of lectures. This lack of improvement in the control group raises questions about the efficacy of relying on only traditional methods of instruction when educating students about complex topics such as the history and philosophy of science. Although these findings are intriguing, they raise a number of questions that might benefit from additional research. First, a clear next step is developing and testing additional modules, or extensions to the game. These modules might feature new tasks or objectives, new characters, or new settings. When exposed to multiple modules, do students evidence even greater increases in learning? Of course, if multiple modules are to be developed, a practical concern comes to the fore. High-fidelity, educational games such as the prototype tested in the present study are very expensive to develop. To manage these costs, partnerships between industry and academia may be a promising approach (Cannon-Bowers and Bowers, 2010). Indeed, this is the approach that was used in the present study. A second area of future research concerns getting a better understanding of the specific elements of educational games that enhance learning. Conducting research that isolates and manipulates specific features of educational games (e.g., simple vs. complex character development, short vs. long dialogue sequences, limited vs. ample opportunities for exploration, mastery vs. performance-oriented tasks/objectives) may be particularly useful for understanding the elements that facilitate or inhibit student learning. Finally, future research investigating the effectiveness of educational video games relative to other instructional methods (e.g. in- class games, debates, role-plays, discussions, team exercises, case studies, etc.) could shed valuable light on the specific types of instructional content that are best suited to being delivered by games. Conclusion As scholars and instructors of the history and philosophy of science, we carefully examine the past in order to inform the present and by extension, the future. Like other areas of academic inquiry, our instructional methods are rooted in rich traditions, many of which rely on passive approaches to student learning (e.g., lecture). In the present study, we showed that relying solely on traditional approaches to teaching the history and philosophy of science may be inadequate. Indeed, our findings demonstrate that gamification may hold great promise as an innovative avenue for complementing traditional teaching and learning methods – particularly when careful attention is paid to psychological learning principles such as promoting active learning, reducing cognitive load, and enhancing fidelity. We hope the present study serves to stimulate greater interest in gamification teaching, learning, and research. Meeting Galileo: Testing the Effectiveness of an Immersive Video Game to Teach History and Philosophy of Science to Undergraduates Logan L. Watts – Peter Barker 144 Acknowledgments: An earlier version of this work was presented at the University of Oklahoma’s annual Teaching Scholar’s Initiative in Norman, Oklahoma, USA. We would like to thank Michael Mumford, of the University of Oklahoma, and Brent Lollis and Michael Tigar of Parnassus LLC for their contributions to this effort. Figure 1 is reproduced courtesy of Parnassus LLC. References Andersen, Hanne, Peter Barker and Xiang Chen. 2006. The Cognitive Structure of Scientific Revolutions. Cambridge: Cambridge University Press. Ariew, Roger. 2006. The Sphere of Jacques du Chevreul: Astronomy at the University of Paris in the 1620s. In Universities and Science in the Early Modern Period, edited by M. Feingold and V. Navarro Brotons, 99-109. Archimedes volume 12. Dordrecht: Springer. Barker, Peter. 2007. Towards a Cognitive History of the Copernican Revolution. Organon 35: 61-72. Biagioli, Mario. 1993. Galileo Courtier: The Practice of Science in the Culture of Absolutism. Chicago: Chicago University Press. Kozlowski, Steve. W. J. and Bradford S. Bell. 2009. Toward a theory of learner-centered training design: An integrative framework of active learning. In Learning, training, and development in organizations, edited by Kozlowski and Salas, 283-320. New York: Routledge. Cannon-Bowers, Jan and Clint Bowers. 2009. Synthetic learning environments: On developing a science of simulation, games, and virtual worlds for training. In Learning, training, and development in organizations, edited by Kozlowski and Salas, 250-282. New York: Routledge. Curd, Martin, J. A. Cover and Christopher Pincock. 2012. Philosophy of Science: The Central Issues. New York: W.W. Norton & Co. Cohen, I. Bernard. 1985. The Birth of a New Physics. New York: W. W. Norton & Co. Cohen, Jacob. 1992. A power primer. Psychological Bulletin 112: 155-159. Copernicus, Nicholas. 1543. De revolutionibus orbium coelestium. Nuremberg: Petreius. Désilets, Patrice, Jade Raymond and Corey May. 2007. Assassin’s Creed. Montreal: Ubisoft. Deterding, Sebastian, Miguel Sicart, Lennart Nacke, Kenton O’Hara and Dan Dixon. 2011. Gamification: Using game-design elements in non-gaming contexts. In Proceedings of the 2011 Annual Conference on Human Factors in Computing Systems, edited by Desney S. Tan and Gerladine Fitzpatrick, 2425-2428. Vancouver, BC, Canada. Dicheva, Darina, Christo Dichev, Gennady Agre and Galia Angelova. 2015. Gamification in education: A systematic mapping study. Journal of Educational Technology & Society 18: 75-88. Finocchiaro, Maurice. 2008. The Essential Galileo. Indianapolis: Hackett. Freeman, Scott et al. 2014. Active learning increases student performance in science, engineering, and mathematics. Proceedings of the National Academy of Sciences 111: 8410-8415. Graney, Christopher M. Setting Aside All Authority: Giovanni Battista Riccioli and the Science Against Copernicus in the Age of Galileo. South Bend, IN: Notre Dame University Press, 2015. Heilbron, John. 2012. Galileo. Oxford: Oxford University Press. Kepler, Johannes, tr. Donahue, William. 2015. Astronomia Nova, new revised edition. Santa Fe, N.M.: Green Lion Press. Kozlowski, Steve W. J. and Richard P. DeShon. 2004. A psychological fidelity approach to simulation-based training: Theory, research and principles. In Scaled worlds: Meeting Galileo: Testing the Effectiveness of an Immersive Video Game to Teach History and Philosophy of Science to Undergraduates Logan L. Watts – Peter Barker 145 Development, Validation, and Applications, edited by Samuel G. Schiflett, 75-99. Burlington, VT: Ashgate. Kuhn. Tomas S. 1962. The Structure of Scientific Revolutions. Chicago: University of Chicago Press. 4th edition 2012. Mayer, Richard E. and Roxana Moreno. 2003. Nine ways to reduce cognitive load in multimedia learning. Educational Psychologist 38: 43-52. Meyers, Chet and Thomas B. Jones. 1993. Promoting Active Learning. Strategies for the College Classroom. San Francisco: Jossey-Bass. Osler, Margaret J. 2010. Reconfiguring the World: Nature, God and Human Understanding, from the Middle Ages to Early Modern Europe. Baltimore: The Johns Hopkins University Press. Popper, Karl R. 1959. The Logic of Scientific Discovery. New York: Basic Books. Popper, Karl R. 1963. Conjectures and Refutations: The Growth of Scientific Knowledge. London: Routledge and Kegan Paul. 4th edition 1981. Shadish, William R., Thomas D. Cook and Donald T. Campbell. 2002. Experimental and Quasi- Experimental Designs for Generalized Causal Inference. Boston: Houghton Mifflin. Sweller, John. 1988. Cognitive load during problem solving: Effects on learning. Cognitive Science. 12: 257-285.