IMCL Template SITUATED SIMULATIONS: A PROTOTYPED AUGMENTED REALITY GENRE FOR LEARNING ON THE IPHONE Situated Simulations: A Prototyped Augmented Reality Genre for Learning on the iPhone doi:10.3991/ijim.v3s1.963 G. Liestøl University of Oslo/Department of Media & Communication, Oslo, Norway Abstract — This paper reports on a project where digital genre design is conducted as a method to invent innovative expressive solutions and document structures based on the ongoing convergence of mobility, broadband, high resolution graphics and GPS-positioning technology. The prototyped genre – Situated Simulations – is a kind of augmented reality system based on the iPhone. This potential genre is aimed at users in contexts of both learning and leisure. In general, thetorical and aesthetic aspects of digital textuality tends to lag behind developments in hardware and software. Digital Genre Design as a constructive humanistic approach to digital media development attempts prevent this deficiency. Index Terms—Situated simulations, augmented reality, genre design, iPhone. I. INTRODUCTION In recent years we have seen the rapid convergence of powerful 3G smartphones, high capacity graphics, bright screens, multimodal textuality and GPS-positioning solutions. This merger of features and functions provides new platforms for invention and innovation in digital communication. How will it influence the way we communicate and exchange knowledge and information? In what way do we relate to this ongoing emergence of digital textuality? How may we influence the situation, and put the technology to work in education and learning? In digital media development, hardware and software continue to coexist and interact in intimate and complex relationships. The additional third level of the digital media hierarchy, however, tends to lag behind when it comes to evolution and change. That is the level of textual discourse where digital material markers are employed to communicate and exchange information, meaning and signification. We have elsewhere labelled this level meaningware (see Figure 1.) [1, 2]. Meaningware is the domain of digital discourse where the individual messages and texts reside. In traditional (analogue) media texts and messages (works) are organized or can be grouped as belonging to different, but related genres, be it the literary novel, the newspaper report, the western movie or the research paper. In some sectors of digital media we see similar and fully developed genre systems, most typically in computer games (1st person shooter, Adventure, MMOG etc.) and web publishing (blogs, podcasts, web news and so on). In other digital domains, however, the genre systems are to a large extent missing, for example in education and learning. The lack of genre development is particularly obvious in areas where the technology, both hardware and software, are rapidly converging and thus changing and destabilizing the conditions for sustainable textual innovation. In the INVENTIO-project (www.inventioproject.no) we aim to compensate for this lack of genre emergence by conducting active genre design, that is purposive experimentation with digital genre prototyping. This paper reports on the genre prototype we have so far called Situated Simulations. II. THE IDEA OF A SITUATED SIMULATION A situated simulation requires a broadband (3G) smartphone with substantial graphics capabilities, GPS- positioning features and an accelerometer. In this case we have been working with the iPhone, primarily because of its impressive performance, rich hardware features and Cocoa Touch, its flexible object-oriented programming environment. In a situated simulation there is approximate identity between a) the users visual perspective and perception of the real physical environment and b) the users visual perspective of a 3D graphics environment as it is represented on the screen. This relative congruity between the real and the virtual is obtained by letting the camera position and movement in the 3D environment be determined by the GPS-coordinates. As the user moves in real space the perspective inside the virtual space changes accordingly. Fig. 1. Digital media can be described as a three-level hierarchy of hardware, software and meaningware. The lower levels in this hierarchy constrains the level(s) above, serve as their environment and make them possible. In digital genre design innovation at the top layer of meaningware is the primary purpose. A situated simulation exists at this level, but is supported and constrained by the lower levels of hardware and software. http://dx.doi.org/10.3991/ijim.v3s1.963� SITUATED SIMULATIONS: A PROTOTYPED AUGMENTED REALITY GENRE FOR LEARNING ON THE IPHONE A situated simulation is closely related to mixed and augmented reality/virtuality [3]. While mixed reality, including augmented reality (AR) and augmented virtuality (AV) is constituted by different combinations of virtual and real representations along the reality–virtuality continuum (both levels occupying the same position on a screen), in a situated simulation there is a distinct (although minor) difference between the virtual perspective via the device and the real perspective of the user. Mixed reality systems often require relatively specialized and expensive hardware/software solutions, while situated simulations in principle can be developed for and implemented on off-the-shelf smartphones, such as Apple's iPhone (or other smartphones like those running the Android or Symbian OS. Mixed or augmented reality systems that resemble situated simulations have been described by Côté et al. in areas such as visualization of blueprints for use on construction sites [4], and Hedley has implemented and discussed visualizations of stratigraphic layers in geological field work [5]. III. THE BASIC FEATURES OF SITUATED SIMULATIONS The combination or juxtaposition of two perspectives (real and virtual) creates a double perspective which makes it possible to present on the site (on the screen) topics or subject matter which are otherwise (in reality) absent or invisible. Such subject matter may include topics from the past, the present or the future. In addition to the temporal differentiation, we can distinguish between three modes of the state of the simulation and the users relation to it: static, dynamic and/or participatory. We will now describe some of these various kinds of situated simulations and how they may interact and combine with different subject matters, topics and themes. IV. SOME EXAMPLES OF SITUATED SIMULATIONS In a situated simulation we are currently developing, the topic is the Oseberg Viking Ship grave mound in eastern Norway. We are attempting to show what the ship and its Fig. 2. Schematic illustration of a situated simulation. In this case the 3D-model is placed on a server connected to the terminal via a 3G network. In the implementations we have conducted so far it has been sufficient to download the whole application to the phone including media, textures and polygon models. Online access to a server solution becomes necessary when high resolution detail and several layers of textures are required. Fig. 3. On top: an early illustration suggesting the potential use of a situated simulation based on the Oseberg Viking Ship 3D model. In the background the reconstructed grave mound as it looks today. Below: A sequence of images showing an early prototype of the Oseberg Situated Simulation in actual use at the Oseberg site, October 2008. SITUATED SIMULATIONS: A PROTOTYPED AUGMENTED REALITY GENRE FOR LEARNING ON THE IPHONE rich content looked like the night before the grave mound was raised on top of it, some time in the year 834 AD. On the site where the mound is now reconstructed (the ship itself is on display in the Viking Ship Museum in Oslo) visitors can direct their iPhone towards the mound, access the application and 'see' the ship 'inside' the mound. As the visitor walks around the mound the ship will continually be viewed from different perspectives. If one ascends the mound one can look down upon the ship deck and study the many ancient artefacts gathered there. Audio narration accompanies the 3D objects as the user navigates through the real and virtual space. Another example under development is a situated simulation representing San Francisco's oldest building, Mission Dolores, constructed in 1791. In this situated simulation the user can navigate around and inside the original (now absent) buildings, accessing information about the historic landmark – on location. Both these examples are static objects reconstructed to the their original, past shapes and virtually placed in their former context. There are no substantial actions or events taking place in these simulations. The user may experience and navigate the environmetns in two modes: experiential or informative. In the experiential mode the purpose is to create the experience of being present there in time, or at least have access to this particular place in time. In the informative mode the purpose is not primarily to create an experience but to inform the user and convey knowledge about the subject matter. When the informative mode is turned on the user have access to audio narration and written information related to the place, the times and the included objects. Fig. 4. Prototype of the Mission Dolores Situated Simulation in use at different locations in the crossroads of Mission and 16th St in San Francisco, February 2009. Fig. 5. The Mission Dolores Situated Simulation in experiential (top) and informative (below) modes. In the informative mode the link layer is represented by the 'ballon labes', when activated these give access to audio narration and/or close up visual representations of smaller objects. Fig. 6. Early prototype of the San Francisco Earthquake 1906 Situated Simulation in actual use at the corner of Stockton and Ellis St, April 2009. Notice the cornice falling to the ground, indicating an event mode. SITUATED SIMULATIONS: A PROTOTYPED AUGMENTED REALITY GENRE FOR LEARNING ON THE IPHONE A third simulation under development has as its subject matter an historic event, namely the San Francisco Earthquake and Fire in 1906. This situated simulation also has a topic from the past, but the subject matter is not limited to a static object, but also includes a dynamic event (see figure 6.). Another event under consideration is the battle of Caesar and Pompey outside Pharsalus in 48 BC. Using the historic battle feature of computer game engines it is possible to simulate this historic event as a sequence of actions on the spot were it actually took place on the plains of Thessaly in Greece. A fourth simulation, of the same kind, which is also under planning, is the German attack on Norway in April 1940. The specific event is the sinking of the heavy cruiser Blücher during the Battle of the Drøbak Sound. Situated simulations, however, are not limited to the past. Future events may also be presented on site, based on predictions. Imagine a group of students carrying out project work on global warming. With a simulation engine they could explore how much the sea level would rise as average temperature increases. Then they could view the shoreline and see which areas would be flooded. Or they could move to the highlands and watch how much the timber line would expand under the same conditions. Such a simulation would combine a dynamic object with a future situation. V. SITUATED SIMULATIONS AS DOUBLE DESCRIPTIONS In traditional augmented reality systems the purpose is to mix the real and the virtual at the same level – on the screen and in the same visual field. In a situated simulation only the virtual 3D representations are on the screen, while the real stays real and is not mediated by digital technology. As mentioned this separation creates a double perspective, akin to what Bateson calls a double description. Bateson asks the following question: "What bonus or increment of knowing follows from combining information from two or more sources?" [6]. According to Bateson and his notion of double description, the combined perspective is of great importance, in fact it constitutes the foundation of his theory of information, and he provides several examples to illustrate this point: "Of all these examples, the simplest but the most profound is the fact that it takes at least two somethings to create a difference […] the difference which becomes information by making a difference." [6]. Another example is stereoscopic vision. Humans are equipped with two eyes, each of which directed at the same object creates a slightly different image on the retina. However, the combination of the two into one visual experience generates an increment: stereoscopic vision. This is a quality which cannot be traced back to any of the original sources, but emerges as a new quality in the combination of the two. We believe that situated simulations create a bonus or increments when the virtual and the real are combined. When using situated simulations with past, present and future objects or events, we expand our understanding of both time and space, how a place including its associated objects, events and actions, change over time and how it may be influenced by human conduct. VI. SOFTWARE SOLUTIONS The software for the situated simulation system is written in the Cocoa Touch programming toolset for the iPhone OS. Some of the 3D models are partly converted from an earlier hypermedia system published in 1996 which was based on boolean models and created in the application Real 3D. The models have then been converted to .obj files and transformed to polygon models in Cinema 4D. Other models were custom designed directly for the project. After the models have been individually prepared they were imported to the 3D game editor Unity iPhone 1.0.1. Unity can export directly to the iPhone, but at the moment the application does not support GPS-positioning (CoreLocations), consequently additional coding have been necessary in Xcode, Apples suite of developments tools ACKNOWLEDGMENT Special thanks to Tomas Stenarson of Memention AB for iPhone programming, collaboration and discussions during the design and development process. Thanks to Lars A. Gundersen and Anders S. Luhr for preparation the 3D models. Thanks also to colleagues in the INVENTIO project at the Department of Media and Communication and InterMedia, both at the University of Oslo for fruitful discussions and feedback, and in particular Andrew Morrison who has also copy edited this paper. REFERENCES [1] G. Liestøl, "Gameplay – From Synthesis to Analysis (and Vice Versa). Topics of construction and interpretation in digital media", in G. Liestøl, A. Morrison and T. Rasmussen (Eds.) Digital Media Revisited. Theoretical and Conceptual Innovations in Digital Domains, Cambridge, MA: The MIT Press, 2003, pp. 389-414. [2] G. Liestøl, "PowerPoint: beyond hardware and software" in A. Morrison (Ed.) Inside Multimodal Composition. Cresskill NJ: Hampton Press (Forthcoming, 2009). [3] P. Milgram & F. Kishino "A Taxonomy of Mixed Reality Visual Displays" in IEICE Transactions on Information Systems, Vol E77-D, No. 12, December 1994. [4] S. Côté et al. 'Experimenting with portable devices for onsite engineering model visualization in construction' in Proceedings from the 2nd International Workshop on Mobile Geospatial Augmented Reality, Laval University, Quebec, Canada 2008. [5] N. Hedley, 'Real-time reification: how mobile augmented reality may redefine our relationship with geographic space', in Proceedings from the 2nd International Workshop on Mobile Geospatial Augmented Reality, Laval University, Quebec, Canada. 2008 Fig. 7. The cases described above as they place themselves in the grid according to how they combine some basic features of situated simulations. The mentioned examples could be expanded to other areas as well. For example, the Battle of Pharsalus could be playable for the user (include participation), but then it would no longer be historically correct since the outcome might change according to the skills of the participating user. The 'present' column could include objects or events that are present in time, but not available to our perception, for example, objects hidden under a surface, like an archaeological artefact still in situ. SITUATED SIMULATIONS: A PROTOTYPED AUGMENTED REALITY GENRE FOR LEARNING ON THE IPHONE [6] G. Bateson, Mind and Nature. A Necessary Unity. New York: Bantam Books, 1988. AUTHOR Gunnar Liestøl is professor of digital media at the Dept. of Media and Communication, University of Oslo, NO 1093 Oslo, Norway This work was conducted within the INVENTIO-project, supported in part by the VERDIKT-programme under The Norwegian Research Council (NFR) and EngageLab at Intermedia. UiO. This article was modified from a presentation at the International Conference on Interactive Mobile and Computer Aided Learning (IMCL2009) in Amman, Jordan, April 2009. Submitted, June, 1st, 2009. Published as resubmitted by the authors on June, 10, 2009. http://www.inventioproject.no/� I. Introduction II. The Idea of a Situated Simulation III. The Basic Features of Situated Simulations IV. Some Examples of Situated Simulations V. Situated Simulations as Double descriptions VI. Software Solutions Acknowledgment References Author