Sergio Bernardes et al / GEOSI Vol 5 No 3 (2020) 352-363 352 Emerging Geospatial Technologies in Environmental Research, Education, and Outreach Sergio Bernardes, Marguerite Madden*, Ashurst Walker, Andrew Knight, Nicholas Neel, Akshay Mendki, Dhaval Bhanderi, Andrew Guest, Shannon Healy, Thomas Jordan Center for Geospatial Research, Department of Geography, University of Georgia, Athens, Georgia, GA 30602, United States of America *Corresponding Author: mmadden@uga.edu Received 1 November 2020/ Revised 16 December 2020 / Accepted 22 December 2020/ Published 30 December 2020 Abstract Drawing on the historical importance of visual interpretation for image understanding and knowledge discovery, emerging technologies in geovisualization are incorporated into research, education and outreach at the Center for Geospatial Research (CGR) in the Department of Geography at the University of Georgia (UGA), USA. This study aimed to develop the 3D Immersion and Geovisualization (3DIG) system consisting of uncrewed aerial systems (UAS) for data acquisition, augmented and virtual reality headsets and mobile devices, an augmented reality digital sandbox, and a video wall. We were working together integrated data products from the UAS imagery, including digital image mosaics and 3D models, and readily available gaming engine software to create augmented and virtual reality immersive visualizations. The use of 3DIG in research is demonstrated in a case study documenting the seasonal growth of vegetables in small gardens with a time series of 3D crop models generated from UAS imagery and Structure from Motion photogrammetry. Demonstrations of 3DIG in geography and geology courses, as well as public events, also indicate the benefits of emerging geospatial technologies for creating active learning environments and fostering participatory community engagement. Keywords: Environmental Education; Geovisualization; Augmented Reality; Virtual Reality; UAS, Photogrammetry 1. Introduction There is an old saying, “Seeing is believing.” Regarding remote sensing and image processing for environmental research, education and outreach, the saying should be adapted to read, “Seeing is believing and understanding.” Long before there were computers available to analyze and classify remotely sensed imagery acquired from aerial or satellite platforms, men and women bent over light tables and examined hardcopy aerial photographs with the aid of a stereoscope and visually interpreted the features they viewed in three dimensions. Anson (1968) described that “Photographic interpretation originated when the images recorded on sensitized GEOSFERA INDONESIA p-ISSN 2598-9723, e-ISSN 2614-8528 Vol.5 No. 3 (2020), 352-363, December, 2020 https://jurnal.unej.ac.id/index.php/GEOSI DOI : 10.19184/geosi.v5i3.20719 Accredited by the Ministry of Research , Technology , and Higher Education of the Republic of Indonesia, No. 30/E/KPT/2019. GEOSFERA INDONESIA p-ISSN 2598-9723, e-ISSN 2614-8528 Vol.5 No. 3 (2020), 301-317, December, 2020 HTTPS://JURNAL.UNEJ.AC.ID/INDEX.PHP/GEOSI DOI : Accredited by the Ministry of Research , Technology , and Higher Education of the Republic of Indonesia, NO. 30/E/KPT/2019. mailto:mmadden@uga.edu https://jurnal.unej.ac.id/index.php/GEOSI https://drive.google.com/file/d/1rSnVAs6cuHOwHl5BJ87CL2L6K5DQz7s6/view https://jurnal.unej.ac.id/index.php/GEOSI https://drive.google.com/file/d/1rSnVAs6cuHOwHl5BJ87CL2L6K5DQz7s6/view Sergio Bernardes et al / GEOSI Vol 5 No 3 (2020) 352-363 353 material by radian energy were first recognized during photographic processing. The recognition process became valid for the individual who recalled or recognized a familiar object, person, or scene which created a response in the mind of the interpreter.” With the advent of computer technology, the conversion of hardcopy to digital imagery and the evolution of analog to digital image processing, researchersexplored the use ofautomated methods of image understanding while maintaining the importance of visual image assessment. The application of computer- based systems by a photo interpreter is to enhance the digitized photographs to facilitate human interpretation. Although great strides have been made in applying digital image processing to automatically classify various surface features, these techniques still fall far short of the human interpreter (Colwell, 1997). By the 2000s, digital image classification was well-established and great advances were made in refining image classification algorithms, multi-sensor image data fusion and mainstream use of dedicated image processing software (Campbell, 2002; Lillesand et al., 2003; Jensen, 2007; Sabins 2007). The need for visual understanding of image features remains and important component of even the most sophisticated computer algorithms. Early image classifications methods (e.g., supervised image classification) relied on training sets identified by humans and of consisting o f pixels identified in an image as representative of ground features to be classified (Lillesand et al., 2003). The Elements of Image Interpretation, (i.e., tone, shape, size, pattern, association, shadow, texture and resolution) published in a seminal publication by Olson (1960), integrated different aspects of image understanding into a single framework that is still used today as the basis of many image processing algorithms including pixel-based classification, object-based image analysis, automated feature extraction, machine learning and deep learning. The subject of human factors in image interpretation, cognition and remote sensing is discussed ascentral to mixed methods research, geospatial intelligence, perceptual processes and the capture of expert knowledge towards the automation of the interpretation process in White et al. (2018). Current challenges include the interpretation and integration of multiple data types, especially dynamic data, while taking advantage of emerging technologies in geovisualization (Hoffman, 2018). The potential of using multimedia in cartography and visualizing environments using virtual reality was recognized by early geospatial visionaries in the 1990s and early 2000s (Taylor, 1994; Cartwright, 1994; Slocum et al., 2001). Cartwright (1994) noted cartographers have always designed and produced maps using numerous media elements and tools including Sergio Bernardes et al / GEOSI Vol 5 No 3 (2020) 352-363 354 computer graphics, photogrammetry, statistical analyses and geographic information systems (GIS). “The ability to integrate picture, sound and movement gives the ability to assemble almost any combination of presentation package required for a particular spatial data presentation, including graphic solutions using space-time data” (Cartwright, 1994). Discussions on the use of geographic multimedia and hypermedia recognized the integration of new methods of digital data integration, display and interaction with users within GIS and early adoption of hypertext for linking internet content within digital cartographic designs (Stefanakis et al., 2006).Virtual realities were also being promoted as gaming technologies evolved and cartographers recognized the advantages of virtual interfaces to geographic content for research and graphic communication (Cartwright, 2006; Bainbridge, 2007). The visualization of changing vegetation patterns related to terrain characteristics and disturbances such as wildfires, insect infestation and human impacts also were explored as three-dimensional, time-series geovisualizations and virtual realities (Madden et al., 2006 & 2009). This study aimed to developed the 3D Immersion and Geovisualization (3DIG) system consisting of uncrewed aerial systems (UAS). 2. Methods Researchers at the Department of Geography’s Center for Geospatial Research at the University of Georgia have created a 3D Immersion and Geovisualization (3DIG) system aimed to bring multiple technological components for data acquisition, extended reality or XR (i.e., virtual, augmented and mixed reality) and geovisualization for Earth Science research, education and outreach. The motivation for 3DIG began with the recognition that students are increasingly visual learners. They enter universities with experience using 3D immersive video games and expectations regarding the use of modern technologies for learning. Recognizing the need for geography-ready immersive technologies that support visual and experiential learning led to the design of the 3DIG system. We hypothesized bringing 3D graphics and visualization technologies to traditional lectures and instructional materials would promote student learning and understanding of geographic concepts that are often difficult to grasp. Sergio Bernardes et al / GEOSI Vol 5 No 3 (2020) 352-363 355 The components of 3DIG include uncrewed aerial systems (UAS), virtual and augmented reality headsets and mobile devices, an augmented reality sandbox and a video wall (Bernardes et al., 2018, Figure 1). A DJI Phantom 4 Pro quadcopter and a DJI Matrice 600 Pro hexacopter are used to acquire high spatial and temporal resolution images and videos. The Phantom 4 Pro UAS has features such as attitude control, automatic take-off and landing, collision avoidance and video game-like flight controls that facilitate flying by students and young researchers with no previous exposure to UAS. The Matrice 600 Pro hexacopter allows a heavier payload with multiple sensors such as the MicaSense 5-band multispectral, Velodyne LiDAR and FLIR thermal sensors. The CGR personnel with Remote Pilot Certification provide flight training, demonstrations and instruction for students to obtain their own certification required for conducting research and outreach. Virtual and augmented reality goggles (e.g., Oculus Rift, Quest and Microsoft HoloLens) are demonstrated in classes and used by students in many courses in the geosciences and other fields to expose students to emerging technologies and provide immersive experiences in virtual and augmented landscapes, international cities and a variety of landforms (Bernardes et al., 2020, Figure 2). Figure 1. The 3D Immersion and Geovisualization (3DIG) system incorporates uncrewed aerial system (UAS) data acquisition, virtual and augmented reality and video wall for use in research, education and outreach Sergio Bernardes et al / GEOSI Vol 5 No 3 (2020) 352-363 356 The augmented reality sandbox was designed and built in-house by UGA-CGR students and faculty. The system consists of two Xbox Kinect sensors configured to interact with the topography of the sand contained within a wooden box supported on a frame at waist height. A downward-pointing projector is connected to a laptop such that the display of geospatial data and model output can be projected onto the surface of the sand. Intended to be interactive and participatory, users gather around the sandbox and are encouraged to move the sand with their hands to build “mountains and valleys” that can even recreate the topography of real locations. See Petrasova et al. (2015), for a complete discussion of tangible modeling and software/hardware requirements for constructing an augmented reality sandbox. The UGA-CGR sandbox was designed to be transportable with casters and a telescopic structure such that the frame can be lowered to fit through doorways and wheeled to classrooms, laboratories, other buildings or even public spaces. In this way, exposure to the augmented reality technology can be maximized and to-date the sandbox and mobile goggles have been used in 55 courses and 13 public events for over 6,000 individual interactions. Figure 2. Virtual reality headsets allow students access to remote environments, immersive experiences in a variety of landscapes and exposure to cultural resources Sergio Bernardes et al / GEOSI Vol 5 No 3 (2020) 352-363 357 Figure 3. (a) Augmented reality sandboxdesigned and built in-house at the University of Georgia’s Geography Department Center for Geospatial Research for (b) participatory tangible experiences and geovisualization of geographic concepts The final component of the 3DIG system is a video wall consisting of four NEC 46-inch high-performance commercial grade video displays that are daisy chained and controlled by a single Windows-based computer using TileMatrix technology. Visual integration of the four individuals displays is facilitated by reducing the bezel width (i.e., the outside edge of the display) to 1.9 mm (minimum) or 3.8 mm (maximum) per display. The use of serviceable wall mounts and inter-display wall calibration also provides an improved visualization experience. When combined with augmented reality goggles, multiple students in the class are able to see what the person wearing the Microsoft HoloLens goggles is experiencing. Although not in three- dimensions, the two-dimensional view displayed on the video wall allows students to see the augmented reality that is otherwise invisible to everyone except the person wearing the headset (Figure 4). (a) (b) Sergio Bernardes et al / GEOSI Vol 5 No 3 (2020) 352-363 358 Figure 4. The video wall combined with augmented reality goggles allows multiple students in the class to see what the person wearing the Microsoft HoloLens goggles is experiencing 3. Results and Discussion A case study applying emerging technologies used by 3DIG for research, education and outreach was performed, by using UA Sand photogrammetry for small farm agricultural monitoring, research and outreach to local communities. Acknowledging small farms and community gardens are important in local culture, economies and food supplies, integrated geospatial techniques of UAS image acquisition, Structure from Motion (SfM) photogrammetry, field-based measurements of plants and 3D crop models were developed to assist farmers and community gardeners who may not normally have access to advanced geospatial technologies. A UAS was used to fly low-altitude imagery of vegetable gardens throughout the growing season. Digital image mosaics and 3D crop models were produced with SfM photogrammetric software (Metashape by Agisoft) that depict the growth forms of individual plants and can be measured to quantify changes in crop biomass over time (Figure 5a). Gaming engine software such as Unity or Unreal Engine can then be used to convert 3D models to augmented reality displays to visualize virtual gardens in a classroom or public setting (Figure 5b). Sergio Bernardes et al / GEOSI Vol 5 No 3 (2020) 352-363 359 a) b) Figure 5. (a) 3D crop models produced by photogrammetry and UAS images of small gardens can be converted to augmented reality displays (b) for public viewing Emerging developments in virtual, augmented and mixed reality and increased accessibility to the hardware and software required to access their implementations have led to renewed interest and new possibilities for uses in research, education and outreach. Çöltekin et (a) (b) Sergio Bernardes et al / GEOSI Vol 5 No 3 (2020) 352-363 360 al. (2020) provide a thorough review of research challenges and future directions in XR within the spatial sciences. Noted is the need for a consensus of definitions for fundamental terms of extended reality because there is ambiguous use of virtual, augmented and mixed reality (Çöltekin et al., 2020). This is especially important when these advanced technologies are being incorporated into educational materials, research manuscripts and outreach demonstrations in order to minimize confusion and harmonize the discussion of advantages and limitations. Milgram & Kishino (1994) presented a continuum of realities ranging from the real environment to the virtual environment and predicted future experiences will be increasingly mixed virtual and real views of the world around us. As of 2020, it is estimated that over 6,000 students, ranging from pre-Kindergarten (i.e., less than five years old) to university levels of education, have been exposed to the 3DIG system in 55 courses or course sessions in a variety of disciplines including physical geography, weather and climate, geographic information systems, geology, Earth sciences, environmental design and psychology (Bernardes et al., 2020). Numerous undergraduate and graduate students have been involved in the UAS data acquisition, image processing and the development of content for extended reality demonstrations (Bernardes et al., 2018). Current research is being conducted to assess the effectiveness of the use of extended reality to enhance the understanding of complex geospatial concepts and understanding of the physical environment. Indeed, the development of the 3DIG has led to the creation of the Disruptive Geospatial Technologies Laboratory within the UGA Center for Geospatial Research that further explores cutting-edge technologies for data acquisition, manipulation, teaching, learning and discovery. These technologies have been developed, implemented or modified by 23 students at UGA, through a variety of on-the-job training and mentorship experiences. See DiGTL (2020) for additional information and updates on projects, research and teaching tools addressing critical issues in human and environment relationships. The 3DIG system in research and outreach components have demonstrated both the advantages and limitations of advanced geovisualization techniques. Trade-offs of enhanced and multi-dimensional visualization of our environment must be weighed against the cost of extended reality devices, the need for advanced skills in computer programming and additional training of instructors. Sergio Bernardes et al / GEOSI Vol 5 No 3 (2020) 352-363 361 4. Conclusion In conclusion, the 3DIG system developed at the University of Georgia combines geospatial technologies with photogrammetry and emerging visualization techniques to spark excitement, foster engagement and create active learning environments. Emerging technologies provide opportunities for research, education and outreach that are rooted in the visual interpretation of aerial imagery and combined with technologies for state-of-the-art training, enhanced geovisualization and participatory community involvement. Conflict of Interest The authors declare that there is no conflict of interest with any financial, personal, or other relationships with other people or organizations related to the material discussed in the article. Acknowledgements The assistance of students at the University of Georgia who have volunteered their time during several phases of the project is gratefully appreciated. This research was supported by the UGA Center for Teaching and Learning, the UGA Center for Research and Education at Wormsloe and the Wormsloe Foundation. References Anson, A. (1968). Photographic interpretation from color aerial photographs. Manual of color aerial photography. First edition. , Virginia : Am. Soc. Photogrammetry, Falls Church. Bainbridge, W. S. (2007). The scientific research potential of virtual worlds. science, 317(5837), 472-476. Bernardes, S., Howard, A., Mendki, A., Walker, A., Bhanderi, D., Le, L., ... & Rukangu, A. (2019). Innovative Technologies in Teaching and Learning: Incorporating Recent Developments in Virtual and Augmented Reality into Active Learning at the University of Georgia. AGUFM, 2019, ED11B-0873. Bernardes, S., Madden, M., Knight, A., Neel, N., Morgan, N., Cameron, K., & Knox, J. (2018). A Multi-Component System For Data Acquisition And Visualization in The Geosciences Based on UAVS, Augmented And Virtual Reality. ISPRS - International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XLII–4, 45–49. https://doi.org/10.5194/isprs-archives-XLII-4-45-2018. Sergio Bernardes et al / GEOSI Vol 5 No 3 (2020) 352-363 362 Campbell, J. B., & Wynne, R. H. (2011). Introduction to remote sensing. New York : Guilford Press. Çöltekin, A., Lochhead, I., Madden, M., Christophe, S., Devaux, A., Pettit, C., … Hedley, N. (2020). Extended reality in spatial sciences: A review of research challenges and future directions. ISPRS International Journal of Geo-Information, 9(7), 439. https://doi.org/10.3390/ijgi9070439. Colwell, R. (1997). History and place of photographic interpretation. Chapter 1, In, Philipson, Warren R. (Ed-in-Chief), Manual of Photographic Interpretation, Second Edition, pp. 3- 47. Maryland: American Society for Photogrammetry and Remote Sensing. DiGTL. (2020). Disruptive Geospatial Technologies Laboratory (DiGTL), Center for Geospatial Research, Department of Geography, University of Georgia, Athens, Georgia, http://www.cgr.uga.edu/labs/digtl/ and http://www.cgr.uga.edu/labs/digtl/interactions.html, Last Accessed, 14 November 2020. Hoffman, R. R. (2018). Cognitive and Perceptual Processes in Remote Sensing Image Interpretation. In Remote Sensing and Cognition (pp. 15-32). Florida : CRC Press. Jensen, J.R. (2007). Remote Sensing of the Environment: An Earth Resource Perspective. Second Edition, New Jersey : Pearson Prentice Hall, Upper Saddle River. Lillesand, T., Kiefer, R. W., & Chipman, J. (2015). Remote sensing and image interpretation. New York : John Wiley & Sons. MacEachren, A. M., & Taylor, D. F. (Eds.). (2013). Visualization in modern cartography. Oxford: Elsevier. Madden, M., Jordan, T., & Dolezal, J. (2006). Geovisualization of Vegetation Patterns in National Parks of the Southeastern United States. In Geographic Hypermedia (pp. 329- 345). , Berlin, Heidelberg : Springer. Madden, M., Jordan, T., Kim, M., Allen, H., & Xu, B. (2009). Integrating remote sensing and GIS: From overlays to GEOBIA and geovisualization. Manual of Geographic Information Systems, 701-720, Maryland : Bethesda. Milgram, P., & Kishino, F. (1994). A taxonomy of mixed reality visual displays. IEICE Transactions on Information and Systems, 77(12), 1321-1329. Olson, C. E. (1960). Elements of photographic interpretation common to several sensors. Photogrammetric Engineering, 26(4), 651-656. Petrasova, A., Harmon, B., Petras, V., & Mitasova, H. (2015). Tangible modeling with open source GIS (p. 135). Cham: Springer International Publishing. https://doi.org/10.3390/ijgi9070439 http://www.cgr.uga.edu/labs/digtl/ http://www.cgr.uga.edu/labs/digtl/interactions.html Sergio Bernardes et al / GEOSI Vol 5 No 3 (2020) 352-363 363 Sabins, F. F. (2007). Remote sensing: principles and applications. , Illinois: Waveland Press. Slocum, T. A., Blok, C., Jiang, B., Koussoulakou, A., Montello, D. R., Fuhrmann, S., & Hedley, N. R. (2001). Cognitive and usability issues in geovisualization. Cartography and Geographic Information Science, 28(1), 61–75. https://doi.org/10.1559/152304001782173998. Stefanakis, E., & Peterson, M. P. (2006). Geographic hypermedia. In Geographic Hypermedia (pp. 1-21). Berlin : Springer. White, R. A., Coltekin, A., & Hoffman, R. R. (Eds.). (2018). Remote sensing and cognition: human factors in image interpretation. Florida : CRC Press. https://doi.org/10.1559/152304001782173998