International Journal of Interactive Mobile Technologies – ISSN: 1865-7923 – Vol. 12, No. 8, 2018 Paper—Evaluating Smartphones for Infrastructure Work Order Management Evaluating Smartphones for Infrastructure Work Order Management https://doi.org/10.3991/ijim.v12i8.9441 Victor L. Guinn, Brent T. Langhals(*), John J. Elshaw Air Force Institute of Technology, Wright-Patterson AFB, United States brent.langhals@afit.edu Abstract—Infrastructure managers require timely and accurate state infor- mation to diagnose, prioritize, and repair the substantial infrastructure assets sup- porting modern society. Challenges in obtaining sufficient information can often be attributed to inadequate data collection procedures (phone calls, paper reports, etc.) or a general lack of knowledge or ability on the part of the reporting indi- vidual to accurately convey what is actually wrong with the facility. Fortunately, modern smart-phone technology offers the potential to improve maintenance work requests by providing better geolocation and problem description accuracy. An experiment simulating real-world maintenance requests was conducted comparing smart-phones with traditional verbal work order request systems. Usefulness and description accuracy ratios revealed smartphone systems gener- ated more useful information regardless of submitter background or experience. However, interestingly the smart-phone applications did not improve asset geo- location and actually negatively impacted the ability of maintenance personnel to accurately relocate the asset needing service. Given the ubiquitous nature of smartphone technology, the potential exists to turn any citizen into an infrastruc- ture sensor. This study takes a step toward understanding the benefits, as well as the limitations, of the smart-phone based work order submission systems. Keywords—mobile phone, application, work-order system, infrastructure 1 Introduction Given various infrastructure assets seemingly fail on a daily basis, rapid identifica- tion and immediate reaction is the critical antidote toward guaranteeing sound facilities for continued operations [1]. To facilitate timely response, more efficient methods must be developed to quickly submit, diagnose, and respond to problems as they arise [2]. The advent of smart-phone technology, accessible to a majority of the population, has led to the development of smart-phone based work order management applications de- signed to take advantage of inherent advantages of smart phones [3]. The assumption is anyone carrying a smartphone can become an infrastructure sensor and report asset problems regardless of maintenance experience. Unfortunately, little research has di- rectly explored the effectiveness of these applications in any rigorous way. Therefore, the following study proposes to answer two crucial questions about the technology: 52 http://www.i-jim.org Paper—Evaluating Smartphones for Infrastructure Work Order Management 1. Do smartphone-based mobile work order submission applications allow non-mainte- nance oriented individuals to transmit more useful information than traditional sys- tems? 2. Will an increase in useful information enhance maintenance effectiveness in terms of problem location and description? 2 Background Two key requirements for generating work orders include the ability to locate the problem (geolocation) and description accuracy. Evaluation of these two areas starts with customer submission and ends with how useful maintenance personnel find the information for diagnosing the type and extent of the repair. Both are discussed in the following sections. Geolocation, essential for timely maintenance action, is the process of not only ac- curately reporting a specific geographic location but also relocating the asset at some point in the future. Without it maintenance craftsmen could never respond to repair the problem. Traditional work order systems typically consist of verbal or text reports and are frequently reduced to short text strings by a controller when inputting into a resource management system [4]. With little built-in capability to capture location information, it is not uncommon for maintenance personnel to experience subsequent geolocation issues, especially when the facility infrastructure is placed in a linear, or grid pattern. While it is well known that individuals have difficulty geographically describing just one item in the pattern, it becomes even more difficult when their perceived location is different that their actual location [5] [6] [7]. It is believed geolocation problems may be mitigated using smart-phones enabled with GPS. For some applications, quick and accurate geolocations were accomplished using GPS with minimal knowledge or effort on the part of the individual trying to determine a location [8] [9]. Unfortunately, under certain conditions, GPS effective- ness may be limited, such as indoors where GPS signals may be interfered with or lack sufficient accuracy to reveal which hallway or floor a problem exists [10]. Therefore, one objective of this study is to determine if GPS enabled smart-phones inherently pro- vide better geolocation information than traditional work order submission systems. After geolocation, object recognition and ability to describe accurately becomes the next necessary component to successful reporting. Being able to visually recognize an object and accurately describe it is the result of several factors. The first factor is an individual’s past experiences with the object. Past experiences build visual representa- tions of objects, which a person remembers when they see the object again. Represen- tations of the object not only consist of a remembered image, they also carry additional information about the object, such as function and name. If the individual cannot de- termine the object as a whole, they resort to assigning the object into rough classes of objects they do remember. Utilizing representations based off past experiences enables individuals to visually recognize the object, and mentally prepare a description to be transmitted [11] [12]. Without experience with an object, individuals can only relay iJIM ‒ Vol. 12, No. 8, 2018 53 Paper—Evaluating Smartphones for Infrastructure Work Order Management remembered representations that resemble the object and can lead to possible work or- der inaccuracies. A simple example of this would be calling an “A19 incandescent soft white interior frosted 60 Watt 120VAC medium base bulb” with the first class to occur to the individual, such as “light bulb.” Clearly identifying the problem with the former description would allow immediate resolution as the maintenance personnel would ar- rive with the correct light bulb on the first visit. The latter, however, requires additional information to remedy the problem, possibly in the form of multiple visits or at a min- imum time spent researching the lighting used in the reported area. Options for mitigating a lack of object expertise are limited regarding facility in- frastructure. One way is to educate all individuals to the visual recognition level of an expert, which isn’t practical. Another, more preferable way, is to enable experts to be able to see what the individual is seeing and thus perform the visual recognition them- selves without having to be there. An audio or text only media channel does not provide this capability. However, with the expanded communications inherent in modern smartphones, transmission of high quality visual information is possible. The addition of expanded channels for communication, especially visual, has been demonstrated to improve the effectiveness of communication [13]. Media richness theory describes a communications mediums ability to reproduce the information sent across it [14]. A visual channel which allows an expert to see exactly what the customer sees would be highly valuable to maintenance personnel and consistent with media richness theory would provide a much richer communication medium as compared to text or voice alone. An underlying principle of media richness theory is the concept that information is only useful as long as it reduces uncertainty for the receiver [15] [14] [16]. As receivers of the reported information, maintenance craftsmen are the most appropriate evaluators regarding usefulness of information submitted, however usefulness for reducing uncer- tainty is largely dependent on what organizational language the information arrives in. Organizations and groups of specialties have their own unique organizational languages as a result of constant internal communication and natural adaptations for efficiency [17] [18]. Information that does not conform to unique organizational languages may have little impact on reducing uncertainty in the receiver. Infrastructure customers frequently have a tough time overcoming organizational language barriers when communicating with maintenance personnel. Due to minimal facility experience, customers often are unable to communicate in the unique ver- bal/text-based language of maintenance personnel. With limited communication chan- nels afforded by typical work order request systems, difficulties in communicating the exact nature of the problem result in frustrations customers and maintainers alike. Smartphones may provide a key to mitigating the language barrier and thus increasing problem description accuracy by providing a mechanism which allows both sides to understand each other’s language. Utilizing the visual channel enabled by smartphones to send live images, customers may be able to transmit all of the information needed to maintenance craftsmen without knowing any additional information about the asset in question. Upon receiving the transmission, maintainers visually recognize the asset and reduce uncertainty about the problem. 54 http://www.i-jim.org Paper—Evaluating Smartphones for Infrastructure Work Order Management To investigate whether or not smartphone technology is capable of overcoming ge- olocation, object recognition, and the lack of maintenance experience and subsequent language barrier obstacles, an experiment was conducted in order to test the following three hypotheses. • Hypothesis 1 (H1): Smartphone-based mobile work order submission applications will provide more useful information than traditional verbal based submission sys- tems. • Hypothesis 2 (H2): Smartphone-based mobile work order submission systems will allow non-maintenance personnel to submit information of equal description accu- racy compared to individuals with a maintenance background. • Hypothesis 3 (H3): Smart-phone based systems will provide more accurate geolo- cation than traditional verbal based systems. If smartphone based work order submission systems do support the hypotheses pre- sented, the immediate effect of reducing communication errors, improving maintenance diagnoses, and reducing travel could be achieved, which would directly translate into saving time and money for the organization. 3 Methodology To evaluate the effectiveness of a smartphone-based work order submission system, the authors conducted an experiment simulating common real-world maintenance sce- narios. A smartphone application was created, modeled on commercial versions avail- able today, to compare with an existing verbal capture system currently in use. Both tools gathered the three main items required for maintenance work orders: point of con- tact, location, and description of the problem. The primary difference of the smartphone-based application was that it allowed subjects to take a photo and simulta- neously capture the phones last known GPS location. The traditional verbal work order system used a Google Voice voicemail account which prompted users to answer ques- tions such as: what is your location and what is the infrastructure problem that you are trying to submit. The automated system conveniently standardized all questions for each subject and the authors were able to record every word uttered for future transcrip- tion and analysis. The authors recruited maintenance craftsmen to serve as experts for the experiment. They were responsible for the primary task of generating a list of the key data required to diagnose and repair each maintenance scenario. After maintenance craftsmen deter- mined what data was considered useful, work order requests placed by subjects were scored to determine the usefulness of the information for a work order (overall useful- ness ratio) and accuracy of the description (description usefulness ratio). The authors then performed statistical analysis to evaluate performance differences between the two systems. iJIM ‒ Vol. 12, No. 8, 2018 55 Paper—Evaluating Smartphones for Infrastructure Work Order Management 3.1 Subjects Subjects (n = 40) were comprised of students, faculty, and family members, and were randomly assigned to a submission method (verbal or smartphone). Subjects com- pleted a pre-experiment survey to determine facility maintenance experience, job back- ground, education level, age, and gender. The backgrounds of all participants were examined to ensure there was a representative mix of organizational languages compa- rable to a typical maintenance organization. This distribution is provided in Figure 1. Fig. 1. Subject Job Skill Distribution Civil engineers (with maintenance backgrounds) made up 50% of the subject pool while the balance of subjects came from the space, financial, instructor, logistics, con- tracting and computer science backgrounds. The corresponding diversity of organiza- tional language allowed splitting the subjects into two groups (maintenance vs. non- maintenance) for comparing smartphone work order submission vs. traditional verbal systems. 4 Analysis and Results To test the study’s hypotheses, the authors calculated description ratios and an over- all usefullness ratio. The description ratio provides a normalized quantitative measure of how accurately subjects described the maintenance issue. The score was calculated by comparing an expert generated list of key description elements for each maintenance scenario with the subject’s actual description of the maintenance problem. Every data point submitted that matched an expert identified key data value was scored as a pass (value = 1); if not then a failure was assigned (value = 0). 𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷 𝑅𝑅𝑅𝑅𝐷𝐷𝐷𝐷𝐷𝐷 = /0123456478 92730 (9;<=026) ?@A /0123456478 92730 (9?B) (1) 56 http://www.i-jim.org Paper—Evaluating Smartphones for Infrastructure Work Order Management Once the description ratio was computed, a second ratio was calculated called the usefulness ratio. The usefulness ratio was used to incorporate the concept of geoloca- tion in the study. As previously discussed, if a maintenance problem could not be lo- cated, any description provided would not be very useful. Geolocation data was converted to binary pass (value = 1)/fail (value = 0) criteria. Geolocation was scored as fail if the location data provided would not allow the expert to accurately locate the broken infrastructure item. 𝑈𝑈𝐷𝐷𝐷𝐷𝑈𝑈𝑈𝑈𝑈𝑈𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷 𝑅𝑅𝑅𝑅𝐷𝐷𝐷𝐷𝐷𝐷 = [H07I72@6478 92730J/0123456478 K@647] M (2) The following example illustrates how a subject’s response was captured and scored for a flickering light bulb using the mobile work order submission process. Figure 2 depicts data captured for the light bulb example. Fig. 2. Work Order Submission Data Collection Once the work order was submitted, the expert generated list of required data ele- ments was used to score the submission. If either the text entered by the subject or the image captured by the smartphone provided one of the expert required data elements, a score of one was assigned. If a key data element was not received through the submis- sion, a score of 0 was assigned. Figure 3 depicts the example scoring. Fig. 3. Scoring Expert Key Data Elements The Description Accuracy Ratio for this example was a result of the subject submit- ting six of the nine key data elements required by the experts. For Geolocation score, a “1” was recorded if either the GPS coordinates accurately returned the expert to the maintenance location, or the subject provided enough detail to rapidly locate the maintenance problem (i.e. building and room). If the problem could not be immediately iJIM ‒ Vol. 12, No. 8, 2018 57 Paper—Evaluating Smartphones for Infrastructure Work Order Management relocated with the given data, a score of 0 was entered. In this example, the GPS loca- tion and the subject description proved insufficient to allow the expert to relocate the flickering light and therefore was scored as a 0. 4.1 Analysis of Usefulness Ratio The authors conducted an independent samples t-test to test the hypothesis that the smartphone work order submission process (n =20) provided more useful information than the traditional verbal system (n = 20). Additionally the usefulness ratio was cal- culated by standardizing the Description Accuracy Ratio and Geolocation scores (to avoid overly weighting one of the scores), summing them, and then dividing by two. The test was significant (t(1148) = 7.42, p < .01), supporting research hypothesis 1. Subjects in the smartphone submission condition (M = .106, SD = .49) on average pro- vided a higher usefulness ratio than subjects in the traditional verbal system (M = -.108, SD = .48). Figure 4 shows the distributions of the two groups. Fig. 4. Usefulness Ration Analysis This test supports the assertion that the smartphone-based work order submission application delivers information that is more useful; however, the question regarding its ability to mitigate organizational language barriers and communicate at the level of an expert regardless of experience remained. 4.2 Analysis of Description Accuracy Ratio by Subject Background The authors conducted a second independent t-test to evaluate the hypothesis that a smartphone-based mobile work order submission system will allow non-maintenance personnel to submit information equal in accuracy to individuals with a maintenance 58 http://www.i-jim.org Paper—Evaluating Smartphones for Infrastructure Work Order Management background. To conduct this test, the subjects were grouped first by experience and then by description accuracy for each of the two submission methods. The first test evaluated description accuracy for the traditional verbal submission system among sub- jects with a civil engineering maintenance background and subjects from the other backgrounds. For the traditional work order submission system, the t-test indicated a significant difference (t(572) = 2.77, P < .05) between the two groups. The description ratio mean for civil engineering organization was 0.42 (sd = 0.25), and the mean for individuals outside of civil engineering organizations was 0.36 (sd = 0.26). Figure 5 shows the distributions of the two groups. Fig. 5. Description Accuracy Using Traditional Submission Method An analysis of the second subset was then accomplished to determine if smartphone- based mobile work order submission technology allowed the individuals without a civil engineering background to describe the maintenance problem as well as individuals with a civil engineering background. The t-test indicated no significant difference be- tween the two methods (t(577) = 1.51, P > 0.05), thus indicating that when utilizing the smartphone-based mobile work order submission technology there is an effect of miti- gating organizational language barriers. The description ratio mean for individuals with a civil engineering background was 0.71 (sd = 0.23), and the mean for individuals out- side of the civil engineering organization was 0.68 (sd = 0.25), thus support exists for hypothesis 2. Figure 6 shows the distribution of the two groups. iJIM ‒ Vol. 12, No. 8, 2018 59 Paper—Evaluating Smartphones for Infrastructure Work Order Management Fig. 6. Description Accuracy Using Mobile Submission Method It should be noted, however, that while there was no significant difference exists between groups in figure 6, the means for each group were much higher than subjects in the traditional work order submission condition shown in figure 5. The results indi- cate evidence exists that in addition to mitigating organizational language barriers, simply using the mobile submission method improves description accuracy for every- one, regardless of background 4.3 Analysis of Geolocation Finally, the authors examined how the different submission methods affected geolocation. Geolocation was scored as a 1 for individuals providing information suf- ficient enough to allow an expert to find the problem, or a 0 for inaccurately defining the location. For the smartphone submission process, the phones’ GPS location service was used in conjunction with subject descriptions to see if it had correctly captured their location, whereas with the traditional method, only the user’s description was used. Surprisingly, smart phone GPS services for interior work orders generally determined the correct building, but beyond that was essentially useless in assisting in the geoloca- tion of the work order, especially when the building contained multiple floors. Smartphones accurately provided geolocation only 65.46% of the time compared to the traditional method which had an accuracy rate of 71.45%. Figure 7 depicts the relative proportion of the responses that resulted in accurate geolocation by submission method. 60 http://www.i-jim.org Paper—Evaluating Smartphones for Infrastructure Work Order Management Fig. 7. Geolocation by Submission Method To evaluate the results, a Pearson’s Chi-square test was calculated. Significant de- viation between the two submission methods was found (χ2(1) = 4.79, p < 0.05). This significant difference can be observed when looking at the descriptive statistics listed in figure 7. The authors found that geolocation when using the smartphone method was poor (65%) compared to traditional submission users (71%). Hypothesis 3 suggested that GPS services inherent in the smartphone would help improve the geolocation of the maintenance issue, however the results indicate the exact opposite, thus no support exists for hypothesis 3. Further, it was interesting to note that the smartphone users actually performed significantly worse, despite the subjects retaining ability to describe the maintenance problems location similar to subjects in the traditional submission pro- cess condition. While impossible to determine the cause of the observed phenomena, the authors speculate that subjects using the smartphones may have relied too heavily upon the GPS services and felt less need to be descriptive when reporting the mainte- nance problem. 5 Conclusions The results of this study indicate smartphone-based mobile work order submission systems may allow individuals to submit more useful problem descriptions regardless of how much familiarity the submitter has with civil engineering or maintenance pro- cedures. The experiment results also imply craftsmen, when receiving more useful in- formation tailored to their needs, can make better informed decisions and enable them to proceed to the job site only once with a clear idea of the work/parts required, thus iJIM ‒ Vol. 12, No. 8, 2018 61 Paper—Evaluating Smartphones for Infrastructure Work Order Management eliminating unneeded effort. However, the results also indicate caution should be taken when employing GPS enabled smart-phone applications. Users should be trained on the limitations of the GPS services within buildings and encouraged to submit a high level of detail regarding geolocation of the reported problem within the problem de- scription. These results clearly indicate while the potential benefits of smartphone asset management applications exists they should not be considered a “silver bullet” towards eliminated all problems with work order submission systems. 6 Acknowledgment A special thanks to the Air Force Civil Engineering Center for their support in de- veloping this research. 7 References [1] Goldfein, D. (2017). America's Air Force Always There. In Wing Commanders Con-ference (p. 4). [2] American Society of Civil Engineers. (2013). Report Card for America's Infrastruc-ture. Retrieved May 17, 2016, from http://www.infrastructurereportcard.org/a/#p/overview/exec- utive-summary [3] Rittatore, D. (2015). SeeClickFix Corona. Retrieved December 2, 2015, from http://gov.seeclickfix.com/customers/corona-ca [4] East, E. W (1999). Infrastructure Work Order Planning Using Genetic Algorithms, Proceed- ings of the Genetics and Evolutionary Computation Conference, 1510-1516. [5] Golledge, R., Jacobson, D., Kitchin, R., and Blades, M. (2000). Cognitive maps, spa-tial abilities and human wayfinding. Understanding Geographic Space without the Use of Vi- sion. Geographical Review of Japan, 73(2): 93-104. https://doi.org/10.4157/grj1984b.73.93 [6] Chang, K., and Antes, J. (1987). Sex and Cultural Differences in Map Reading. The Amer- ican Cartographer, 14(1), 29–42. https://doi.org/10.1559/152304087783875345 [7] Thorndyke, P., and Hayes-Roth, B. (1982). Differences in spatial knowledge acquired from maps and navigation. Cognitive Psychology, 14(4) 560-589. https://doi.org/10.1016/0010- 0285(82)90019-6 [8] Maguire, E., Woollett, K., and Spiers, H. (2006). London taxi drivers and bus drivers: A structural MRI and neuropsychological analysis. Hippocampus, 16(12), 1091–1101. https://doi.org/10.1002/hipo.20233 [9] William J. Hughes Technical Center. (2016). Global Positioning System (GPS), Standard Positioning Service (SPS) Performance Analysis Report. Atlantic City. Re-trieved from http://www.gps.gov/systems/gps/performance/accuracy/ [10] Boukerche, A., Oliviera, H, Nakamura, E. and Loureiro, A. (2008) Vehicular Ad-Hoc Net- works: A New Challenge for Localisation-Based Systems. Computer Communica-tions, 31(12), 2838-2849. https://doi.org/10.1016/j.comcom.2007.12.004 [11] Bar, M. (2003). A Cortical Mechanism for Triggering Top-Down Facilitation in Visual Ob- ject Recognition. Journal of Cognitive Neuroscience, 15(4), 600–609. https://doi.org/10.1162/089892903321662976 62 http://www.i-jim.org Paper—Evaluating Smartphones for Infrastructure Work Order Management [12] Shea, N. (2014). "Distinguishing Top-Down from Bottom-Up Effects" in S. Biggs et al, Eds. Perception and Its Modalities. Oxford University Press. https://doi.org/10.1093/ac- prof:oso/9780199832798.003.0004 [13] Carlson, J. R., and Zmud, R. W. (1999). Channel Expansion Theory and the Experien-tial Nature of Media Richness Perceptions. Academy of Management Journal, 42: 153-170. [14] Daft, R., and Lengel, R. (1986). Organizational Information Requirements, Media Richness and Structural Design. Management Science (Vol. 32). INFORMS. https://doi.org/10.1287/ mnsc.32.5.554 [15] Daft, R., and Griffin, R. (1983). Information Richness: A New Approach to Manageri-al Behavior and Organization Design. Office of Naval Research Technical Report Se-ries. Col- lege Station. Retrieved from http://bert.lib.indiana.edu:2048/login?url=http://search.ebsco- host.com/login.aspx?direct=true&db=lxh&AN=ISTA1903457&site=ehost-live https://doi.org/10.21236/ADA128980 [16] Galbraith, J. (1974). Organization Design: An Information Processing View. Inter-faces, 4(3), 28–36 https://doi.org/10.1287/inte.4.3.28 [17] Evered, R. (1980). The Language Of Organizations. Navy Postgraduate School, Mon-terey, CA. [18] Pei, M. (1966). Language of the Specialists: A Communications Guide to Twenty Dif-ferent Fields. New York: Funk & Wagnalls. 8 Authors Victor L. Guinn is Captain in the United States Air Force. He is a recent graduate of the Air Force Institute of Technology earning a MS in Engineering Management. His address is 2950 Hobson Way, Wright Patterson AFB, OH 45433. Brent T. Langhals is an Assistant Professor of Information Research Management in the Department of Systems Engineering and Management at the Air Force Institute of Technology. His address is 2950 Hobson Way, Wright Patterson AFB, OH 45433. John J. Elshaw is an Assistant Professor of Systems Engineering in the Department of Systems Engineering and Management at the Air Force Institute of Technology. His address is 2950 Hobson Way, Wright Patterson AFB, OH 45433. Article submitted 30 August 2018. Final acceptance 09 October 2018. Final version published as submit- ted by the authors. iJIM ‒ Vol. 12, No. 8, 2018 63 iJIM – Vol. 12, No. 8, 2018 Evaluating Smartphones for Infrastructure Work Order Management