SKIN September 2019 Volume 3 Issue 5 Copyright 2019 The National Society for Cutaneous Medicine 309 ORIGINAL RESEARCH SkinSpecs: A Mobile Solution That Addresses an Unmet Need for Tracking Chronic Skin Diseases in the Office and at Home Olga K. Afanasiev, MD, PhD1, Mika Tabata, BS2, Akhila Narla, BA2, Gregory Scott, MD, PhD3, Justin M. Ko, MD, MBA1 1Stanford Health Care, Department of Dermatology 2Stanford School of Medicine 3Stanford Health Care, Department of Pathology Chronic skin diseases are challenging and frustrating to manage for both patients and physicians. Patients often experience difficult-to-capture flares in between visits and see multiple providers for their skin condition. Dermatologists currently use subjective measures to document chronic skin conditions; this is problematic at follow up visits when dermatologists lack ABSTRACT Introduction/Objectives: Currently there are no portable solutions to robustly document and longitudinally monitor dynamically changing chronic skin conditions. This study set out to engineer and test a mobile-based 3D imaging solution for chronic skin diseases to enhance clinical workflow and patient care. Methods: SkinSpecs uses smartphone-captured videos of patients’ skin disease and renders 3D true-to-life models that were evaluated by Stanford Health Care dermatologists. Results: We utilized video input to accurately reconstruct interactive 3D models of 16 different skin conditions from 31 patients. Assessment of SkinSpecs 3D reconstruction is faster (p<0.05) compared to descriptive exam, standard photographs or original videos. Dermatologists maintained highest accuracy, confidence and satisfaction with 3D reconstruction. Dermatologist preferred SkinSpecs for documentation over other capture modalities. SkinSpecs was favorably used by dermatologists, with high satisfaction with resolution, breadth of visual information, time and ability to pick up incidental findings. Impact: We identified a proof-of-concept solution to objectively and robustly capture skin disease, with an office and home workflow that is acceptable to providers and patients. This fast, scalable method is deployable on smartphones and could be utilized to augment clinical decision making in the clinic and to empower patients at home. INTRODUCTION 2019 Resident Research Competition – 1st Place SKIN September 2019 Volume 3 Issue 5 Copyright 2019 The National Society for Cutaneous Medicine 310 confidence in assessing disease change in patients seen by another provider.1 While there are many emerging tools and mobile applications for monitoring individual neoplasms,2 dermatologists and patients lack an objective and efficient tool to robustly document and longitudinally monitor changes in chronic and inflammatory skin conditions. Existing smartphone applications either track pre-existing and patient-defined lesions on specific anatomic body parts or have only a rough full body overlay of disease. Additionally, office photography is inconsistent, overly cumbersome and labor- intensive for clinical workflows. Lastly, 3D full body multi-camera imaging systems are expensive and with limited access. This study engineers and tests a proof-of-concept smartphone-based imaging solution “SkinSpecs” that has the potential to improve current workflows in the office and skin disease monitoring strategies at home to positivity impact patient care. Institutional review board approval was received for this study and patients provided written consent. Video capture: High resolution videos (1080p-4K at 30-60 FPS) using iPhone 7 and X were obtained from 31 patients seen at Stanford Health Care dermatology clinic from July 2018 to January 2019. 180°-360° video of the full distribution of skin disease or subject of interest was obtained. Ambient clinic lights were used during video capture. No special setup or training for video capture was required. Mannequins were used in this study and simulated markings or stickers were applied to minimize ambiguity and to ensure the participants surveyed did not have to rely on clinical knowledge. 3D model rendering and analytics: Captured videos were rendered using PhotoScan (Agisoft, LLC) photogrammetry software to create a 3D model that was exported to a ~5-50 megabyte PDF that can be viewed on any device. This 3D reconstruction software automatically generates textured polygonal 3D models using the digital photographs extracted from videos. We automatically extract frames from videos using the scripting language for VLC player, an open source video editing software, and generate individual compressed images of frames using the lossy “jpeg” format. Frames are batch processed into 3d models using PhotoScan .xml commands which stepwise 1) align frames in three dimensional space using a proprietary linear coregistration algorithm, 2) generate sparse then dense point clouds based on the transformation matrix which represents the perspective shift/parallax effect, 3) generate a tessellated 3d model using the dense point cloud as vertices, and finally 4) overlay a high in-plane resolution colored texture onto the model surfaces. We utilized 3D PDF Reader (Tech Soft 3D version 3.8) to view the 3D output on the iPad. For analytics and surface area calculations, we use MeshLab (2016) to manually select the lesion area using a multipoint polygon region‐selection tool for precise planimetry applications. Survey data: Stanford Health Care dermatologists were randomly assigned and anonymously surveyed regarding various “capture” modalities that were presented on an iPad Pro. Captures included: 1) 3D reconstruction obtained as described above, 2) Descriptive exam (which was obtained either from consensus exam from dermatologist asked to describe a mannequin or directly gathered from a clinic note), 3) Photographs, or 4) Video. Complete survey questionnaires are available from the METHODS SKIN September 2019 Volume 3 Issue 5 Copyright 2019 The National Society for Cutaneous Medicine 311 corresponding author. A total of 14 dermatologists were randomized to closely examine 1 specific capture modality and answer specific questions. 34 dermatologists compared all 4 captures side by side and were asked about their preferred capture of choice. Data analyses were performed in Excel 16.0. We captured 31 short videos (mean time=31.8 seconds; range 8-75 seconds) for 3D reconstruction from patients with 16 dermatologic conditions that were rendered into life-like interactive 3D reconstructions as described in methods above (Figure 1-2). Assessment of SkinSpecs 3D reconstruction is fast and accurate with high confidence and satisfaction (Figure 3). Dermatologists were presented with one of four mannequin captures: 3D reconstruction, descriptive exam (text), standard photographs or video (Figure 3A). They were then asked to identify color and shapes of specific lesions as well as percent of face that was erythematous. Physicians who used the 3D reconstruction answered these questions significantly faster (mean=27.8 seconds) than those with descriptive exam (64.7 seconds), photo (51.3 seconds) or video (50.5 seconds) captures, (p<0.05, Figure 3B). Importantly they maintained as high or higher rate of accuracy, confidence in answers and satisfaction with capture compared to those with descriptive exam, photo or video captures (Figure 3C-3D). Survey regarding satisfaction included satisfaction with the overall capture for the purposes of documentation and monitoring for change, and, as applicable, satisfaction with resolution, breadth, ability to pick up incidental findings. SkinSpecs 3D reconstructions has analytic capabilities to accurately and objectively extract data from the life-like subject. The 360 view of 3D model allows for segmentation of face and extraction of lesions (Figure 4A) and 3-dimentional surface areas (Figure 4B). We show that while the dermatologists evaluating the 3D capture come closest to the true estimate of erythematous surface area (15.7% of face), dermatologists overestimate the surface area using all modalities: 3D capture (23.8%), photographs (28.3%), descriptive exam (30%, most were not able to answer from provided description), or video (31.9%) (Figure 4C). Dermatologist preferred SkinSpecs 3D reconstruction over written description, standard photographs or video capture for documentation, but photographs were the capture of choice for longitudinal monitoring of chronic skin disease and for educational purposes (Figure 5). Dermatologists were presented with 1 of 4 patient captures: 3D reconstruction, descriptive exam, standard photographs or video (Figure 5A). For documentation, dermatologists preferred to use 3D capture (29.4%) over descriptive exam (17.6%), photographs (23.5%) or video (17.6%) (Figure 5B). Some preferred a combination of several captures (11.8%). For tracking changes between visits, dermatologist preferred to use photographs (41.2%) over 3D captures (29.4%), descriptive exam (5.9%), video (17.6%) or combination (5.9%). Lastly, for education purposes, physicians had a preference for photographs (48.3%) compared to 3D captures (24.1%), descriptive exam (3.4%) or video (24.1%). In summary, we propose an easy workflow that is able to objectively capture skin disease and its emotional and functional impact on the patient and then digest it to a RESULTS SKIN September 2019 Volume 3 Issue 5 Copyright 2019 The National Society for Cutaneous Medicine 312 capsule summary that can be interpreted by both the patient and provider (Figure 6). Figure 1. SkinSpecs workflow from video to SkinSpecs 3D reconstruction. A) Capture of video using ambient conditions without any prior training or setup required using any mobile device. B) Extraction of relevant frames from video with virtual and generation of a dense cloud. C-D) 3D reconstruction model of a subject, which can be viewed and manipulated using Adobe Acrobat Reader or other freely available applications that can be downloaded on a mobile device. SKIN September 2019 Volume 3 Issue 5 Copyright 2019 The National Society for Cutaneous Medicine 313 Figure 2. Diversity of captured and reconstructed skin diseases. A) Table listing all captured skin disease. B-G) Sample 3D reconstructions (presented here as still images, however 3D PDFs are fully interactable and can be provided upon request): B) psoriasis, C) lipodermatosclerosis, D) dyshidrotic eczema on hand, E) melanoma on right cheek, F) atypical mole syndrome, G) nevi and xerosis. SKIN September 2019 Volume 3 Issue 5 Copyright 2019 The National Society for Cutaneous Medicine 314 Figure 3. Assessment of SkinSpecs 3D reconstruction, descriptive exam, photographs and video captures in terms of time, accuracy, confidence and satisfaction among dermatologists. Assessment of SkinSpecs 3D reconstruction is fast and accurate with high confidence and satisfaction. A) Representative captures that dermatologists were provided with during survey. B) Length of time spent evaluating and answering 3 specific questions (color of simulated lesion, shape of lesion and body surface area) regarding the model. C) Confidence of assessing lesional color and shape as well as body surface area (BSA) among 4 capture on a Likert 5-point scale. D) Summary of time, accuracy, confidence (agree to strongly agree on Likert scale) and satisfaction (agree to strongly agree on Likert scale) among 4 captures. *p<0.05, **p<0.01. SKIN September 2019 Volume 3 Issue 5 Copyright 2019 The National Society for Cutaneous Medicine 315 Figure 4. Analytic capabilities of SkinSpecs 3D capture. A) 360 view of 3D model allows for segmentation of face and extraction of relevant findings. B) Analysis of surface area of full face (left with red-orange mask overlay) and erythematous component only (right with red-orange mask overlay). C) Comparison of physician estimates regarding percentage of face that is erythematous from 3D reconstruction, descriptive exam, photographs, or video capture compared to the calculated true surface area involvement (dashed red line at 15.7%). SKIN September 2019 Volume 3 Issue 5 Copyright 2019 The National Society for Cutaneous Medicine 316 Figure 5. Dermatologists’ preference among 4 different capture modalities for documentation, monitoring and education purposes. A) Representative captures provided to dermatologists for assessment. B) Preferred capture modalities for in-office documentation, disease monitoring between visits and educational purposes among dermatologists. SKIN September 2019 Volume 3 Issue 5 Copyright 2019 The National Society for Cutaneous Medicine 317 Figure 6. Proposed schematic of a SkinSpecs workflow. A short video of relevant body part with skin findings is obtained (no prior training or setup required). The video is then rendered into a 3D model which can be interactively viewed on any device. Patients have the option of additional inputs (such as symptoms, emotional or functional burden, current treatments, etc.). A summary dashboard integrates relevant data captured over time for monitoring by patients and or evaluation by physicians. SKIN September 2019 Volume 3 Issue 5 Copyright 2019 The National Society for Cutaneous Medicine 318 Dermatologists and patients currently lack an objective, universal, cheap, easy way to document and track changes in chronic skin diseases. This is a problem because physicians cannot accurately assess dynamic change in disease severity or response to treatment between office visits, and patients cannot accurately capture and relay their functional and emotional burden of disease while at home. This leads to frustration among physicians and patients due to delay in diagnosis, inadequate control of disease and inconsistency in care. This study engineers and tests SkinSpecs, a smartphone-based, fast, accurate, scalable, low-cost, proof-of-concept solution that is immediately deployable to patients and physicians and could be utilized as a clinical decision support tool to provide better care to our patients with chronic skin diseases. To the authors’ knowledge, this is the first clinically tested mobile 3D imaging solution that robustly captures a large diversity of chronic skin conditions. We show that SkinSpecs enables a fast and accurate assessment of chronic skin diseases with high confidence and satisfaction among dermatologists. Currently, most physicians use subjective measures (such as gestalt, descriptive exam, body surface area (BSA)) to document chronic skin conditions as validated scoring systems and photographs are cumbersome and time consuming.1 Notably, prior literature showed that seemingly objective numeric BSA assessment is poorly estimated by physicians and 2D capture underestimates surface area by 20%.3–5 Furthermore, poor descriptive terms and lack of photographs from referring physicians has led to delayed follow up.6 In our study, descriptive exam, which is currently the gold standard documentation, underperformed in speed, accuracy, confidence and satisfaction compared to 3D reconstruction, photographs or videos. We report that dermatologists came closest to the true BSA when evaluating SkinSpecs 3D reconstruction compared to all other modalities. Furthermore, the 3D capture has the potential to automatically measure skin surface changes and to dynamically and precisely track these changes over time. This ability to accurately capture disease severity, pattern, location and response to treatment is critical to dermatologists’ ability to move along their therapeutic ladders. Interestingly, although providers preferred SkinSpecs 3D reconstructions for documentation, they preferred standard photographs for monitoring disease between visits and for educational purposes. This could be because they are most familiar with standard photography, which is the capture of choice for atypical/dysplastic nevi and occasionally for rashes during an office visit. Similarly, most dermatology curriculums rely on photographs from textbooks and image atlases. Unfortunately, these resources are unable to accurately capture or convey lesional topography (scale, induration, thickness, etc.) or distribution of disease. However, medical education curriculums are now integrating 3D digital virtual simulations to supplement standard curriculums.7,8 Novel repository of 3D reconstructions will better simulate in-person visits to increase medical providers' exposure to rare skin diseases (such as those presented at grand rounds) and many varied presentations of common skin conditions, preserving the ability to revisit them for clinical and didactic purposes. SkinSpecs addresses patients’ need to document their skin disease. Currently, in office professional whole-body photography is too time consuming and infrequent to be DISCUSSION SKIN September 2019 Volume 3 Issue 5 Copyright 2019 The National Society for Cutaneous Medicine 319 useful, and multi-camera imaging systems are expensive and scarcely available.5 A recent study showed that while 90% of patients report it is moderately to extremely important to track their disease, only 16% of patients consistently do, citing lack of optimized tools for documentation. Indeed, patients are willing to spend 10-15 minutes weekly to monthly for documentation, with a preference of using a smartphone.1 Given that SkinSpecs capture takes 1 minute at most, this is well within the realm of time patients are willing to spend to document their disease, making the SkinSpecs workflow conducive to frequent and robust disease capture at home. Integrating the objective disease summary simultaneously with patient-reported outcomes (symptoms, emotional and functional impact)9,10 can further fully capture the impact of the disease on the patient. Dermatologists only recently, but successfully, integrated photography of concerning lesions and biopsy sites into their clinical workflow, which sets precedence for a similar workflow adaptation for chronic skin diseases. One could argue that thorough and robust photo documentation is even more critical for dynamically changing skin rashes that are often elusive during sporadic office visits. An integrated dashboard summary that includes relevant compiled analytics of disease trajectory and response to treatments between office visits and at home could save the physician time in history gathering that currently relies on imperfect memories. Future applications of similar workflows could include inpatient consult services, teledermatology platforms, surgical follow ups, cosmetic services and clinical trials. Furthermore, as we gather larger data sets of full body images as well as other complex associated features (quality of life, associated symptoms, responses to treatments), these could all be used to train deep neural networks and artificial intelligence algorithms for augmented clinical decision making. As hospital systems and payers are moving towards value-driven and precision medicine, individualized disease tracking tools could lead to improved outcomes at a decreased cost to the healthcare system. SkinSpecs could be easily deployed for primary care or the teledermatology workflows. Importantly the Centers for Medicare & Medicaid Services have finalized plans to reimburse healthcare providers for certain remote patient monitoring and telehealth services.11 Limitations of this study include a single academic center recruitment of patients and physicians. The study design included a second person who captured patient videos, which can make it impractical for patients that lack assistance for video capture. Future steps include validation of this tool for specific diseases and determination of resolution limitations. In summary, SkinSpecs is a scalable and mobile-device deployable solution that enables practical, fast, cheap, and accurate capture of high-quality data with great potential to improve care, patient empowerment, educational tools and dermatology access. Conflict of Interest Disclosures: None Funding: Stanford Society of Physician Scholars IRB/Consent: An appropriate institutional review board approved the project, and informed consent was appropriately obtained. Corresponding Author: Olga Afanasiev, MD, PhD olga54@gmail.com SKIN September 2019 Volume 3 Issue 5 Copyright 2019 The National Society for Cutaneous Medicine 320 References: 1. Afanasiev OK, Tabata M, Ko JM. The need for objective and remote tracking of chronic skin diseases. Submitted manuscript. 2019. 2. Guido N, Hagstrom EL, Ibler E, et al. A novel total body digital photography smartphone application designed to detect and monitor skin lesions: a pilot study. J Surg Dermatol. 2018;3(2). doi:10.18282/jsd.v3.i2.177 3. Ramsay B, Lawrence CM. Measurement of involved surface area in patients with psoriasis. Br J Dermatol. 1991;124(6):565-570. doi:10.1111/j.1365-2133.1991.tb04952.x 4. Nichter LS, Williams J, Bryant CA, Edlich RF. Improving the accuracy of burn-surface estimation. Plast Reconstr Surg. 1985;76(3):428- 433. 5. Kohli I, Isedeh P, Al‐Jamal M, et al. Three- dimensional imaging of vitiligo. Exp Dermatol. 2015;24(11):879-880. doi:10.1111/exd.12791 6. Börve A, Dahlén Gyllencreutz J, Terstappen K, et al. Smartphone teledermoscopy referrals: a novel process for improved triage of skin cancer patients. Acta Derm Venereol. 2015;95(2):186- 190. doi:10.2340/00015555-1906 7. Deng X, Zhou G, Xiao B, Zhao Z, He Y, Chen C. Effectiveness evaluation of digital virtual simulation application in teaching of gross anatomy. Ann Anat Anat Anz Off Organ Anat Ges. 2018;218:276-282. doi:10.1016/j.aanat.2018.02.014 8. Zilverschoon M, Kotte EMG, van Esch B, Ten Cate O, Custers EJ, Bleys RLAW. Comparing the critical features of e-applications for three- dimensional anatomy education. Ann Anat Anat Anz Off Organ Anat Ges. 2019;222:28-39. doi:10.1016/j.aanat.2018.11.001 9. Swerlick RA, Zhang C, Patel A, Chren MM, Chen S. The Skindex-mini: a streamlined QOL measurement tool suitable for routine use in clinic. J Am Acad Dermatol. December 2018. doi:10.1016/j.jaad.2018.12.035 10. Armstrong A, Puig L, Langley R, et al. Validation of psychometric properties and development of response criteria for the psoriasis symptoms and signs diary (PSSD): results from a phase 3 clinical trial. J Dermatol Treat. 2019;30(1):27-34. doi:10.1080/09546634.2017.1364694 11. Final Policy, Payment, and Quality Provisions Changes to the Medicare Physician Fee Schedule for Calendar Year 2019 | CMS. https://www.cms.gov/newsroom/fact-sheets/final- policy-payment-and-quality-provisions-changes- medicare-physician-fee-schedule-calendar-year. Accessed April 16, 2019.