SKIN May 2022 Volume 6 Issue 3 (c) 2022 THE AUTHORS. Published by the National Society for Cutaneous Medicine. 201 ORIGINAL RESEARCH Online Patient Health Resources in Nonmelanoma Skin Cancer: An Assessment of Readability, Quality, and Comprehensiveness Terri Shih BS1*, Allison Brimacombe BS, MGH1*, Vivian Y. Shi MD2, Jennifer L. Hsiao MD3 1 David Geffen School of Medicine, University of California Los Angeles, Los Angeles, CA 2 Department of Dermatology, University of Arkansas for Medical Sciences, Little Rock, AR 3 Department of Dermatology, University of Southern California, Los Angeles, CA *Authors have contributed equally to the work Nonmelanoma skin cancer (NMSC) is the most common skin cancer globally1 and affects over 3 million people in the United States annually.2 The incidence of basal cell carcinoma (BCC) and squamous cell carcinoma (SCC) has continued to increase over the last decade, especially in elderly populations.3 Low health literacy negatively correlates with the ability to evaluate online health information.4 The majority (80%) of internet users search online for health information.5 However, previous studies have found that the readability of online health resources for dermatologic conditions such as melanoma6,7 and atopic dermatitis8 fails to meet the American Medical Association (AMA) recommended readability level of 6th grade. There is a paucity of data on the features and accessibility of patient online health resources on NMSC. Here, we investigate the readability, quality, and comprehensiveness of online resources pertaining to NMSC. ABSTRACT Background: Nonmelanoma skin cancer (NMSC), the most prevalent skin cancer, affects over 3 million in the United States annually. Although most patients search for health information on the internet, the readability and quality of NMSC online resources is unknown. Objective: We examine the readability, quality, and comprehensiveness of content in NMSC websites. Methods: Top 50 Google search results for “nonmelanoma skin cancer”, “basal cell skin cancer”, and “squamous cell skin cancer” were evaluated. Advertisements, blogs, scientific articles, and non- comprehensive, professional, and irrelevant sites were excluded. Six readability scales assessed readability. JAMA Benchmark and Discern Instrument assessed quality. Content was evaluated for comprehensiveness. Pearson’s correlation examined the relationship between readability and quality. Results: Seventy-nine websites met inclusion criteria. Average readability level was 11th grade (range 6.8-17.9). No websites met the full criteria of JAMA Benchmark. Only 16.4% had “good”/“excellent” quality per Discern Instrument. Quality and readability scores were not correlated. Most websites discussed risk factors (87.3%), prevention (73.4%), and surgical (87.3%) and nonsurgical (82.3%) treatments. 50.6% included skin cancer images, of which only 17.5% were of skin of color patients. Conclusion: Online NMSC resources need improved readability, quality, and diversity in their representation of skin types. INTRODUCTION SKIN May 2022 Volume 6 Issue 3 (c) 2022 THE AUTHORS. Published by the National Society for Cutaneous Medicine. 202 A search was conducted using the Google search engine on June 10th, 2021 using three independent search terms: “nonmelanoma skin cancer”, “basal cell skin cancer”, and “squamous cell skin cancer”. Browser search history and cookies were cleared, and the searches were performed in incognito mode to avoid influence by personal user data. The first 50 results for each term were evaluated. Websites that were advertisements, blog posts, intended for healthcare professionals, scientific articles, non-comprehensive, or irrelevant to the studied topics of NMSC, BCC, or SCC were excluded. A Preferred Reporting Items for Systematic Reviews and Meta-analysis (PRISMA) diagram outlining the search schema is shown in Figure 1. Readability was assessed using six established and validated readability scales (Flesch Reading Ease, Flesch-Kincaid Grade Level, Gunning-Fog Index, SMOG Index, Coleman-Liau Index, Automated Readability Index) which measure variables such as sentence length and syllables per word. Quality was assessed with the JAMA Benchmark and Discern Instruments. JAMA Benchmark assesses 4 criteria: authorship, attribution (i.e. references), disclosure (i.e. mention of conflicts of interest), and currency (i.e. whether the resource is dated). The modified Discern Instrument (15 items)9 analyzes overall reliability and quality of written health information. Discern scores were independently assessed by two reviewers (AB and TS); discrepancies within two points or fewer were averaged, and any other discrepancies were discussed to consensus with a third independent reviewer (JLH). Data were also collected on author source (i.e. author degree, whether author is a dermatologist), date of most recent website modification, and if websites included discussion of: NMSC risk factors, preventative care, nonsurgical treatments, and surgical treatments. As another gauge of website comprehensiveness, inclusion of multimedia resources besides text, such as videos and images, was measured. The inclusion of skin of color (SOC) images was also evaluated. Correlation between quality and readability was assessed using Pearson’s correlation. P-values < 0.05 were considered statistically significant. All analyses were completed in R V4.1.0. Nonmelanoma skin cancer (NMSC) A total of 50 websites were reviewed for the search term “nonmelanoma skin cancer”, and 24 of these met inclusion criteria (Supplemental Table 1). The top 5 readable websites are shown in Table 1. The average grade level was 11.0 (11th grade) and ranged from 7.5 to 16.3. The websites with the most difficult readability were Mayo Clinic (16.3), Texas Oncology (13.7), and Moffitt (13.6). None of the websites met the recommended 6th grade reading level per the AMA. The average JAMA benchmark score was 1.6 out of 4, with no websites achieving the full 4 points. References were disclosed in 66.7% of websites (16/24), conflicts of interest readily visible in 16.7% (4/24), and date of publication or review was noted in 54.2% (13/24). Author name was provided METHODS RESULTS SKIN May 2022 Volume 6 Issue 3 (c) 2022 THE AUTHORS. Published by the National Society for Cutaneous Medicine. 203 Figure 1. Schematic of search strategy and evaluation of top searched online health resources for nonmelanoma skin cancer SKIN May 2022 Volume 6 Issue 3 (c) 2022 THE AUTHORS. Published by the National Society for Cutaneous Medicine. 204 Table 1. Top 5 websites in readability and in quality for nonmelanoma skin cancer, basal cell skin cancer, and squamous cell skin cancer Skin condition Top 5 Websites in Readability (average readability grade level)^ Top 5 Websites in Quality* (modified Discern score / JAMA Benchmark scores) Nonmelanoma skin cancer Florida Cancer (7.5) Cancer.org (68.5 / 2) Hopkins Medicine (7.7) Cancer.net (65 / 3) Cedars Sinai (8.2) Cancer.gov (56 / 3) Skin Cancer (8.8) University of Michigan Health (56 / 3) University of Michigan Health (9.3) Web MD (50 / 3) Basal cell skin cancer WebMD (6.8) Cancer Research UK (69 / 3) Hopkins Medicine (7.7) Cancer.org (68.5 / 2) Cancer Research UK (8.9) Cancer.net (64.5 / 3) Mount Sinai (9.1) Mount Sinai (50 / 3) Center for Disease Control and Prevention (9.2) Dana Farber (49 / 2) Squamous cell skin cancer Medline Plus (8.2) Cancer Research UK (69 / 3) Mount Sinai (8.2) Cancer.org (68.5 / 2) Cancer Research UK (8.9) Cancer.net (64.5 / 3) American Osteopathic College of Dermatology (9.1) Cancer.gov (56 / 3) Center for Disease Control and Prevention (9.2) Very Well Health (52.5 / 3) ^Average readability grade level is the average of Flesch Kincaid Grade Level, Gunning Fog Score, SMOG Index, Coleman-Liau Index, and Automated Readability Index scores. *Websites listed based on modified Discern scores. The modified Discern Instrument has a maximum score of 75. The JAMA Benchmark has a maximum score is 4. on 20.8% (5/25) websites, of which 100% (5/5) were dermatologists with an MD degree. The average Discern score was 40.8, characterized as “fair,” ranging from 17.5 to 68.6; 11 websites (45.8%) were “poor” or “very poor.” The highest quality websites based on the Discern Instrument were Cancer.org (68.5), Cancer.net (65), Cancer.gov (56), and University of Michigan Health (56), for which readability levels were 11.1, 10.4, 10.1, and 9.3 respectively. In assessing comprehensiveness, 79.2% of websites (19/24) mentioned risk factors, 62.5% (15/24) mentioned preventative care, 75.0% (18/24) discussed nonsurgical treatments, and 75.0% (18/24) discussed surgical treatments. Over half (62.5%) of the websites included multimedia resources, of which 20.8% (5/24) included video, and 58.3% (14/24) included images. Only 37.5% (9/24) websites included images of skin cancer, of which only 2 of the 9 (22.2%) included images of skin cancer in SOC patients. 45.8% (11/24) of websites were available in at least one other language. There was no significant correlation between Discern quality score and average readability score) (R2=0.017, p=0.54). Basal cell carcinoma (BCC) A total of 50 websites were reviewed for the search term “basal cell skin cancer”, and 27 SKIN May 2022 Volume 6 Issue 3 (c) 2022 THE AUTHORS. Published by the National Society for Cutaneous Medicine. 205 of these met inclusion criteria (Supplemental Table 2). The average grade level was 11.0 (11th grade) and ranged from 6.8 to 17.9. The websites with the most difficult readability were Dermnet NZ (17.9), Wikipedia (14.7), and City of Hope (14.7). There were no websites that met the AMA recommended 6th grade reading level. The average JAMA benchmark score was 1.6 out of 4, with no websites achieving 4/4 points. References were disclosed on 48.1% (13/27) of websites, conflicts of interest were readily visible on 22.2% (6/27) of websites, and date of publication or review was noted on 59.3% (16/27) of websites. Author name was provided on 25.9% (7/24) of websites, of which 85.7% (6/7) were dermatologists with an MD degree. The average Discern score was 40.5, characterized as fair, ranging from 16.0 to 69.0; 12 websites (44.4%) were “poor” or “very poor.” The highest quality websites based on the Discern Instrument were Cancer Research UK (69), Cancer.org (68.5), and Cancer.net (64.5), for which readability levels were 8.9, 11.1, and 10.4 respectively. In assessing content, 85.2% (23/27) of websites mentioned risk factors, 81.5% (22/27) mentioned preventative care, 81.5% (22/27) discussed nonsurgical treatments, and 88.9% (24/27) discussed surgical treatments. Most websites (92.6%) included multimedia, of which 44.4% (12/27) included video, and 81.5% (22/27) included images. Only 55.6% (15/27) of websites included images of skin cancer, of which only 1 of the 15 (6.7%) included images of skin cancer in SOC patients. 40.7% (11/27) of websites were available in at least one other language. There was no correlation between quality (Discern score) and average readability score (R2=0.002, p=0.84). Squamous cell carcinoma (SCC) A total of 50 websites were reviewed for the search term “squamous cell skin cancer”, and 28 of these met inclusion criteria (Supplemental Table 3). The average grade level was 11.2 (11th grade) and ranged from 8.2 to 15.5. The websites with the most difficult readability were Dermnet NZ (15.5), Merck Manuals (13.6), and WebMD (13.5). There were no websites that met the recommended 6th grade reading level per the AMA. The average JAMA benchmark score was 1.8 out of 4, with no websites achieving the full 4 points. References were provided in 53.6% (15/28) of websites, conflicts of interest were readily visible on 25.0% (7/28) of websites, and date of publication or review was noted on 64.3% (18/28) of websites. Author name was provided on 32.1% (9/28) of websites, of which 77.8% (7/9) were dermatologists with an MD degree. The average Discern score was 40.9, characterized as fair, ranging from 23.0 to 69.0; half of the websites (14/28) were “poor” or “very poor.” The highest quality websites based on the Discern Instrument were Cancer Research UK (69), Cancer.org (68.5), and Cancer.net (64.5), for which readability levels were 8.9, 11.1, and 10.4 respectively. In assessing content, 96.4% (27/28) of websites mentioned risk factors, 75.0% (21/28) mentioned preventative care, 89.3% (25/28) discussed nonsurgical treatments, and 96.4% (27/28) discussed surgical treatments. Most websites (78.6%) included multimedia, of which 21.4% (6/28) included video, and 75.0% (21/28) included images. Only 57.1% (16/28) of websites included images of skin cancer, of which 4 out of the 16 (25.0%) included images of skin cancer in SOC patients. 35.7% (10/28) of websites were available in at least one other SKIN May 2022 Volume 6 Issue 3 (c) 2022 THE AUTHORS. Published by the National Society for Cutaneous Medicine. 206 language. There was no correlation between quality (Discern score) and readability (average readability score) (R2=0.079, p=0.15). Overall, out of the total 79 NMSC, BCC and SCC websites that were examined, the majority of websites discussed risk factors (87.3%), preventative care (73.4%), nonsurgical (82.3%) and surgical treatment options (87.3%) and included multimedia resources (78.5%) (Figure 2). Images of skin cancer were used in half (40/79, 50.6%) of the websites, and of those, less than a fifth (7/40, 17.5%) included images in patients. Online health resources for NMSC, BCC, and SCC fail to meet recommended reading levels and are highly variable in quality and comprehensiveness. None of the websites in this analysis met the AMA recommended 6th grade reading level for patients, with an average readability level of 11th grade found across all websites analyzed. In terms of quality assessment, no websites met all 4 criteria on the JAMA Benchmark, and fewer than a fifth of the websites were characterized as “Good” or “Excellent” per the Discern Instrument. Only 21/79 (26.5%) websites listed authorship. There was no correlation between quality and readability amongst the websites, highlighting an opportunity to improve readability without compromising the quality of content in written online health materials. Across websites, the least discussed topic was NMSC preventative care. Over a quarter (26.5%) of websites did not include this topic, therefore limiting patient education in primary prevention strategies. Results of this study are consistent with previously published readability studies for melanoma. Two studies on melanoma reported similar inaccessibility in readability and high variation in quality of content.6,7 Additionally, a prior readability study of online content from a search of the term “skin cancer” reported 90% of sites to have an unacceptable readability score on at least one assessment.10 Our study highlights the wide range in quality, readability, and comprehensiveness when specifically evaluating NMSC websites. Low health literacy is directly correlated with poorer health outcomes, including an increase in hospitalizations and decrease in preventative screenings.11 In the United States, it is estimated that 89 million people have limited health literacy,12 highlighting the urgent need to improve the readability of NMSC online health content. Incorporation of multimedia has been reported to be a valuable tool in optimizing DISCUSSION SKIN May 2022 Volume 6 Issue 3 (c) 2022 THE AUTHORS. Published by the National Society for Cutaneous Medicine. 207 Figure 2. Comprehensiveness of nonmelanoma skin cancer, basal cell carcinoma, and squamous cell carcinoma websites patient education.13 Exposure to skin cancer images may improve patient self-exams and earlier detection.14 However, only half of the websites we analyzed included images of skin cancer, representing a missed opportunity for patient education. Although the incidence of NMSC is lower in patients of SOC patients compared to White patients, SOC patients have been found to have disproportionately poor outcomes from NMSC, often presenting with later stage or more aggressive cancers. 15 Despite this, images of skin cancer in SOC patients were only present in 8.9% of websites analyzed in this study. Our findings support a prior study that analyzed the top Google image search results of 71 skin conditions, and found that out of 3700 photographs, less than 10% represented darker skin tones.16 Larger scale studies are needed to address questions regarding the disparities in skin cancer outcomes in ethnic minority groups.17 Online health resources can take a step towards closing this gap by increasing the representation of SOC patients in skin cancer images. Although Google is the top utilized search engine globally,18 one limitation of this study is the utilization of a single search engine. Even though only the top 50 search results were evaluated, this likely represents the websites that are utilized most often by patients, as the first page of search results has been shown to capture 71-92% of all search traffic clicks.19 SKIN May 2022 Volume 6 Issue 3 (c) 2022 THE AUTHORS. Published by the National Society for Cutaneous Medicine. 208 Although a majority of online health resources for NMSC are comprehensive in content, readability and quality of the websites vary widely and are generally suboptimal. Additionally, online images of NMSC in SOC patients are severely underrepresented. This study highlights the need for improved readability and quality of online NMSC resources. As the use of online health resources becomes increasingly popular, it is crucial for dermatologists to vet online dermatologic resources for patients, and to prioritize readability, accessibility, and representation of diverse skin types when creating online content. Conflict of Interest Disclosures: JLH is on the Board of Directors for the Hidradenitis Suppurativa Foundation, has served as a consultant for Boehringer Ingelheim, Novartis, and UCB, and has served as a consultant and speaker for AbbVie. VYS is on the board of directors for the Hidradenitis Suppurativa Foundation (HSF), is a stock shareholder of Learn Health and has served as an advisory board member, investigator, speaker, and/or received research funding from Sanofi Genzyme, Regeneron, AbbVie, Eli Lilly, Novartis, SUN Pharma, LEO Pharma, Pfizer, Incyte, Boehringer-Ingelheim, Aristea Therapeutics, Menlo Therapeutics, Dermira, Burt’s Bees, Galderma, Kiniksa, UCB, WebMD, TARGET-Pharmasolutions, Altus Lab, MYOR, Polyfin, GpSkin and Skin Actives Scientific. There was no financial transaction for the preparation of this manuscript. All other authors report no conflicts of interest. Funding: This study was funded by a Short Term Training Program (STTP) grant supported by the David Geffen School of Medicine at UCLA (Allison Brimacombe) Corresponding Author: Jennifer L. Hsiao, MD 1441 Eastlake Ave Ezralow Tower, Suite 5301 Phone: (310) 601-3367 Email: j.hsiao.publications@gmail.com References: 1. Bray F, Ferlay J, Soerjomataram I, et al. Global cancer statistics 2018: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA Cancer J Clin. 2018;68(6):394–424. 2. Rogers HW, Weinstock MA, Feldman SR, et al. Incidence Estimate of Nonmelanoma Skin Cancer (Keratinocyte Carcinomas) in the U.S. Population, 2012. JAMA Dermatol. 2015 Oct;151(10):1081–6. 3. Ciążyńska M, Kamińska-Winciorek G, Lange D, et al. The incidence and clinical analysis of non- melanoma skin cancer. Sci Rep. 2021 Feb 22;11(1):4337. 4. Diviani N, van den Putte B, Giani S, et al. Low health literacy and evaluation of online health information: a systematic review of the literature. J Med Internet Res. 2015 May 7;17(5):e112. 5. Fox S. The Social Life of Health Information, 2011 [Internet]. Pew Research Center’s Internet & American Life Project; 2011. Available from: http://pewinternet.org/Reports/2011/Social‐Life‐ of‐Health‐Info.aspx 6. Ibrahim AMS, Vargas CR, Koolen PGL, et al. Readability of online patient resources for melanoma. Melanoma Res. 2016 Feb;26(1):58– 65. 7. Alshaikh EA, Almedimigh AF, Alruwaili AM, et al. Patient-Focused Online Resources for Melanoma: Highly Variable Content and Quality. J Cancer Educ. 2019 Aug 1;34(4):775–781. 8. Modiri O, Yee D, Shi VY, et al. Readability, Quality, and Timeliness of Online Health Resources for Atopic Dermatitis. Dermat Contact Atopic Occup Drug. 2020 Dec 1; 9. Weil AG, Bojanowski MW, Jamart J, et al. Evaluation of the quality of information on the Internet available to patients undergoing cervical spine surgery. World Neurosurg. 2014 Aug;82(1–2):e31-39. 10. Basch CH, Fera J, Ethan D, et al. Readability of online material related to skin cancer. Public Health. 2018 Oct 1;163:137–140. 11. Weiss BD. Health Literacy and Patient Safety: Help Patients Understand. Manual for Clinicians. 2nd ed. [Internet]. [cited 2021 Jul 27]. Available from: https://psnet.ahrq.gov/issue/health-literacy-and- patient-safety-help-patients-understand-manual- clinicians-2nd-ed 12. Kutner M, Greenberg E, Jin Y, et al. The Health Literacy of America’s Adults: Results from the 2003 National Assessment of Adult Literacy [Internet]. National Center for Education CONCLUSION mailto:j.hsiao.publications@gmail.com SKIN May 2022 Volume 6 Issue 3 (c) 2022 THE AUTHORS. Published by the National Society for Cutaneous Medicine. 209 Statistics; 2006 [cited 2021 Jul 27]. Available from: https://nces.ed.gov/pubsearch/pubsinfo.asp?pu bid=2006483 13. Delcambre M, Haynes D, Hajar T, et al. Using a Multimedia Tool for Informed Consent in Mohs Surgery: A Randomized Trial Measuring Effects on Patient Anxiety, Knowledge, and Satisfaction. Dermatol Surg Off Publ Am Soc Dermatol Surg Al. 2020 May;46(5):591–8. 14. McWhirter JE, Hoffman-Goetz L. Visual images for patient skin self-examination and melanoma detection: a systematic review of published studies. J Am Acad Dermatol. 2013 Jul;69(1):47–55. 15. Davis DS, Robinson C, Callender VD. Skin cancer in women of color: Epidemiology, pathogenesis and clinical manifestations. Int J Womens Dermatol. 2021 Mar;7(2):127–134. 16. Kurtti A, Austin E, Jagdeo J. Representation of skin color in dermatology-related Google image searches. J Am Acad Dermatol. 2021 Mar 17;S0190-9622(21)00582-X. 17. Hogue L, Harvey VM. Basal Cell Carcinoma, Squamous Cell Carcinoma, and Cutaneous Melanoma in Skin of Color Patients. Dermatol Clin. 2019 Oct;37(4):519–26. 18. Search engine market share worldwide [Internet]. Statista. [cited 2021 Aug 6]. Available from: https://www.statista.com/statistics/216573/world wide-market-share-of-search-engines/ 19. Shelton K. Council Post: The Value Of Search Results Rankings [Internet]. Forbes. [cited 2021 Aug 4]. Available from: https://www.forbes.com/sites/forbesagencycoun cil/2017/10/30/the-value-of-search-results- rankings/