Developing Evidence-based Population Health Informatics curriculum: Integrating competency based model and job analysis 1 Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * 13(1):e10, 2021 OJPHI Developing Evidence-based Population Health Informatics curriculum: Integrating competency based model and job analysis Ashish Joshi PhD, MBBS, MPH1*, Irene Bruce, MPH2, Chioma Amadi, PhD3, Jaya Amatya, MPH4, 1CUNY Graduate School of Public Health and Health Policy, NY 2CUNY Graduate School of Public Health and Health Policy, NY 3Adjunct Faculty, CUNY Graduate School of Public Health and Health Policy, NY 4Montefiore Medical Center, NY Abstract With the rapid pace of technological advancements, public health professions require a core set of informatics skills. The objective of the study is to integrate informatics competencies and job analysis to guide development of an evidence-based curriculum framework and apply it towards creation of a population health informatics program. We conducted content analysis of the Population Health Informatics related job postings in the state of New York between June and July 2019 using the Indeed job board. The search terms included “health informatics” and “population health informatics.” The initial search yielded 496 job postings. After removal of duplicates, inactive postings and that did not include details of the positions’ responsibilities resulted in 306 jobs. Information recorded from the publicly available job postings included job categories, type of hiring organization, educational degree preferred and required, work experience preferred and required, salary information, job type, job location, associated knowledge, skills and expertise and software skills. Most common job title was that of an analyst (21%, n=65) while more than one-third of the hiring organizations were health systems (35%, n=106). 95% (n=291) of the jobs were fulltime and nearly half of these jobs were in New York City (47%, n=143). Data/statistical analysis (68%, n=207), working in multidisciplinary teams (35%, n=108), and biomedical/clinical experience (30%, n=93) were the common skills needed. Structured query language (SQL), Python, and R language were common programming language skills. A broad framework of integrating informatics competencies, combined with analysis of the skills the jobs needed, and knowledge acquisition based on global health informatics projects guided the development of an online population health informatics curriculum in a rapidly changing technological environment. Key words: Population health informatics, Workforce, Competencies, Public health, Training, Skills Correspondence: *ashish.joshi@sph.cuny.edu DOI: 10.5210/ojphi.v13i1.11517 Copyright ©2021 the author(s) mailto:ashish.joshi@sph.cuny.edu Developing Evidence-based Population Health Informatics curriculum: Integrating competency based model and job analysis 2 Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * 13(1):e10, 2021 OJPHI 1.0 Introduction There is a growing need for evidence-driven population health interventions. The emerging science of population health informatics has continually demonstrated significant promise in revolutionizing the delivery of population health interventions. Population health informatics tools and technologies are rapidly enhancing the capacity to translate growing population data into meaningful information that can inform actionable insights. These population health technologies have already shown promise in improving the effectiveness of core public health services such as outbreak detection and control, immunizations and real-time information on health impacts of emergencies [1]. Expertise in population health informatics is critical in supporting the activities of public health agencies toward building and sustaining information capabilities that meet evolving population health needs. Owing to the wealth of public health functions that rely on data assessments and the rapid pace of technological advancements, public health professions require a core set of informatics skills [1]. In addition, health agencies need health informaticians who work at junior and senior organizational levels to oversee the development and implementation of successful IT systems [1]. There are numerous training programs in health informatics in the United States. More than 90 universities provide training in biomedical informatics at certificate, masters and doctoral levels [2]. A plethora of training programs in the field of public health informatics has emerged in the past decade. Compared to the public health informatics programs offered, population health informatics is a relatively recent advancement. Studies have shown that despite the growing volume of training programs in Informatics as well as public health, gaps exist in the competencies of public health graduates in meeting the requisite skills and demands of the dynamic public health workforce [2-4]. In particular, existing workforce assessments have shown that public health professionals across many local health departments lack adequate health informatics skills [5]. A gap analysis of biomedical informatics graduate education competencies identified key topics that were missing from the existing graduate curricula including community health, translational and clinical research, knowledge representation, data mining, communication and evidence-based practice [2]. An initial assessment of existing Public/Population Health Informatics programs that offered certificate, masters and doctoral levels, was conducted [6]. The study explored informatics-training programs currently available to meet the growing demand for a trained workforce in global settings. These programs then matched against the 14 competencies determined by the Centers for Disease Control and Prevention (CDC). These competencies included workforce supporting development of strategic direction for public health informatics within the enterprise, participating in development of knowledge management tools for the enterprise, use informatics standards, ensure that knowledge, information, and data needs of project or program users and stakeholders are met, support information system development, procurement, and implementation that meet public health program needs, manage IT operations related to project or program (for public health This is an Open Access article. Authors own copyright of their articles appearing in the Online Journal of Public Health Informatics. Readers may copy articles without permission of the copyright owner(s), as long as the author and OJPHI are acknowledged in t he copy and the copy is used for educational, not-for-profit purposes. Developing Evidence-based Population Health Informatics curriculum: Integrating competency based model and job analysis 3 Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * 13(1):e10, 2021 OJPHI agencies with internal IT operations), monitor IT operations managed by external organizations, communicate with cross-disciplinary leaders and team members, evaluate information systems and applications, participate in applied public health informatics research for new insights and innovative solutions to health problems, contribute to development of public health information systems that are interoperable with other relevant information systems, support use of informatics to integrate clinical health, environmental risk, and population health, implement solutions that ensure confidentiality, security, and integrity while maximizing availability of information for public health and conduct education and training in public health informatics. Our findings revealed that all the programs addressed at least some of the competencies for Public Health Informatics. The predominant competencies covered included strategy development, interoperability, integration of clinical and population health, knowledge and project management. One of the key findings; however was that none of the Public Health Informatics programs addressed all the CDC competencies [6]. Curriculum is critical to foster effectiveness in higher education [7] both in the short and long- term [8]. Results of previous study has demonstrated the need for curricular change in existing programs to facilitate critical contributions toward assuring clinical, social, and population health [9,10]. Variety of approaches to curriculum development predominantly focus on medical school curricula. Six-step approach to curriculum development in medical schools include problem identification and general needs assessment, targeted needs assessment, goals and objectives, educational strategies, and implementation [11]. Another study suggested a 5-step theoretical framework to curriculum development that includes environmental analysis, graduate competencies, curriculum development, pedagogical strategies, and implementation, evaluation, feedback [8]. Informatics curricula defined using educational competencies and requirements based on literature reviews, internet searches, needs assessments and assessments of HIT workforce such as local health departments [12-14]. The International Medical Informatics Association (IMIA) has extended a global perspective on educational competencies with a three- dimensional framework of educational needs and recommended course content [15]. Other approaches to curricula development have included a combination of literature reviews, focus groups, and expert panel reviews/decision making by subject-matter experts (including researchers, practitioners and policy makers) to define specific competencies that course evaluations can address [16-19]. While the variety of approaches to curriculum development utilize existing evidence and educational strategies, the majority of these approaches fail to incorporate a workforce skills assessment and integration into curriculum development. Several informatics workforce needs assessments and agendas repeatedly recommend appropriate training and integration of eHealth in existing curricula as methods for addressing the skills gaps in the informatics workforce [5,20-24]. Evaluating health informatics programs will foster effective curricula development that will meet the diverse needs of the informatics workforce [25]. An integrated approach of mapping curriculum competencies to workforce demands can help guide development of curriculum that can specifically address demands of the emerging, fast changing job market and make it possible to review and refine the existing health informatics training program competencies. The objective of our study is to integrate program specific competencies with analysis of available positions on job search engine “Indeed” to identify population health informatics workforce needs, Developing Evidence-based Population Health Informatics curriculum: Integrating competency based model and job analysis 4 Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * 13(1):e10, 2021 OJPHI skills and expertise in the state of New York. Findings of the study inform development of an evidence based graduate population health informatics curriculum at CUNY Graduate School of Public Health and Health Policy. Methods We conducted content analysis of the Population Health Informatics related job postings in the state of New York between June and July 2019 using the Indeed job board. Indeed.com is one of the leading job sites. It curates millions of job listings across company career sites, job boards, associations, newspaper classifieds and other online sources [26].The aim of the search was to characterize the knowledge, skills and expertise required by the employers of the informatics workforce. The search terms included “health informatics” and “population health informatics.” The initial search yielded 496 job postings. After removal of duplicates and inactive postings 321 job postings remained. Excluding postings that did not include details of the positions’ responsibilities resulted in 306 jobs. Content analysis on the remaining final 306 jobs done. The content analysis process involved an inductive and deductive approach [27]. An inductive approach helped to identify and categorize core job posting information followed by a deductive approach that helped to evaluate the skills and expertise required for specific job categories. Steps include; 1. Identification and reclassification of job titles: Individual reviews of all 306-job postings conducted. Some examples of the job postings included clinical informatics analyst, application analyst, health informatics specialist, and healthcare analytics manager. The 306 job positions were subsequently classified into 14 unique job functions including administrator, analyst, consultant, coordinator, data governance, designer/developer, director, engineer, faculty/researcher, informatics specialist, manager, quality informatics, specialist, and others (representing job functions that did not fit into the listed roles). This involved an iterative deductive process in which related job positions were combined based on job titles and job descriptions. The degrees required/preferred, and experience required/preferred abstracted from the newly created job categories. 2. Extraction and categorization of technical skills: Review of the existing literature on informatics workforce needs identified 44 skills across various informatics jobs in diverse settings [28]. These skills used as a template to identify the skills listed across the job postings in the current study. Seventy-eight skills extracted from the current study. The process of identifying skills involved an extensive and iterative review of each job description and summary details. The final skills (n=78) identified in our study included both new skills from the current postings in addition to skills that had been previously identified (n=44) in the literature. Developing Evidence-based Population Health Informatics curriculum: Integrating competency based model and job analysis 5 Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * 13(1):e10, 2021 OJPHI Figure 1. Flow chart describing the job assessment process. Variable extraction Following variables extracted from each job posting and included; • Job categories: Information recorded from the publicly available job postings and characterized into different categories as analyst, director, faculty/researcher, manager, informatics specialist, designer or developer. • Hiring organization type: Information recorded from publicly available job postings on the types of organizations where these jobs were available including health system, hospital, academia, pharmacy, and biotechnology/consulting or other. Developing Evidence-based Population Health Informatics curriculum: Integrating competency based model and job analysis 6 Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * 13(1):e10, 2021 OJPHI • Degree (Required/Preferred): Information was extracted on both required and preferred degree requirements and was characterized into bachelors/ or masters, PhD and above. Information also recorded if the degree requirements not listed. The required degree requirement reflected the minimal education that the applicant should possess to submit the application for the job. • Work experience (Required/Preferred): Information recorded on both required and preferred work experience. The work experience characterized as either 1-2 years, 3 to 4 years, 5 to 7 years, 8 to 10 years and 11 to 15 years. In addition, there were job postings that reported as either commensurate with degree or reported as not listed. • Salary: Information was recorded on salary and income categories included an annual salary of <$40,000, followed by several other salary categories including $40,000 - $49,999, $50,000- $60,099, $60,100- $70,099, $70,100 - $80,099, $81,000 - $90,199, $90,200 - $123,499, $110,000 - $123,549, $123,500+ and $145,400 and above. There were jobs whose salary could not be determined. • Job type: Information recorded on job types including whether the job postings were contract, full time, internship, art-time or temporary. • Location: Information recorded on geographic distribution of the population health informatics jobs. • Knowledge, skills and expertise: Variable information extracted from the job descriptions so that specific knowledge, skills and expertise needed identified. • Software skills: Analysis performed to determine the most common software skills listed with each job description. 2.1 Statistical Analysis Descriptive statistics computed for the job posting characteristics and skills identified. The results presented as percentages and frequencies. The job postings reclassified into broader categories representing unique job functions. Cross tabulations conducted to examine the degrees required/preferred and experience required/preferred across each of the job categories. Cross tabulations conducted to examine the distribution of skills across the various job categories. SAS text mining used to identify the key terms that used in job description and terms characterized into knowledge, skills and expertise categories. Analysis performed in SAS v 9.4 and Microsoft Excel. Results 3.1. Characteristics of the job postings A total 306 job postings evaluated. The job title of the various positions were characterized as analyst (21%, n=65), director (15%, n=46) and faculty/researcher (14%, n=44) (Table 1). These were followed by other titles such as manager (10%, n=32), informatics specialist (9%, n=26), and designer/developers (8%, n=23). More than one-third of the hiring organizations for these jobs were health systems (35%, n=106) followed by hospitals (19%, n=59), and academia (17%, n=51). Almost 100% of the jobs were fulltime (n=291). Around half the jobs were based in New York City (47%, n=143) and 11% of them were based in Westchester in NY state (n=33). More than half of the jobs listed required either a bachelor’s degree as the minimum degree requirement (63%, Developing Evidence-based Population Health Informatics curriculum: Integrating competency based model and job analysis 7 Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * 13(1):e10, 2021 OJPHI n=192), 13% masters (n=41) and 9% doctoral degree (n=26). Only 19% of the listings preferred a master’s degree (n=58). However, 71% of them did not list the degree preferred. Almost half of the job listings required that candidates possess 3 to 7 years of experience (44%, n=132). Thirteen percent required 1 to 2 years of experience (n=40). The most common annual salary range for the jobs were $70,100 to $80,099 (22%, n=68), and $110,000 to $123,549 (19%, n=58). Twelve percent of the jobs had a salary range of $81,000 to $90,199. Table 1 indicates the majority of jobs were available in New York. Table 1. Descriptive analysis of the job postings Variables assessed N=306 Job categories Analyst 21% (n=65) Consultant 2% (n=7) Specialist 6% (n=17) Quality informatics 4% (n=12) Coordinator 4% (n=9) Data governance 2% (n=5) Director 15% (n=46) Faculty/Researcher 14% (n=44) Manager 10% (n=32) Informatics Specialist 9% (n=26) Designer/Developer 7% (n=23) Other 6% (n=20) Hiring organization type Health System 35% (n=106) Hospital 19% (n=59) Academia 17% (n=51) Insurance 3% (n=8) Pharmacy 9% (n=27) Biotechnology 4% (n=13) Consulting 4% (n=13) Other 9% (n=29) Degree required Developing Evidence-based Population Health Informatics curriculum: Integrating competency based model and job analysis 8 Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * 13(1):e10, 2021 OJPHI Bachelor's and Above 63% (n=192) Master's and Above 13% (n=41) PhD and MD or Equivalent 9% (n=26) Advanced degree 3% (n=10) Not listed 12% (n=37) Degree preferred Bachelor's and Above 2% (n=7) Master's and Above 19% (n=58) PhD and MD or Equivalent 6% (n=19) Advanced Degree 1% (n=4) Not listed 71% (218) Experience required (years) 1 to 2 years 13% (n=40) 3 to 4 1% (n=2) 5 to 7 22% (n=66) 8 to 10 4% (n=13) Commensurate with degree 3% (n=10) Not listed 36% (n=109) Experience preferred (years) 1 to 2 years 2% (n=6) 3 to 4 2% (n=5) 5 to 7 2% (n=5) 8 to 10 2% (n=4) Not listed 92% (n=286) Salary (based on Indeed estimation) < $40,000 1% (n=3) $40,000 - $49,999 3% (n=7) $50,000- $60,099 7% (n=22) $60,100- $70,099 8% (n=23) $70,100 - $80,099 22% (n=68) $81,000 - $90,199 12% (n=36) Developing Evidence-based Population Health Informatics curriculum: Integrating competency based model and job analysis 9 Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * 13(1):e10, 2021 OJPHI $90,200 - $123,499 3% (n=40) $110,000 - $123,549 19% (n=58) $123,500+ 3% (n=9) $145,400+ 1% (n=2) NA 12% (n=38) Job Type Contract 1% (n=3) Full-Time 95% (n=291) Internship 1% (n=2) Part-time 3% (n=7) Temporary 1% (n=3) Location of Jobs: Analysis of the job locations showed that majority of the jobs were available in New York County (48%; n=145) followed by Westchester (11%; n=33), Nassau (5%; n=15) and Suffolk (4%; n=13) (Figure 2). Ninety –five percent (n=291) of the jobs were full time. Of the 145 jobs available in NY, majority of them were full time (95%; n=138). Similar trends seen across other counties in the NY state. It was found that of the total jobs available in New York County, majority of them (31%; n=45/145) were offering salary in the range of $110,000 - $123,549; 16% (n=23) offered salary between $70,100 - $80,099 and 15% offered $90,200 - $123,499. 145 33 15 13 12 12 12 12 9 8 6 5 0 20 40 60 80 100 120 140 160 New York Westchester Nassau Suffolk Albany Bronx Erie Monroe Queens Onondaga Kings Rockland Distribution of number of population health informatics jobs in New York State using Indeed Search Engine Figure 2. Distribution of population health informatics jobs in New York State Developing Evidence-based Population Health Informatics curriculum: Integrating competency based model and job analysis 10 Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * 13(1):e10, 2021 OJPHI 3.2. Stratified descriptive analysis of the job postings 3.2.1. Degree qualifications required across the job categories Eighty-seven percent (n=57/65) of the analyst job category postings mentioned the degree requirements required. A hundred percent of coordinator and data governance jobs indicated degree requirements required in their job postings. Less number of consultant and quality informatics jobs had indicated required degree requirements as compared to other job categories (Figure 2). Figure 3. Percentage distribution of job categories reporting degree requirements Eighty four percent of the postings in the analyst job category required individuals to have bachelors and above degree requirements while 25% (n=15) of them preferred Master’s degree. More than half of the positions in the job category of specialist indicated bachelors and above degree requirements (69%; n=11/16) and 31% (n=5) preferred Master’s. Doctoral degree requirements were primarily required for faculty/researcher, director and manager job categories. Nearly 37% of the director positions preferred Masters. Table 2. Distribution of degrees required for each of the job categories Job Category Role Total jobs (N) Reporte d degree (N) Required Degree Degree Preferred Advance d Degree (N) Bachelor's and Above (N) Master’s degree and Above (N) PhD and MD or equivalent (N) Masters Analyst 65 57 3 48 6 15 Consultant 7 5 4 1 1 Developing Evidence-based Population Health Informatics curriculum: Integrating competency based model and job analysis 11 Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * 13(1):e10, 2021 OJPHI Specialist 17 16 11 5 Quality informatics 12 9 7 1 1 2 Coordinator 9 9 1 7 1 2 Data governance 5 5 5 3 Director 46 43 1 25 9 8 16 Faculty/Res earcher 44 39 17 10 12 4 Manager 32 27 20 3 4 6 Informatics Specialist 26 22 3 17 2 4 Designer/De veloper 23 20 1 15 4 3 Other 20 17 1 16 2 Total 306 3.2.3. Knowledge, skills and expertise required across the job postings Data/statistical analysis (68%, n=207), collaboration/working in multidisciplinary teams (35%, n=108), biomedical/clinical experience (30%, n=93), electronic health record/electronic medical record (28%, n=85), and communicating findings/delivering actionable results (275, n=-82) were the most common skills required across various job postings (Table 3). Around 20% of the jobs required skills in management/organization (23%, n=71), consumer-centric/patient outcome knowledge (23%, n=71), data governance/standards/quality skills (22%, n=67), performance improvement (22%, n=66), data manipulation (21%, n=64), and quality measures (21%, n=65) (Figure 4) Developing Evidence-based Population Health Informatics curriculum: Integrating competency based model and job analysis 12 Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * 13(1):e10, 2021 OJPHI Figure 4. Percentage distribution of knowledge, skills and expertise required for various job categories Other knowledge, skills and expertise included familiarity with R language (10%; n=32), MS Office (10%; n=32), SAS (9%; n=28) and familiarity with HL7 knowledge (8%; n=23). Some of the other skills included digital health knowledge (8%; n=23), innovative skills (8%; n=23), identifying business problem skills (7%; n=21), and data management platforms (7%; n=22). Other technology skills included java script (8%; n=21), C++ (5%; n=16), dashboard skills such tableau (5%; n=14), machine learning and requirement analysis skills (4%n=12). 3.2.4. Distribution of skills required across the job categories Data/statistical analysis skills were mostly required by analysts (26%, n=51), directors (15%, n=31) and faculty/researchers (12%, n=25) (Figure 4). Skill in collaboration/working in multidisciplinary teams was most commonly required among analysts (23%, n=25) and directors (21%, n=23). Biomedical/clinical experience and EHR/EMR knowledge were most commonly required among analysts (22%, n=20; 22%, n=19) and informatics specialists (18%, n=17; 18%, n=15). Management/organizational skills were mostly required among directors (24%, n=17), analysts (18%, n=13), and managers (14%, n=10). The least common skills included dashboards (4%, n=13), machine learning (4%, n=12), developing algorithms (2%, n=6), prioritization/resource planning (1%, n=4), and population health skills (1%, n=3) were mostly required by analysts, faculty/researchers, managers, and quality informatics professionals. Developing Evidence-based Population Health Informatics curriculum: Integrating competency based model and job analysis 13 Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * 13(1):e10, 2021 OJPHI Figure 5. Percentage distribution of skills required across the job categories 3.2.5. Software skills required across the job categories Structured query language (SQL) (15%, n=46), Python (14%, n=44), and R language (11%, n=33) were the most common programming language requirements. MS Office (10%, n=31) and Tableau (5%, n=15) were the most common data visualization tool skills required. Less than ten percent of the job postings mentioned experience with SAS (9%, n=27), and Java (8%, n=24). The most common relational database management skills required across the postings included Microsoft SQL server (5%, n=14) and Oracle (4%, n=11). SQL was most common software skill mentioned among the 65 positions characterized as Analyst (35%, n=23), followed by SAS (22%; n=14) and R (22%; n=14). Skills in Python (15%; n=10), MS Office (15%; n=10) and tableau were additional skills reported for the job category of analyst. Similarly, Python (17%; n=2), R (17%; n=2), SQL (17%; n=2) and MS Office (17%; n=2) were commonly reported skills for the job category Quality Informatics. Python skills were most commonly reported also for the job categories such as data governance (40%; n=2) and designer/developer (43%; n=10), coordinator (22%; n=2) and informatics specialist (15%; n=4) (Figure 6). Developing Evidence-based Population Health Informatics curriculum: Integrating competency based model and job analysis 14 Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * 13(1):e10, 2021 OJPHI Figure 6. Software skill requirements across job postings. 3.2.6. Time commitment required across the job categories, and by location. Almost all the job postings required a full-time commitment (95%, n=291 out of 306) (Table 2). Analysts (21%, n=64), director (15%, n=46), faculty/researcher (13%, n=39), and manager (10%, n=31) were the most common full-time jobs. Other less common full-time positions included informatics specialists (8%, n=26), designer/developer (7%, n=22), and specialists (5%, n=16). Around half of the full-time jobs (48%, n=138) were based in New York. Eleven percent (n=32) of the full time jobs were based in Westchester. Additional full-time jobs were available in Nassau (5%; n=15), followed by Albany (4%; n=12), Bronx (4%; n=12), Monroe (4%; n=12), Suffolk (4%; n=12), Erie (4%; n=12) and Queens (3%; n=8) (Table 3). Table 3. Time commitment by job location Total (N=306) Contract Full- time Internship Part- time Temporary Albany N=12 N=12 Bronx N=12 N=12 Brooklyn N=6 N=6 Broome N=2 N=2 Developing Evidence-based Population Health Informatics curriculum: Integrating competency based model and job analysis 15 Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * 13(1):e10, 2021 OJPHI Clinton N=1 N=1 Columbia N=3 N=3 Dutchess N=3 N=3 Erie N=12 N=1 N=10 N=1 Herkimer N=1 N=1 Monroe N=12 N=12 Nassau N=15 N=15 New York N=145 N=1 N=138 N=2 N=2 N=2 Oneida N=1 N=1 Onondaga N=6 N=5 N=1 Queens N=9 N=1 N=8 Rensselaer N=1 N=1 Rockland N=5 N=4 N=1 Saratoga N=2 N=2 Schenectady N=3 N=3 St. Lawrence N=2 N=2 Suffolk N=13 N=12 N=1 Tompkins N=2 N=2 Ulster N=1 N=1 Warren N=2 N=2 Westchester N=33 N=32 N=1 Yates N=1 N=1 3.2.7. Expected salaries across the job categories, and by location The annual salary range for majority of the analyst job categories was $70,100 - $80,099 (31%; n=20) followed by 17% (n=11) having a salary range of $60,100- $70,099 (Figure 7). Of the seven consultant job positions, most of them were in a salary range of $90,200 - $123,499 (43%; n=3). Of the seventeen positions in the specialist job category, 35% (n=6) of them had a salary range of $70,100 - $80,099 while 23% (n=4) had a salary range of $50,000- $60,099. Similarly, for other positions advertised across different job categories such as quality informatics (33%; n=4), coordinator (33%; n=3), faculty/researcher (25%; n=11), and informatics specialist (35%; n=9), the salary range was $70,100 - $80,099. However, majority of the positions in the job category of Director (48%; n=22) or Designer/Developer (26%; n=6) had an annual salary range of $110,000 - $123,549. Developing Evidence-based Population Health Informatics curriculum: Integrating competency based model and job analysis 16 Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * 13(1):e10, 2021 OJPHI Figure 7. Percentage distribution of most common salary distribution among various job categories Thirty-two percent (N=45) of the jobs in the New York County were in the salary range of $110,000 - $123,549, 17% (n=23) in the salary range of $70,100 - $80,099, 16% (n=22) in the salary range of $90,200 - $123,499. Similarly, in Westchester county, 31% (n=10) of the jobs had a salary range of $70,100 - $80,099. Study findings demonstrate skills that are essential to fulfill the population health informatics job needs. Table 4 demonstrates how skills often listed as required or desired for positions in population health informatics. Findings of this study utilized to inform the design and development of a population heath informatics curriculum. Table 4. Skills prioritized by employers mapped to competencies and incorporated into the curriculum Example of Skills identified in job postings Informatics Competency Public Health competency Working in multidisciplinary teams Assess stakeholder data, information, and knowledge needs Perform effectively on interprofessional teams Requirement skills Design, develop, and implement user-centered population health information systems effectively Propose strategies to identify stakeholders and build coalitions and partnerships for influencing public health outcomes Consumer-centric Establish frameworks for evaluating the implementation process of information systems and applications, and make Developing Evidence-based Population Health Informatics curriculum: Integrating competency based model and job analysis 17 Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * 13(1):e10, 2021 OJPHI recommendations to improve user satisfaction and outcomes Data management Assess stakeholder data, information, and knowledge needs Select quantitative & qualitative data collection methods appropriate for a given public health context Data/statistical analysis Establish frameworks for evaluating the implementation process of information systems and applications, and make recommendations to improve user satisfaction and outcomes Select quantitative & qualitative data collection methods appropriate for a given public health context Interpret results of data analysis for public health research, policy or practice. Electronic health record Design, develop, and implement user-centered population health information systems effectively Design a population-based policy, program, project or intervention Communicating findings Design, develop, and implement user- centered population health information systems effectively Select communication strategies for different audiences and sectors Describe the importance of cultural competence in communicating public health content Communicate audience-appropriate public health content, both in writing and through oral presentation Management/organization skills Manage and direct health informatics planning for projects related to public health and information technology Apply principles of leadership, governance and management, which include creating a vision, empowering others, fostering collaboration and guiding decision making Data governance Recommend strategies and solutions that integrate informatics knowledge within organizations and communities and ensure confidentiality, security, and integrity Discuss multiple dimensions of the policy-making process, including the roles of ethics and evidence Standards Apply informatics standards appropriately and contribute to standards development efforts Familiarity with HL7 knowledge Apply informatics standards appropriately and contribute to standards development efforts Developing Evidence-based Population Health Informatics curriculum: Integrating competency based model and job analysis 18 Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * 13(1):e10, 2021 OJPHI Innovation Manage and direct health informatics planning for projects related to public health and information technology Recommend strategies and solutions that integrate informatics knowledge within organizations and communities Apply systems thinking tools to a public health issue Assess population needs, assets and capacities that affect communities’ health Dashboard/Data Visualization Design, develop, and implement user- centered population health information systems effectively Communicate audience-appropriate public health content, both in writing and through oral presentation Tableau, Python and MS SQL Design, develop, and implement user-centered population health information systems effectively Analyze quantitative and qualitative data using biostatistics, informatics, computer-based programming and software, as appropriate Job analysis combined with competencies drawn from various sources such as Council on Education for Public Health (2016 CEPH) [29] criteria, AMIA working group pre-symposium [30] and public health informatics competencies created by the Center for Disease Control [31] and knowledge acquisition through global implementation of informatics projects utilized to design and develop the MS population health informatics curriculum (Figure 8). Core competencies included both public health and Population Health Informatics. Public health competencies [31] were addressed through the required MPH curriculum courses and integrated to informatics competencies through a wide range of course offerings including fundamentals of population health informatics, design and development of population health information systems, surveillance systems and applications of population health informatics, mobile health, population health dashboards, and monitoring and evaluation. Figure 8. An Integrated approach to academic program development: The Graduate School of Public Health and Health Policy (SPH) of the City University of New York established first of its kind, Master of Science (MS) in Population Health Informatics, effective fall 2019. The program emphasis on how to design, develop, implement, and evaluate Developing Evidence-based Population Health Informatics curriculum: Integrating competency based model and job analysis 19 Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * 13(1):e10, 2021 OJPHI technological interventions and innovations using data driven, evidence-based approaches to guide policy making for the improvement of population health outcomes. While most programs, take a hospital-centric approach, the program primarily focuses on how to operationalize informatics solutions to address important public health challenges influencing individuals, families, communities, and the environment in which they live. The MS program in population health informatics is an online, 39 graduate-level credits program including 15 credits of public health coursework, 18 credits of coursework in health informatics, and 6 credits of a dual practice/culminating experience. The purpose of the Master of Science in Population Health Informatics is to equip students with an in-depth understanding of informatics that supports the development of interventions addressing 21st century public health challenges. Students will acquire the skills necessary to apply technology at the intersection of clinical care and health sciences, strengthening preventive care at the population level. The five public-health specific courses will cover the public health competencies, adopted from the Council on Education for Public Health (CEPH). The three credit each six-concentration courses include Fundamentals of Population Health Informatics, Principles of Consumer Health Informatics, Applications of Population Health Informatics, Population Health Dashboards, Design, development and evaluation of Health technologies and Mobile Health Interventions. The program also offers practicum/Culminating experience. The practicum involves workshop approach where students not only work on their informatics projects but also acquire skills in the areas of MS SQL, Python, data visualization tools such as Excel and Tableau, API, HTML and android toolkit as these skills broadly identified to be critical in the job analysis that conducted. Through this project and the portfolio, students demonstrate their ability to integrate and synthesize the knowledge and skills they have attained in the program. Discussion Education continues to advance, driven by a combination of changes in the practice domain, theoretical developments, and workplace demands [32]. Curricula evolve over time, both because of the newness of the field and in response to the job market. There is an increased focus on public health information and knowledge management systems highlighting need for public health professionals skilled in designing and implementing these systems. Prior study has shown that four competency sets adequately represented in public health informatics curriculum including Leadership and System Thinking Skills, followed by Financial Planning and Management Skills, Community Dimensions of Practice Skills, and Policy Development/Program Planning Skills [33]. In another prior study, the public health informatics design focused on public health core competencies and organization/systems/information sciences related to analyzing, designing, implementing, and disseminating effective information systems in public health [34]. Though many theoretical frameworks and competency guidelines proposed for informatics education, less attention paid to informatics education using analysis of the jobs available today. Results of our study helped identify knowledge, skills and expertise that match employer requirements and ensuring that curriculum meet those requirements in this rapidly changing job market. Our study combined competencies and analyzed job skills as an integrated approach to design and develop our curriculum. This is a perspective that has been lacking in previous literature Developing Evidence-based Population Health Informatics curriculum: Integrating competency based model and job analysis 20 Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * 13(1):e10, 2021 OJPHI on this topic. There are a growing number of ‘informatics’ programs, but they vary substantially by academic level and focus [32]. This suggests studying existing educational programs and their curricula, and examining how such knowledge can contribute to defining the informatics professions. Results of our study show that analyst was the most common job category, followed by director and faculty/researcher. Population Health Informatics professionals work in practice, research, or academia and whose primary work function is to use informatics to improve the health of populations. Results of our study showed that most common hiring organizations were health system and hospitals followed by academic institutes. Recent studies highlight the need for informatics training in the public health workforce [35]. Similarly, the Council on Education for Public Health (CEPH) includes informatics as a foundational competency for accreditation of public health programs. The need for public health education programs informed by current public health needs and employment opportunities long noted. More than half of the jobs required bachelors and above. However, 12% of the jobs did not list the degree required for the particular job. Nineteen percent of the jobs listed indicated masters as the preferred while more than half of the jobs did not list the degree preferred. However, there are still significant gaps in workforce skills training. Studies examining students’ perspectives on required skill sets are limited. Academicians, students, employers, and people working in the health care industry have different perspectives and priorities for required informatics skills. Identifying health informatics skill sets for graduate students has always been a challenge. Health informatics programs should consider specialized tracks that include specific skills to meet the complex health care delivery and market demand, and specific training components defined for different specialties. The specific skill sets required by different employers vary owing to the increasing rate of technological developments. Results of our study showed Python, MS SQL and Tableau as one of the most common technical skills desired in these job openings. In addition, the skills needed for health informaticians vary significantly depending on the position, and health informatics students need skills pertinent to their professional experience for their future career paths [32]. Requirements use cases, and business process definitions produced by systems analysis and require expertise in the system development life cycles. Our study also showed that various job categories required requirement analysis, business processes and working in multidisciplinary teams skills. International Medical Informatics Association (IMIA) determined 3 domains and 12 learning outcomes that related to data training and skills. These learning outcomes focus on health data management principles; structure and design principles of health records; principles of data representation and analysis; ethical and security issues; nomenclatures, vocabularies, terminologies, ontologies, and taxonomies; health administration and economics; basic informatics terminology; ability to communicate electronically; and methods of practical and theoretical informatics, mathematics, biometry, and epidemiology. There are several international and national initiatives developing health informatics competencies, determining the standard comprehensive health information skill sets has always been a challenging task because of the continually evolving technology. New job opportunities for health informatics professionals require specific skill sets to utilize new cutting- edge, patient-focused delivery tools. Developing Evidence-based Population Health Informatics curriculum: Integrating competency based model and job analysis 21 Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * 13(1):e10, 2021 OJPHI The major limitation of this study is its specificity to one geographical area. While it is probable that similar results found for other urban areas, with similar demographics, other areas may have different employee needs. A larger study including wider and more representative geographical areas needed to generalize findings to broader population areas. Another limitation is that the position listings examined were from one source such as Indeed. These may not be representative of all the positions available to people with degrees in population health informatics. We are expanding the current framework by conducting analysis of the additional jobs available globally across other platforms that will help to formulate various skills needed in this rapidly evolving field of population health informatics. Population health informatics curriculum designed and developed took into account various available program specific competencies and integrated knowledge, skills and expertise based on the analysis of the job positions. We will assess the skills of the students using a wide range of methods including case study scenarios, evaluation of existing health information systems, designing and developing health information systems, and project management skills through independent project practicum. In addition, feedback from student preceptors obtained. Academicians have also been discussing the integration of skills training into the health informatics curriculum for a long time. Faculty members who work in academic institutions, health informatics departments, and programs expected to follow up on the changing requirements and update the content of their curriculum continuously. Results of this pilot study can serve as a framework to guide the development of a curriculum by the various academic units, and program directors based on this integrated approach. The framework can also guide evaluation of academic programs on an ongoing basis to identify skills that would be essential for students to acquire and make them job ready in this rapidly changing environment. References 1. Miller C, Ishikawa C, DeLeon M, Huang M, Ising A, et al. 2015. Joint Recommendations for the Public Health Informatics Infrastructure. J Public Health Manag Pract. 21(5), 516- 18. PubMed https://doi.org/10.1097/PHH.0000000000000325 2. Ritko AL, Odlum M. 2013. Gap analysis of biomedical informatics graduate education competencies. AMIA Annu Symp Proc. 2013, 1214-23. PubMed 3. Fleming ML, Parker E, Gould T, Service M. 2009. Educating the public health workforce: Issues and challenges. Aust New Zealand Health Policy. 6, 8. PubMed https://doi.org/10.1186/1743-8462-6-8 4. Gaba A, Shrivastava A, Amadi C, Joshi A. 2015. The Nutrition and Dietetics Workforce Needs Skills and Expertise in the New York Metropolitan Area. Glob J Health Sci. 8(6), 14- 24. PubMed https://doi.org/10.5539/gjhs.v8n6p14 5. Shah GH, Leider JP, Castrucci BC, Williams KS, Luo H. 2016. Characteristics of Local Health Departments Associated with Implementation of Electronic Health Records and Other Informatics Systems. Public Health Rep. 131(2), 272-82. PubMed https://doi.org/10.1177/003335491613100211 https://pubmed.ncbi.nlm.nih.gov/26214538 https://doi.org/10.1097/PHH.0000000000000325 https://pubmed.ncbi.nlm.nih.gov/24551403 https://pubmed.ncbi.nlm.nih.gov/19358714 https://doi.org/10.1186/1743-8462-6-8 https://pubmed.ncbi.nlm.nih.gov/26755482 https://doi.org/10.5539/gjhs.v8n6p14 https://pubmed.ncbi.nlm.nih.gov/26957662 https://doi.org/10.1177/003335491613100211 Developing Evidence-based Population Health Informatics curriculum: Integrating competency based model and job analysis 22 Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * 13(1):e10, 2021 OJPHI 6. Joshi A, Perin DM. Gaps in the existing public health informatics training programs: a challenge to the development of a skilled global workforce. Perspect Health Inf Manag. 2012; 9(fall):1-13. Epub 2012 Oct 1. PMID: 23209452; PMCID: PMC3510646. 7. Rosemary D. 2005. Engaging the Curriculum in Higher Education. Int Stud Sociol Educ. 15, 107-12. https://doi.org/10.1080/0962021980020157 8. Khan M, Law L. 2015. An Integrative Approach to Curriculum Development in Higher Education in the USA: A Theoretical Framework. Int Educ Stud. 8(3), 66-76. https://doi.org/10.5539/ies.v8n3p66 9. Brownson RC, Samet JM, Chavez GF, Davies MM, Galea S, et al. 2015. Charting a future for epidemiologic training. Ann Epidemiol. 25(6), 458-65. PubMed https://doi.org/10.1016/j.annepidem.2015.03.002 10. Patel SG, Keswani R, Elta G, Saini S, Menard-Katcher P, et al. 2015. Status of Competency- Based Medical Education in Endoscopy Training: A Nationwide Survey of US ACGME- Accredited Gastroenterology Training Programs. Am J Gastroenterol. 110(7), 956-62. PubMed https://doi.org/10.1038/ajg.2015.24 11. Thomas PA, Kern D, Hughes MT, Chen BY. Curriculum development for medical education: A six-step approach. John Hopkins University Press. 2015. 1-300. 12. Huang QR. 2007. Competencies for graduate curricula in health, medical and biomedical informatics: a framework. Health Informatics J. 13(2), 89-103. PubMed https://doi.org/10.1177/1460458207076465 13. Hersh W. 2010. The health information technology workforce: estimations of demands and a framework for requirements. Appl Clin Inform. 1(2), 197-212. PubMed https://doi.org/10.4338/ACI-2009-11-R-0011 14. Hersh W, Wright A, Garets D, Davis M, Pilolla L. Characterizing the health information technology workforce: Analysis from the HIMSS Analytics™ database. 2008 Portland, OR: http://www billhersh info/# workforce. 15. Mantas J, Ammenwerth E, Demiris G, Hasman A, Haux R, et al. 2010. IMIA Recommendations on Education Task Force. Recommendations of the International Medical Informatics Association (IMIA) on Education in Biomedical and Health Informatics. First Revision. Methods Inf Med. 49(2), 105-20. PubMed https://doi.org/10.3414/ME5119 16. Jogerst K, Callender B, Adams V, Evert J, Fields E, et al. 2015. Identifying interprofessional global health competencies for 21st-century health professionals. Ann Glob Health. 81(2), 239-47. PubMed https://doi.org/10.1016/j.aogh.2015.03.006 17. Tabak RG, Padek MM, Kerner JF, Stange KC, Proctor EK, et al. 2017. Dissemination and Implementation Science Training Needs: Insights from Practitioners and Researchers. Am J Prev Med. 52(3) (Suppl 3), S322-29. PubMed https://doi.org/10.1016/j.amepre.2016.10.005 https://doi.org/10.1080/0962021980020157 https://doi.org/10.5539/ies.v8n3p66 https://pubmed.ncbi.nlm.nih.gov/25976024 https://doi.org/10.1016/j.annepidem.2015.03.002 https://pubmed.ncbi.nlm.nih.gov/25803401 https://pubmed.ncbi.nlm.nih.gov/25803401 https://doi.org/10.1038/ajg.2015.24 https://pubmed.ncbi.nlm.nih.gov/17510222 https://doi.org/10.1177/1460458207076465 https://pubmed.ncbi.nlm.nih.gov/23616836 https://doi.org/10.4338/ACI-2009-11-R-0011 https://pubmed.ncbi.nlm.nih.gov/20054502 https://doi.org/10.3414/ME5119 https://pubmed.ncbi.nlm.nih.gov/26088089 https://doi.org/10.1016/j.aogh.2015.03.006 https://pubmed.ncbi.nlm.nih.gov/28215389 https://doi.org/10.1016/j.amepre.2016.10.005 Developing Evidence-based Population Health Informatics curriculum: Integrating competency based model and job analysis 23 Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * 13(1):e10, 2021 OJPHI 18. Pruitt Z, Mhaskar R, Kane BG, Barraco RD, DeWaay DJ, et al. 2017. Development of a health care systems curriculum. Adv Med Educ Pract. 8, 745-53. PubMed https://doi.org/10.2147/AMEP.S146670 19. Gonzalo JD, Haidet P, Blatt B, Wolpaw DR. 2016. Exploring challenges in implementing a health systems science curriculum: a qualitative analysis of student perceptions. Med Educ. 50(5), 523-31. PubMed https://doi.org/10.1111/medu.12957 20. Li S, Bamidis PD, Konstantinidis ST, Traver V, Car J, et al. 2019. Setting priorities for EU healthcare workforce IT skills competence improvement. Health Informatics J. 25(1), 174- 85. PubMed https://doi.org/10.1177/1460458217704257 21. Fridsma DB. 2017. Developing the health informatics workforce of the future: academic and industry partners. J Am Med Inform Assoc. 24(3), 677-78. PubMed https://doi.org/10.1093/jamia/ocx027 22. LaVenture M, Brand B, Baker EL. 2017. Developing an Informatics-Savvy Health Department: From Discrete Projects to a Coordinating Program Part II: Creating a Skilled Workforce. J Public Health Manag Pract. 23(6), 638-40. PubMed https://doi.org/10.1097/PHH.0000000000000658 23. Gibson PJ, Shah GH, Streichert LC, Verchick L. Urgent Challenges for Local Public Health Informatics. J Public Health Manag Pract. 2016 Nov-Dec; 22 Suppl 6, Public Health Informatics (Suppl 6):S6-S8. 24. Kharrazi H, Lasser EC, Yasnoff WA, Loonsk J, Advani A, et al. 2017. A proposed national research and development agenda for population health informatics: summary recommendations from a national expert workshop. J Am Med Inform Assoc. 24(1), 2-12. Epub Mar 2016. doi:https://doi.org/10.1093/jamia/ocv210. PubMed 25. Whetton S, Hazlitt C. 2015. Educating the Health Informatics Professional: The Impact of an Academic Program. Stud Health Technol Inform. 214, 159-66. PubMed 26. Polner E. The 10 Best Job Search Websites of 2020 [Internet]. The Balance Careers. 2020 [cited 2020Sep10]. Available from: https://www.thebalancecareers.com/top-best-job- websites-2064080 27. Elo S, Kyngäs H. 2008. The qualitative content analysis process. J Adv Nurs. 62(1), 107-15. PubMed https://doi.org/10.1111/j.1365-2648.2007.04569.x 28. Meyer MA. 2019. Healthcare data scientist qualifications, skills, and job focus: a content analysis of job postings. J Am Med Inform Assoc. 26(5), 383-91. PubMed https://doi.org/10.1093/jamia/ocy181 29. Council on Education for Public Health. Accreditation Criteria Schools of Public Health & Public Health Programs. Silver Spring, MD: Council on Education for Public Health (2016). https://pubmed.ncbi.nlm.nih.gov/29138614 https://doi.org/10.2147/AMEP.S146670 https://pubmed.ncbi.nlm.nih.gov/27072441 https://doi.org/10.1111/medu.12957 https://pubmed.ncbi.nlm.nih.gov/28441906 https://doi.org/10.1177/1460458217704257 https://pubmed.ncbi.nlm.nih.gov/28403380 https://doi.org/10.1093/jamia/ocx027 https://pubmed.ncbi.nlm.nih.gov/28957904 https://doi.org/10.1097/PHH.0000000000000658 https://doi.org/10.1093/jamia/ocv210 https://pubmed.ncbi.nlm.nih.gov/27018264 https://pubmed.ncbi.nlm.nih.gov/26210434 https://pubmed.ncbi.nlm.nih.gov/18352969 https://pubmed.ncbi.nlm.nih.gov/18352969 https://doi.org/10.1111/j.1365-2648.2007.04569.x https://pubmed.ncbi.nlm.nih.gov/30830169 https://doi.org/10.1093/jamia/ocy181 Developing Evidence-based Population Health Informatics curriculum: Integrating competency based model and job analysis 24 Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * 13(1):e10, 2021 OJPHI 30. Gamache R, Kharrazi H, Weiner JP. 2018. Public and Population Health Informatics: The Bridging of Big Data to Benefit Communities. Yearb Med Inform. 27(1), 199-206. PubMed https://doi.org/10.1055/s-0038-1667081 31. US Department of Health and Human Services (HHS) and Centers for Disease Control and Prevention. (CDC) and University of Washington School of Public Health and Community Medicine’s Center for Public Health Informatics. Competencies for Public Health Informaticians 2009. Atlanta: U.S. Department of Health and Human Services, Centers for Disease Control and Prevention; (2009) 32. Kampov-Polevoi J, Hemminger BM. 2011. A curricula-based comparison of biomedical and health informatics programs in the USA. J Am Med Inform Assoc. 18(2), 195-202. PubMed https://doi.org/10.1136/jamia.2010.004259 33. Wholey DR, LaVenture M, Rajamani S, Kreiger R, Hedberg C, et al. 2018. Developing Workforce Capacity in Public Health Informatics: Core Competencies and Curriculum Design. Front Public Health. 6, 124. PubMed https://doi.org/10.3389/fpubh.2018.00124 34. Hsu CE, Dunn K, Juo HH, Danko R, Johnson D, et al. 2012. Understanding public health informatics competencies for mid-tier public health practitioners—a web-based survey. Health Informatics J. 18(1), 66-76. PubMed https://doi.org/10.1177/1460458211424000 35. Sapci AH, Sapci HA. 2020. Teaching Hands-On Informatics Skills to Future Health Informaticians: A Competency Framework Proposal and Analysis of Health Care Informatics Curricula. JMIR Med Inform. 8(1), e15748. PubMed https://doi.org/10.2196/15748 https://pubmed.ncbi.nlm.nih.gov/30157524 https://doi.org/10.1055/s-0038-1667081 https://pubmed.ncbi.nlm.nih.gov/21292707 https://doi.org/10.1136/jamia.2010.004259 https://pubmed.ncbi.nlm.nih.gov/29770321 https://doi.org/10.3389/fpubh.2018.00124 https://pubmed.ncbi.nlm.nih.gov/22447878 https://doi.org/10.1177/1460458211424000 https://pubmed.ncbi.nlm.nih.gov/31961328 https://doi.org/10.2196/15748