International Journal of Interactive Mobile Technologies (iJIM) – eISSN: 1865-7923 – Vol. 13, No. 11, 2019 Paper—Usability Evaluation of Mobile Health Application from AI Perspecti in Rural Areas of ... Usability Evaluation of Mobile Health Application from AI Perspective in Rural Areas of Pakistan https://doi.org/10.3991/ijim.v13i11.11513 Abdul Samad Dahri (*), Ahmed Al-Athwari Universiti Utara Malaysia, Kedah, Malaysia Dahriabdulsamad@gmail.com Azham Hussain UUM School of Computing, Kedah, Malaysia Abstract—Purpose: This study endorses AI enabled Mobile Health appli- cation and investigates usability evaluation of the Mobile Health application by patients’ task performance evaluation and satisfaction. Materials and Methods: International Organization for Standards (ISO) 9241-11 standard metrics were used and 15 patients performed tasks on task success rate, errors, efficiency (time spent), satisfaction (SUS scale). Results: Getting registered was a top easy task while finding a relevant doc- tor was the most difficult task for users. The satisfaction scores by SUS suggest good rather excellent application user experience. Male were successful task achievers, while educational level and mobile know-how influence the usability scores in terms of time consumed, task errors occurred, and task completed. Conclusion: Methods used in this study suggest future research from differ- ent contexts. Using ISO 9241-11 usability standards, the SUS instrument for satisfaction, and measuring user characteristics influence performance and can provide considerable Mobile Health design. Keywords—Healthcare, Mobile Health (mHealth), Usability, Artificial intelli- gence, Developing countries. 1 Introduction 1.1 Healthcare-Related Developing Countries Issues Healthcare not only provides economic expansion potential but also the basic needs of the country (Mahmud &Parkhurst, 2007). Over a decade global healthcare grabbed attention assembling institutions and organizations to deliver better healthcare ser- vices that meet population needs in developing nations (Mills, 2014). The WHO rec- ommends 5% of their GDP threshold to invest on healthcare to achieve set targets (William, 2003).However, this remains a dream for developing nations particularly, African and Asian countries which spend less than 5% of GDP on healthcare (Cas- sels&Janovsky, 1998).According to UN (2018) reported about 90% of the global iJIM ‒ Vol. 13, No. 11, 2019 213 https://doi.org/10.3991/ijim.v13i11.11513 https://doi.org/10.3991/ijim.v13i11.11513 https://doi.org/10.3991/ijim.v13i11.11513 Paper—Usability Evaluation of Mobile Health Application from AI Perspecti in Rural Areas of ... population lives in rural areas, and this number will reach its peak by 2020. Where, life expectancy is worse, poverty, limited access to quality healthcare facilities, lack of trained healthcare workers, transport difficulty and so on, contributing to low quali- ty of healthcare among rural population (Strasser, Kam, &Regalado, 2016). Besides that, skyrocketing costs, high price drugs, and hospital-acquired infection, and failure of care delivery leads to adverse healthcare events (Neill, 2013). Therefore, the im- mediate innovative intervention is needed in rural areas of developing countries. For the past few years from a massive computer system to low-cost tablet, technol- ogy has decentralized computational capabilities and modification in healthcare sys- tem architect (Lanzola, Gatti, Falasconi, &Stefanelli, 1999). Moreover, in the 20th century, doctors operate with rare and expensive devices that allow ease with the accurate characterization of disease, such as heavy technology and specialist provid- ers (Koop, Mosher, Kun, Geiling, Grigg, Long, & Rosen, 2008). These advances heavily initiated the inclusion of Artificial Intelligence (AI) agents to enhance the predictiveness in healthcare workflow (Bui, 2000). In healthcare transaction flow between doctor and patient, where, the advantages of AI have broadly triumphed in the medical literature (Jha&Topol, 2016).An AI system uses sophisticated algorithm to ‘Learn’ and extracts useful information from a large patient population to assist in making real-time inferences for health outcome predic- tion (Neill, 2013). Moreover, the medical information volume is pilling twice in every three years and the estimated reading work by a physician is 29 hours to remain up-to- date completely which is not possible (Curioni-Fontecedro, 2017). This enforcesthe use of AI techniques to analyse data into information to improve quality and lower the cost of patient care (Neill, 2013) to assist in the clinical practices. There are more than 97000 AI enabled mobile healthcare applications(mHa) avail- able on Google Play and Apple’s App which would be downloaded by 500 million people till 2015 (Jahn& Houck, 2013-2017) while, 50% of these appswere estimated to be downloaded on smartphones by 2017 (Siltala, 2013).This phenomenon has turned smartphones into medical kits for real-time health monitoring of patient’s ac- tivities, early predictability,disease screening, improved medication adherence (Alemdar&Ersoy, 2010)by medical professionals and reduce diagnostic errors that are inevitable in human clinical practices (Pearson, 2011). Advantageously, over 85% of global population is under umbrella of commercial wireless signals (WHO, 2011) and 80% of them use the internet on smartphones (Dave, 2018), and developing countries like Pakistan has mobile user base above 90% with 2G internet coverage (PTA annual report 2014-2015).Keeping this impact, a satellite-based e-medicine initiative with USA training support to 45 doctor/ nurses in Sindh with SUPARCO, the space agency of Pakistan, was launched, limited to Kara- chi and Shikarpur (Malik, 2007). Though all these initiatives ended in ashes yet, pene- tration of cheaper access to mobile technology is new breath to healthcare system in Pakistan through mHealth apps for cost-efficient, effective, and quality healthcare services.Therefore, by AI-powered mHealth apps holds the future for efficient quality healthcare services, particularly, in remote areas of developing countries like Paki- stan. 214 http://www.i-jim.org Paper—Usability Evaluation of Mobile Health Application from AI Perspecti in Rural Areas of ... The past research comprises heavily of a useful literature review or surveys rather ground studies compared to its western counterparts. Therefore, there is dire need to conduct field investigation on mHealth app usability (Ozdalga, Ozdalga, &Ahuja, 2012), as a higher percentage of studies lack empirical evidence of field validation for applications (Insfran& Fernandez, 2008). To serve the field-testing urge, mobile usa- bility tests are best suitable to best understand the usage of smartphone technology. Accordingly, mobile application usability is considered a quality feature that indi- cates how this product enables users to learn and use without difficultyand actual performance of the application. Thus, usability evaluation has become vital for the smartphones as well to prevent the application being difficult to use (Coursaris& Kim, 2011) and indeed it determines the success of that application (Baharuddin, Singh, &Razali, 2013). Usability, in other words, is the facility with which users can use a technology artefact to achieve a specific goal (Coursaris& Kim, 2011) with the key feature are efficiency, effectiveness, and satisfaction, according to ISO 9241– 11. Importantly, despite the mHealth system popularity, 95% of applications are no tested (Furlow, 2012). Therefore, prior to the trial of mHealth technology, it is neces- sary for designers to consider the usability of these technologies (Brown III, Yen, Rojas, &Schnall, 2013). Surprisingly, most of the advancements are exclusive of the patient’s interface (Heathfield& Wyatt, 1993). Where, often the patient complaint is rectified rather patient treatment (Coiera, 1997). Therefore, patients’ involvement is important to evaluate and attempt to improve healthcare quality (Kim, Trace, Mey- ers,& Evens,1997) including greater patient satisfaction, increased adherence to treatment, and positive treatment outcomes (Tennstedt, 2000).As a result, further investigation is needed to ensure the appropriate designing of mobile healthcare tech- nologies before considering them for health interventions (Wolf, Moreau, Akilov, Patton, English, &Ho et al., 2013). 1.2 Measuring Usability ISO 9241-11 provides quantitative usability design (see fig: 1) that outlines metrics for the user with steps by user-centred-design (UCD) which provides user hands-on procedure. Typical task performance is most common usability test to measures ade- quate effectiveness: to what extent the user achieves desired goals, efficiency: the level of effort and resource usage put in by the user in relation to accuracy and com- pleteness, and satisfaction: association or discontent experienced by the user while performing the task. at an acceptable usability level (ISO, 1998, Jaja, Pares-Avila, &Wolpin, 2010). iJIM ‒ Vol. 13, No. 11, 2019 215 Paper—Usability Evaluation of Mobile Health Application from AI Perspecti in Rural Areas of ... Fig. 1. Fig. 1.ISO 9241-11 Usability framework Effectiveness and efficiency are measured by task completion and counting num- bers errors in an attempt to interact with the application. While efficiency is measured by efforts and resource put by the user deemed to achieve the task. Whereas, satisfac- tion is measured objectively with available instruments.Such as the System Usability Scale (SUS). Developed and designed by Brooke, the SUS encompasses 10-items on 0-4 range Likert scale (Brooke, 1996), typically administered after user application interaction to record the experience. This study used SUS for validity, reliability, and sensitivity scores ranging from 0-100. The score of 50or below is poor, above 70 is considered as good, and a score of 85 or above is excellent usability score (Bangor, Kortum, & Miller, 2009). 2 Materials and Methods Inclusion criteria for users in this study were patients with different types of sea- sonal diseases such as cold, cough, chest congestion etc, adults above 18 years of age, some knowledge of computer and mobile phone use, with at least read, listen, and speak English, own smartphone, and have no interaction with mHealth application before this study. After performing tasks, participants were requested to respond few scheduled questions at nearby suitable place or room within premises of conveniently selected hospital of Sindh province, Pakistan. 2.1 System Description The ‘’Pharmapedia Pakistan’’ by pharma developers is the top mobile healthcare app by Google play downloaded by users in Pakistan. However, this application fea- tures medicine details from chemical compound to alternate names and availability to the users. Due to the limited scope of ’Pharmapedia Pakistan’’, ‘’Mytabeeb’’ mHealth application is currently downloaded by 3000 doctors in 60 hospitals across Pakistan since January 2016 (Farah, 2017). Moreover, it features doctor and patient interaction 216 http://www.i-jim.org Paper—Usability Evaluation of Mobile Health Application from AI Perspecti in Rural Areas of ... (see figure 2&3 as a sample). This app makes a list of recommended good doctors to access any time. Yet, developers are working for ratings on doctor efficiency and database that manages the efficiency of this app as well which makes this app sound for usability evaluation in this study. Fig. 2. Main Menu Fig. 3. Doctor Selection Portal iJIM ‒ Vol. 13, No. 11, 2019 217 Paper—Usability Evaluation of Mobile Health Application from AI Perspecti in Rural Areas of ... 2.2 User Assessment Tasks, instruments, and Measures The tasks are given to participants included: • Getting registered with the system • Identifying the target disease of patient in the application • Specifying the illness/ problem • Searching for related doctor • Setting reminders as per doctor recommendations • Setting appointment reminders with doctor The tasks were based on the real case scenario as to how patients would interact with the system in a real-life situation and were validated by healthcare professionals as well. Followed by instrument measuring their age, gender, educational background, and their experience. The ISO measures were the guidelines, as effectiveness was measured by: • User was able to complete the task without help • Completed the task with some difficulty or help • User failed to complete the task even after help An error was coded that user fails to solve to complete the task. Efficiency was measured by averaging the time taken by the individual users to complete each task from start to till exiting the app by the user. Satisfaction was measured by using SUS and scores were calculated following Brooke’s guidelines. Such that, 1 point was deducted from item 1,3,53,7,9, and 5 points from 2,4,6,8, 10 items respectively. The user application interaction process began after finishing demographic characters. The collected data from 15 users then proceeded for data analysis and results were calcu- lated using Microsoft Excel spreadsheet. Descriptive statistics such as means and standard deviations were calculated in SPSS version 24. 3 Results The 15 patient users shared different demographic characteristics. 8 patients were female and 7 were male. Most (70%) of them were middle age adults (30-39 years), 30% were between 20-29 years of age, 70% had a college education, while, 90% were familiar with computer and mobile usage. 60% of patients were diagnosed with the 3- month-chronic disease while remaining were affected less than 3 months. 3.1 Effectiveness Task 5 and 6 were difficult with 30% and 40% failure rates respectively. Task 1, 2, 3, 5, and 6 were completed easily without errors. Task 4 was the most difficult to complete and accumulated the largest errors. The kinds of errors that occurred were difficult to remember steps by patients, seemingly similar options, specifically, (4) selecting the related doctor. 218 http://www.i-jim.org Paper—Usability Evaluation of Mobile Health Application from AI Perspecti in Rural Areas of ... 3.2 Efficiency As may be seen in Table 1, task 3 and 4 consumed the longest amount of time, as might be expected given the difficulties with task success and errors mentioned above. On the other hand, Tasks 2 and 5 took the shortest times. Tasks of specifying the illness/ problem and searching for related doctor (3 and 4) had mean scores 2–3 times as long as those related to interpreting values. As from table 1 most of the time consumed on task (3) and (4), while, task (5) took the shortest time. The mean scores for (2) and (6) were 2-3 times as long as those related to getting registered. Table 1. Time on task per task Task 1 Task 2 Task 3 Task 4 Task 5 Task 6 Time per task (min) Mean (SD) 1.09 (0.70) 1.99 (0.69) 4.45 (1.67) 3.68 (1.76) 1.36 (0.39) 2.21 (0.73) Range 0.94–2.38 0.91–2.50 1.56–6.47 1.39–6.89 0.89–2.01 1.27–3.77 3.3 Satisfaction The average SUS score for the group was 85.5 (SD 12) indicating good satisfaction across these mHealth application users as seen in figure 4. However, a wide variation in the score from low of 61.5 and a high score of 98.5 (at a 37-point range).While the highest ranged from 78.5 to 98.5 good or excellent (30% of the patient sample) to the lowest from 61.5 to 75 to minimally acceptable (30% of the patient sample). Fig. 4. Overview of modified SUS 3.4 Descriptive and Usability Metrics User characteristics and objective data were assessed, additional insight is observed in table 2. Descriptive statistics indicate a different across gender, age, and patient experiences. Males in this sample had higher average success, lower error rates, and higher mean SUS scores than female. The younger patient also had higher average iJIM ‒ Vol. 13, No. 11, 2019 219 Paper—Usability Evaluation of Mobile Health Application from AI Perspecti in Rural Areas of ... task completion rates on tasks, only one error average on tasks, lower task completion times, and higher mean satisfaction scores. Patients with a recent disease had a higher mean satisfaction score, task success rates, fewer errors, short average time on tasks, and higher SUS score. Education seems to have no influence on satisfaction scores, while mobile familiarity added fewer errors to complete tasks. Table 2. Comparison between user characteristics and usability metrics User characteristics Success rate, mean (SD) Error rate, mean (SD) Time on task, mean (SD) Satisfaction, mean (SD) Gender Male 95.05 (10.50) 1.25 (2.63) 1.05 (0.30) 85.31 (2.50) Female 85.58 (11.92) 3.01 (2.71) 2.01 (0.45) 75.09 (13.08) Age (years) 20–29 100 (0.00) 1.00 (1.00) 1.50 (0.25) 89.65 (8.44) 30–39 90.50 (12.55) 4.40 (2.50) 2.45 (0.75) 75.40 (12.50) Disease (months) 0–1 100 (0.00) 1.05 (1.09) 1. 70 (0.30) 85.10 (6.07) 3 85.42 (12.29) 4.75 (2.78) 2.90 (0.80) 78.33 (11.75) Education High school 93.75 (8.84) 3.40 (0.69) 2.31 (0.20) 80.25 (11.37) College/university 90.63 (12.94) 3.65 (3.77) 2.22 (0.87) 85.35 (12.53) Comp/ mobile Experi- ence Less experienced 85.50 (16.04) 6.35 (3.54) 2.76 (0.20) 65.86 (19.14) More experienced 92.10 (11.50) 2.60 (2.01) 2.10 (0.52) 89.52 (10.48) Sample (n 10) 91.05 (11.80) 3.45 (2.73) 2.35 (0.65) 80.50 (11.85) 4 Discussion This study demonstrates Usability Evaluation Model for the application and how researchers may consider relevant characteristics during mHealth application interac- tion. These recommendations are in line with growing mHealth usage rates (El-Gayar, Timsin, &Nawar, 2013). This study depicts the deployment of ISO standards to eval- uate application usability in terms of effectiveness, efficiency, and patient satisfaction. The inclusive set of factors allow in-depth understanding of application usability by the user, their task, and their performance interaction requirements for health applica- tion. Moreover, this study shows how patient’s characteristics may influence interac- tion performance and how developers might enhance eHealth and mHealth applica- tions on practical grounds. 4.1 Interpretation of Task Performance Results, Satisfaction, and Demographic Trends Findings of this study show that task 4 (selecting relevant doctor) and 6 (setting appointment reminders with doctor) had errors, difficult, and were time-consuming to 220 http://www.i-jim.org Paper—Usability Evaluation of Mobile Health Application from AI Perspecti in Rural Areas of ... user practice. These outcomes may be due to the steps involved and limited options due to application comprehensiveness. As compared to the task (1 registration) that involves one easy step and most registration, in general, it has become a routine task over internet portals in contemporary life settings. The overall satisfaction results were good for application usability rather excellent. These results indicate areas that need to be improved by developers. For example, multiple steps, repeated infor- mation, limited or similar options, and exact selection of keywords was difficult as in task 2 (identifying the target disease of the patient in the application) and task 4 (searching for related doctor). Demography of the sample reveals that male were more aware than female re- spondents and performed with slightly better SUS scores. Similarly, younger patients also performed faster with few errors. This directs that developers must consider the demographic characters for more user-friendly interaction that will benefit a wide range of users. As results depict that user at ease with application achieved tasks with fewer errors, faster, and successfully completed the task which also increases user satisfaction. 4.2 Contributions to the Literature To our knowledge, it is the first usability study on the mHealth application in the healthcare system in Sindh province of Pakistan in terms of assessing usability effec- tiveness, efficiency, and satisfaction by using validated measure. Additionally, the study utilized ISO designed usability standards and SUS instrument to compare usa- bility metric performance outcomes to pertinent patient user characteristics. This study addressed the recommended gap of usability studies on patient product interac- tion by scholars (e.g., Mulvaney, Ritterband, &Bosslet, 2011; El-Gayar, Timsin, &Nawar, 2013). Further, addressed the need to explore the demographic characteristic and technology influence on usability performance scores to design future interven- tions for targeted populations (e.g., Coursaris& Kim, 2011; Or, & Tao, 2012). Also, addressed a recommendation to measure effectiveness, efficiency, and satisfaction (Lyles, Sarkar, & Osborn, 2014). In the past authors focused on negative outcomes of usability findings, this study depicted positive mHealth usability application. This study followed valid ISO standard methodology to explain the core usability issues that need to be addressed and provides specific technique on which scholars can capi- talize to evaluate and improve mHealth usability application in the healthcare system in the Asian context. 5 Conclusion The study findings serve as an exemplar for Usability Evaluation Model with good perceived usability satisfaction, and nearly 1/ 3rd of users rated at poor usability of the application. The results show objective data for developers and directly needed corrections. The objectivity of the results shows variation in the users that indicate developer and user mismatch that needs to be addressed. iJIM ‒ Vol. 13, No. 11, 2019 221 Paper—Usability Evaluation of Mobile Health Application from AI Perspecti in Rural Areas of ... This study utilizes a systematic quantitative approach by considering the different needs of the user who interacted with the mHealth application. It also reveals the practicality of the performance of different kinds of assessment measures. The study used the ISO standard method to measure effectiveness, efficiency, and satisfaction with validated tools such as the SUS instrument. Together, these application usability measures provide better understanding and serve as an exemplar for the methodologi- cal approaches by designers and researchers. Literature gaps were addressed in this study by examining different characters among patients with technological know-how which provides feasibility for the de- veloper to improve the design of mHealth application for better user usability on prac- tical grounds. Moreover, results also indicate that socio-geographical, and personal features may influence the user experience. Therefore, in a wider perspective and applicability these results would play influencing role on mHealth application usabil- ity and developmental experience in future. 6 References [1] Alemdar, H., & Ersoy, C. (2010). Wireless sensor networks for healthcare: A sur- vey. Computer networks, 54(15), 2688-2710. https://doi.org/10.1016/j.comnet.2010.05.003 [2] Top App Store apps in Pakistan | Medical – SimilarWeb (Access May 5, 2019).https://www.similarweb.com/apps/top/google/store-rank/pk/medical/top-free [3] Baharuddin, R., Singh, D., &Razali, R. (2013). Usability dimensions for mobile applica- tions-a review. Res. J. Appl. Sci. Eng. Technol, 5(6), 2225-2231. [4] Bangor, A., Kortum, P., & Miller, J. (2009). Determining what individual SUS scores mean: Adding an adjective rating scale. Journal of usability studies, 4(3), 114-123. [5] Brown III, W., Yen, P. Y., Rojas, M., &Schnall, R. (2013). Assessment of the Health IT Usability Evaluation Model (Health-ITUEM) for evaluating mobile health (mHealth) tech- nology. Journal of biomedical informatics, 46(6), 1080-1087. https://doi.org/10.1016/j.jbi. 2013.08.001 [6] Bui, T. (2000). Building agent-based corporate information systems: An application to tel- emedicine. European Journal of Operational Research, 122(2), 242-257. https://doi.org/ 10.1016/s0377-2217(99)00231-3 [7] Cassels, A., &Janovsky, K. (1998). Better health in developing countries: are sector-wide approaches the way of the future? The Lancet, 352(9142), 1777-1779. https://doi.org/10.10 16/s0140-6736(98)05350-1 [8] Coiera, E. (1997). Guide to medical informatics, the internet and telemedicine. Chapman & Hall, Ltd. [9] Coursaris, C. K., & Kim, D. J. (2011). A meta-analytical review of empirical mobile usa- bility studies. Journal of usability studies, 6(3), 117-171. [10] Curioni-Fontecedro, A. (2017). A new era of oncology through artificial intelligence. [11] Darcy, A. M., Louie, A. K., & Roberts, L. W. (2016). Machine learning and the profession of medicine. Jama, 315(6), 551-552. https://doi.org/10.1001/jama.2015.18421 [12] Dave Chaffey (11 Jul, 2018) “Mobile marketing statistics compilation’. 05-June-2016.” [Online]. Available: http://www.smartinsights.com/mobile-marketing/ mobile-marketing- analytics/mobile-marketing-statistics/ https://doi.org/10.15358/9783800653140-121 222 http://www.i-jim.org https://doi.org/10.1016/j.comnet.2010.05.003 https://doi.org/10.1016/j.comnet.2010.05.003 https://www.similarweb.com/apps/top/apple/store-rank/pk/medical/top-free/iphone https://www.similarweb.com/apps/top/apple/store-rank/pk/medical/top-free/iphone https://www.similarweb.com/apps/top/google/store-rank/pk/medical/top-free https://www.similarweb.com/apps/top/google/store-rank/pk/medical/top-free https://doi.org/10.1016/j.jbi.2013.08.001 https://doi.org/10.1016/j.jbi.2013.08.001 https://doi.org/10.1016/s0377-2217(99)00231-3 https://doi.org/10.1016/s0377-2217(99)00231-3 https://doi.org/10.1016/s0140-6736(98)05350-1 https://doi.org/10.1016/s0140-6736(98)05350-1 https://doi.org/10.1001/jama.2015.18421 https://doi.org/10.1001/jama.2015.18421 https://doi.org/10.15358/9783800653140-121 Paper—Usability Evaluation of Mobile Health Application from AI Perspecti in Rural Areas of ... [13] El-Gayar, O., Timsina, P., Nawar, N., & Eid, W. (2013). Mobile applications for diabetes self-management: status and potential. Journal of diabetes science and technology, 7(1), 247-262. https://doi.org/10.1177/193229681300700130 [14] Farah Saleem (2017). Top 5 Healthcare and Medical Apps for Pakistani Patients. (access May 5, 2019). https://propakistani.pk/2016/03/15/top-5-healthcare-and-medical-apps-for- pakistani-patients/ [15] Furlow, B. (2012). mHealth apps may make chronic disease management easier. Retrieved January, 10, 2013. [16] Heathfield, H. A., & Wyatt, J. (1993). Philosophies for the design and development of clinical decision-support systems. Methods of information in medicine, 32(01), 01-08. https://doi.org/10.1055/s-0038-1634896 [17] International Organization for Standardization. Ergonomic requirements for office work with visual display terminals (VDTs) Part 11 Guidance on usability. International standard, 9241-11. Geneve: ISO; 1998. https://doi.org/10.3403/01822507u [18] Insfran, E., & Fernandez, A. (2008, September). A systematic review of usability evalua- tion in web development. In International Conference on Web Information Systems Engi- neering (pp. 81-91). Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540- 85200-1_10 [19] ISO/IEC 25010:2011 System and software quality models. [20] Jahns, R.-G., Houck, P., Mobile Health Market Report 2013–2017. http://www.research2guidance.com/shop/index.php/mobile-healthtrends-and-figures 2013– 2017. Accessed 23 Nov 2013, 2013. [21] Jaja, C., Pares-Avila, J., Wolpin, S., & Berry, D. (2010). Usability evaluation of the inter- active Personal Patient Profile-Prostate decision support system with African American men. Journal of the National Medical Association, 102(4), 290-302. https://doi.org/10.10 16/s0027-9684(15)30601-5 [22] Jha, S., &Topol, E. J. (2016). Adapting to artificial intelligence: radiologists and pathologists as information specialists. Jama, 316(22), 2353-2354. https://doi.org/10.1001/ jama.2016.17438 [23] Koop, C. E., Mosher, R., Kun, L., Geiling, J., Grigg, E., Long, S., & Rosen, J. M. (2008). Future delivery of health care: Cybercare. IEEE Engineering in Medicine and Biology Magazine, 27(6), 29-38. https://doi.org/10.1109/memb.2008.929888 [24] Kim, J., Trace, D., Meyers, K., & Evens, M. (1997). An empirical study of the Health Sta- tus Questionnaire System for use in patient-computer interaction. In Proceedings of the AMIA Annual Fall Symposium (p. 86). American Medical Informatics Association. [25] Lanzola, G., Gatti, L., Falasconi, S., &Stefanelli, M. (1999). A framework for building co- operative software agents in medical applications. Artificial intelligence in medicine, 16(3), 223-249. https://doi.org/10.1016/s0933-3657(99)00008-1 [26] Lyles, C. R., Sarkar, U., & Osborn, C. Y. (2014). Getting a technology-based diabetes in- tervention ready for prime time: a review of usability testing studies. Current diabetes re- ports, 14(10), 534. https://doi.org/10.1007/s11892-014-0534-9 [27] Mahmud, A., &Parkhurst, M. (2007). The role of the health care sector in expanding eco- nomic opportunity. Corporate Social Responsibility Initiative Report, (21). [28] Malik, A. Z. (2007, June). Telemedicine country report-Pakistan. In 2007 9th International Conference on e-Health Networking, Application and Services (pp. 90-94). IEEE. https://doi.org/10.1109/health.2007.381611 [29] Mat Siltala (Aug 20, 2013). “Is mobile healthcare the future?, (Access May 5, 2019). [Online]. Available: http://www.greatcall.com/greatcall/lp/ is-mobile-healthcare-the- future-infographic.aspx iJIM ‒ Vol. 13, No. 11, 2019 223 https://doi.org/10.15358/9783800653140-121 https://doi.org/10.1177/193229681300700130 https://doi.org/10.1177/193229681300700130 https://propakistani.pk/2016/03/15/top-5-healthcare-and-medical-apps-for-pakistani-patients/ https://propakistani.pk/2016/03/15/top-5-healthcare-and-medical-apps-for-pakistani-patients/ https://propakistani.pk/2016/03/15/top-5-healthcare-and-medical-apps-for-pakistani-patients/ https://doi.org/10.1055/s-0038-1634896 https://doi.org/10.1055/s-0038-1634896 https://doi.org/10.3403/01822507u https://doi.org/10.3403/01822507u https://doi.org/10.1007/978-3-540-85200-1_10 https://doi.org/10.1007/978-3-540-85200-1_10 http://www.research2guidance.com/shop/index.php/mobile-healthtrends-and-figures 2013�2017 http://www.research2guidance.com/shop/index.php/mobile-healthtrends-and-figures 2013�2017 https://doi.org/10.1016/s0027-9684(15)30601-5 https://doi.org/10.1016/s0027-9684(15)30601-5 https://doi.org/10.1001/jama.2016.17438 https://doi.org/10.1001/jama.2016.17438 https://doi.org/10.1109/memb.2008.929888 https://doi.org/10.1109/memb.2008.929888 https://doi.org/10.1016/s0933-3657(99)00008-1 https://doi.org/10.1016/s0933-3657(99)00008-1 https://doi.org/10.1007/s11892-014-0534-9 https://doi.org/10.1007/s11892-014-0534-9 https://doi.org/10.1109/health.2007.381611 https://doi.org/10.1109/health.2007.381611 Paper—Usability Evaluation of Mobile Health Application from AI Perspecti in Rural Areas of ... [30] Mills, A. (2014). Health care systems in low-and-middle-income countries. New England Journal of Medicine, 370(6), 552-557 [31] Mulvaney, S. A., Ritterband, L. M., &Bosslet, L. (2011). Mobile intervention design in di- abetes: review and recommendations. Current diabetes reports, 11(6), 486. https://doi.org/ 10.1007/s11892-011-0230-y [32] Neill, D. B. (2013). Using artificial intelligence to improve hospital inpatient care. IEEE Intelligent Systems, 28(2), 92-95. https://doi.org/10.1109/mis.2013.51 [33] Or, C., & Tao, D. (2012). Usability study of a computer-based self-management system for older adults with chronic diseases. JMIR research protocols, 1(2), e13. https://doi.org/10.2 196/resprot.2184 [34] Ozdalga, E., Ozdalga, A., &Ahuja, N. (2012). The smartphone in medicine: a review of current and potential use among physicians and students. Journal of medical Internet re- search, 14(5), e128. https://doi.org/10.2196/jmir.1994 [35] Pakistan Telecommunication Authority. “Pta annual report 2014-2015, (Access May 5, 2019).” [Online]. Available: http://www.pta.gov.pk/index.php?option=com content & view= article & id=2184: pta-annual-report-2015&catid=94: news updates [36] Pearson T. How to replicate Watson hardware and systems design for your own use in your basement. 2011 https://www.ibm.com/ developerworks/community/blogs/Inside SystemStorage/entry/ibm_watson_how_to_build_your_own_watson_jr_in_your_baseme nt7? (accessed May 5, 2019). https://doi.org/10.1002/9781119449409.ch2 [37] Strasser, R., Kam, S. M., &Regalado, S. M. (2016). Rural health care access and policy in developing countries. Annual review of public health, 37, 395-412. https://doi.org/10.1146/ annurev-publhealth-032315-021507 [38] The market for mHealth applications. (Access May 5, 2019) [Online]. Available: http:// research2guidance.com/2013/03/07/the-market-for-mHealth-app-services-will-reach-26- billion-by-2017/ [39] United Nations (2018). Department of Economic and Social Affairs https://www.un.org/ development/desa/en/news/population/2018-revision-of-world urbanization-prospects.html [40] WilliamSavedoff (2003). How Much Should Countries Spend on Health? https://www. who.int/health_financing/en/how_much_should_dp_03_2.pdf [41] Wolf, J. A., Moreau, J. F., Akilov, O., Patton, T., English, J. C., Ho, J., & Ferris, L. K. (2013).Diagnostic inaccuracy of smartphone applications for melanoma detection. JAMA dermatology, 149(4), 422-426. https://doi.org/10.1001/jamadermatol.2013.2382 [42] World Health Organization, mHealth: New horizons for health through mobile technolo- gies, 2011. 7 Authors Abdul Samad Dahri, is a Ph.D. in HRM and progressive publication are in differ- ent fields of research mainly related to human resource management. His current research revolves around harmonizing management with innovation technology in different sectors. Ahmed Al-Athwari is a former web developer at YemenSoft Company, Sana’a, Yemen. Currently, he is a master student at school of computing, Universiti Utara Malaysia. He is doing research on Software Engineering and IT Management. ah.alathwari20112@hotmail.com 224 http://www.i-jim.org https://doi.org/10.1007/s11892-011-0230-y https://doi.org/10.1007/s11892-011-0230-y https://doi.org/10.1109/mis.2013.51 https://doi.org/10.1109/mis.2013.51 https://doi.org/10.2196/resprot.2184 https://doi.org/10.2196/resprot.2184 https://doi.org/10.2196/jmir.1994 https://doi.org/10.2196/jmir.1994 https://doi.org/10.1002/9781119449409.ch2 https://doi.org/10.1002/9781119449409.ch2 https://doi.org/10.1146/annurev-publhealth-032315-021507 https://doi.org/10.1146/annurev-publhealth-032315-021507 http://research2guidance.com/2013/03/07/the-market-for-mHealth-app-services-will-reach-26-billion-by-2017/ http://research2guidance.com/2013/03/07/the-market-for-mHealth-app-services-will-reach-26-billion-by-2017/ http://research2guidance.com/2013/03/07/the-market-for-mHealth-app-services-will-reach-26-billion-by-2017/ https://www.un.org/development/desa/en/news/population/2018-revision-of-world urbanization-prospects.html https://www.un.org/development/desa/en/news/population/2018-revision-of-world urbanization-prospects.html https://www.un.org/development/desa/en/news/population/2018-revision-of-world urbanization-prospects.html https://doi.org/10.1001/jamadermatol.2013.2382 https://doi.org/10.1001/jamadermatol.2013.2382 Paper—Usability Evaluation of Mobile Health Application from AI Perspecti in Rural Areas of ... Azham Hussain is the Associate Professor of Software Engineering at School of Computing, Universiti Utara Malaysia, Kedah, Malaysia. He is the founder of Hu- man-Centered Computing Research Group, which is affiliated with the Software Technology Research Platform Center at School of Computing, Universiti Utara Ma- laysia. Azham Hussain is a member of the US-based Institute of Electrical and Elec- tronic Engineers (IEEE), and actively involved in both IEEE Communications and IEEE Computer societies. azham.h@uum.edu.my Article submitted 2019-08-13. Resubmitted 2019-09-26. Final acceptance 2019-09-27. Final version published as submitted by the authors. iJIM ‒ Vol. 13, No. 11, 2019 225