International Journal of Interactive Mobile Technologies (iJIM) – eISSN: 1865-7923 – Vol. 15, No. 10, 2021 Paper—Performing Usability Evaluation on Multi-Platform Based Application for Efficiency and… Performing Usability Evaluation on Multi-Platform Based Application for Efficiency, Effectiveness and Satisfaction Enhancement https://doi.org/10.3991/ijim.v15i10.20429 Nik Azlina Nik Ahmad (), Nur Iman Mardhiah Hamid Universiti Kuala Lumpur, Kuala Lumpur, Malaysia nikazlina@unikl.edu.my Anitawati Mohd Lokman Universiti Teknologi MARA, Shah Alam, Malaysia Abstract—Usability evaluation is a crucial activity to contribute to a higher standard of user experience. Without proper usability evaluation, the risk of delivering a dissatisfying product is likely to increase. This study conducted a usability evaluation on multi-platform applications according to usability attributes defined in ISO 9241-11 standard: effi- ciency, effectiveness, and satisfaction. The evaluation was conducted using the think-aloud protocol on both platforms, web and mobile ap- plications, and fifty participants. The results discovered the suitability of think-aloud protocol on multi-platform usability evaluation and re- vealed some usability issues. Therefore, to further enhance the users' satisfaction, efficiency, and effectiveness, a set of improvement rec- ommendations were proposed. Insights from this study provide a stand- ard basis on the usability evaluation procedure, which can guide the us- ability evaluation of multi-platform applications. Keywords—ISO9241, multi-platform, usability attributes, mobile usability, usability evaluation 1 Introduction Nowadays, where almost everything is digitized, technology plays a significant fundamental role in amplifying the way humankind lives their life. Technology not only advances to alleviate human beings, but it can also promote humanity to human beings if correctly embraced. This study aims to perform a usability evaluation on a multi-platform-based application that runs on the Web and Android platform to en- hance its efficiency, effectiveness, and users' satisfaction. A multi-platform applica- tion refers to an application that runs on two platforms or more, which enable each of them to send and receive updates [1]. Multi-platform development focused on optimi- zation by having the same code programmed among the versions of various platforms iJIM ‒ Vol. 15, No. 10, 2021 103 https://doi.org/10.3991/ijim.v15i10.20429 mailto:nikazlina@unikl.edu.my Paper—Performing Usability Evaluation on Multi-Platform Based Application for Efficiency and… [2]. Among the benefits of multi-platform are cost and time saving, while the mobile type device is said to be more popular as they are more usable than others [3, 4]. Mul- ti-platform applications could be between web and mobile, hybrid, or interpreted web applications [5, 6, 7]. The application was developed to support the pet boarding system in terms of res- ervation, package management, and reporting. The evaluation was conducted on both platforms; the web and mobile application by examining seven main tasks of different scenarios and collecting important attribute such as task completion (number of peo- ple completed the task), time is taken to complete on each task, errors on performing the task and satisfaction ratings given by participants. A post-evaluation questionnaire was distributed, consisting of required demographic information and application usa- bility feedback based on those main attributes: effectiveness, efficiency, and satisfac- tion level. The usability level can be identified for the application, and hidden prob- lems obstructing its usability potential can be revealed. Previous research on a pet application was conducted [8], where a mobile application was developed to assist users in finding a temporary pet adopter. The application support location detection to find the nearest pet's medical assistance using a map. The application was described as easy to use and allows its users to leave ratings. A study [9] involved a mobile app development for a shelter to manage pet lost and found and promote pet adoption on stray animals. Another study that focuses on pet adoption, which is equipped with a pet product business, was also conducted [10]. A study conducted by [11] involved creating a system to carry out animal-related tasks more efficiently in a temporary animal shelter. However, none of the pet-related studies focused on evaluating the application's usability that was implemented on multi-platform. Therefore, the study aims to identify the aspects of consumer demands on multi-platform-based application in the context of pet boarding under different user categories to discover the usability issues by validating the effectiveness, efficiency, and user satisfaction level towards the application. 2 Literature Review ISO 9241-11 standard described usability as an attribute of quality in use and as a result of perceived effectiveness, efficiency, and satisfaction [12]. The two most fre- quent usability characteristics are effectiveness (how reliably users can accomplish their objectives) and effectiveness (resources that can be utilized efficiently to accom- plish that purpose [13]. There is a strong correlation between effectiveness and effi- ciency. The simpler the interface is, the more resources can be devoted to other tasks, which leads to task performance improvement. The study has noted a significant rela- tionship between those two with user satisfaction, the beneficial outcome of pleasant interactions, including feelings of satisfaction. Assessing the usability involved meas- uring user interaction performance with the system, which involves efficiency, effec- tiveness, and error rate [14]. Since it is newly designed, the user satisfaction level is vital in how the application portrays; its first impression can affect its decision to put it into fair use. Hence, usability evaluation is an effective way to measure the applica- 104 http://www.i-jim.org Paper—Performing Usability Evaluation on Multi-Platform Based Application for Efficiency and… tion's performance and possibly search for improvement rooms. Furthermore, explor- ing various similar studies on how the usability assessment can be done helps decide to choose a suitable method. In this case, usability testing is the best approach as it is a commonly used technique that had been proven to be effective from various usabil- ity evaluation research studies as it encourages feedback from the end-user. From various usability evaluation studies, the overall usability aspect differs for each study, where each brings a refreshing perspective of its' own. Usability is a par- ticular attribute that influences the software system's consistency and should satisfy user requirements [15]. It complements user experience as usability focuses on the functional part while user experience is emphasized on the emotions stemmed from the aspects of the product, as explained in [16]. It considers user satisfaction as quality indicators and aims to improve user experience. In a way, it relates to work conducted [17], which utilized three attributes that are considered vital as a measurement of how a user can use software to complete a set of operations successfully. These attributes are efficiency, effectiveness, and user satisfaction. Besides, a mobile application's usability challenges include limited connectivity in certain areas, small screen size, small storage capability, and the unconventional input mechanism. Furthermore, [18] stated that usability should be approached from multiple points of view to become sensitized to elements that possibly affect the system usage. On the other hand, usabil- ity is characterized as how successful user concerns are addressed. [19]. Hence, in- stead of adopting the standardized usability model, many studies include attributes considered important in developing any application. For example, in the study con- ducted [20], usability is vital to increase application accuracy and reduce reaction time for several user activities. Overall, the various definition of usability stated above actually describes the advantages it brings indirectly. To summarize, there are plenty of benefits of implementing usability evaluation, such as reducing training effort, improving the product quality, and users' satisfaction. In terms of usability attributes to pay attention to, various usability evaluation stud- ies implemented standardized usability models, as presented in Table 1. Besides that, usability evaluation studies only implement attributes vital in the application-specific context without following any specific usability models. There are also usability eval- uation studies that combined selected usability models and attributes vital to the dedi- cated application. iJIM ‒ Vol. 15, No. 10, 2021 105 Paper—Performing Usability Evaluation on Multi-Platform Based Application for Efficiency and… Table 1. Variety of usability attributes No. Usability Attributes Description Source 1 Effectiveness, Efficiency and Satisfaction. The international organization of standardization came up with the ISO 9241-11 model, which consists of 3 attributes: a performance measure of completing tasks on time, efficiency in completing the task success- fully, and satisfaction as users' acceptability. [21], [22], [18], [17]. 2 Learnability, Efficiency, Memorability, Errors and Satisfaction. Jakob Nielsen model consists of five attributes, which are learnability as in easy to learn, efficiency as in easy to finish a task or navigate, memo- rability as in easy to remember or reestablish, errors as in the low rate of error detection and satisfaction as in it is pleasant to use by the user. [16], [22], [15], [23]. 3 Functionality Conformance, Efficiency, Ease of learning, Memorability, Correctness and Esthetics. Factors in the article journal include compliance with features specified as ISO software operability, efficiency or latency, ease of learning as in understandable, memorability as easy to remember, correctness in lieu to dealing with system failure, and esthetics as attractive. [19] 4 Errors, Under- standability, Accessibility. In addition to effectiveness, efficiency, and satisfaction, these additional factors were used to assess mobile applications and the visually im- paired. [24] 5 Adjustability, Inability, Relia- bility and Satis- faction. Factors include adjustability that refers to the degree of the user accept- ability to the platform used to learn, inability as in a mobile learning platform, and the concept pleased the users. Reliability in a mobile learning platform means that the platform should not perform unex- pected ways and satisfaction as in learner's satisfaction and the learner's assurance to perform tasks with any external activities via their mobile device. [21] 6 Effectiveness, Efficiency and Learnability. The researchers decided to exclude satisfaction as an attribute and absorbed learnability to ensure a more parsimonious framework. Attributes learnability refers to how much learning a system is conven- ient for users. [25] 7 Effectiveness, Learnability, Efficiency, Memorability, Errors, Cognitive Load, Satisfac- tion, and Timeli- ness. It is a combination of the PACMAD usability model developed by Harrison and a single attribute that was deemed vital in mobile learning, in which timeliness refers to whether students can receive teachers' messages quickly. PACMAN attributes such as learnability, efficiency, memorability, error, and satisfaction are absorbed from Jakob Nielsen's model, where all of them serve the same purpose. The cognitive load attribute relates to the number of perceptual abilities needed by the program when being utilized. [20] 2.1 Usability assessment From various researches on various usability evaluation, there is more than one way to conduct the assessment. Usability evaluation can be conducted using various methods such as hallway usability testing, a cheap method in which a group of ran- dom people will be selected to do a set of tasks. Users will interact with the system while thinking aloud. It is a commonly used technique that helped numerous usability evaluation research [16, 26]. The expert review involves usability experts using a set of guidelines to measure essential usability criteria, while an automated expert review is a similar procedure of expert review but in an automated way [16]. Another as- sessment method is inspection, where users act as an observer while experts do evalu- 106 http://www.i-jim.org Paper—Performing Usability Evaluation on Multi-Platform Based Application for Efficiency and… ation and testing. Inspection can be done by a heuristic evaluation, which utilizes the usability principle to implement. It is cheap or conducted by a cognitive walkthrough based on user interface evaluation by experts that consider end-user opinions [22, 23]. Cognitive walkthrough is similar to heuristic evaluation because experts do it, but it does not necessarily require a whole system. A system prototype is just adequate. However, a pluralistic walkthrough [27, 28, 29] is a method for a team to investigate the system's paper prototype with the users' focus group to discuss system interface issues [26]. [16] however, highlighted the non-face-to-face assessment by suggesting remote usability testing. This method separates test conductor and user and is catego- rized by the synchronous time such as remote application sharing software or asyn- chronous time where data will be collected from logging user's activities. Further- more, there are a different kind of data gathering techniques can be utilized to extract the outcomes of the evaluation which can either be test monitoring, direct recording, a think-aloud method which require the user to talk aloud while using the system or performing the particular task as explained in [23]. On top of that, there are also other methods of conducting usability testing, such as Metrics for Usability Standards in Computing by (MUSiC), which was introduced in 1998 [30], Software Usability Measurement Inventory (SUMI). In 1992, Diagnostic Recorder for Usability Meas- urement (DRUM) by [31] was introduced. Usability assessment has been conducted in various aspects such as counseling [32], education [20, 21, 23, 24, 25, 33], shopping [16, 17], tourism [18], Web portal [15, 19], and visual impairment study [24]. However, minimal pet-related studies concentrated on evaluating the usability in the multi-platform context. 3 Methodology The usability evaluation was conducted on both platforms, web and mobile plat- forms, measuring the efficiency, effectiveness, and satisfaction outlined in ISO 9241- 11. The usability assessment framework for a multi-platform application used in this study is presented in figure 1. Fig. 1. Usability assessment framework for multi-platform application iJIM ‒ Vol. 15, No. 10, 2021 107 Paper—Performing Usability Evaluation on Multi-Platform Based Application for Efficiency and… 3.1 Data collection The methodology involved hiring the test participants, preparing a test plan, and fi- nalizing the test scenario according to both platforms' set of tasks. Necessary technical preparation was made, which include device and site set up. Data was collected using pre-testing, post-testing questionnaire, observation technique, and think-aloud proto- col with fifty participants. A pre-testing questionnaire was used to collect demograph- ic data, whilst the post-testing questionnaire was equipped with 15 questions to identi- fy the participants' satisfaction level. The questionnaire's findings were objectively analyzed, interpreted as a mean rate of satisfaction calculation, while the outcome of the think-aloud protocol was evaluated and categorized according to the right usabil- ity category as recommended [12] in order to determine efficiency and effectiveness and provide relevant recommendations. 3.2 Usability evaluation session The session was carried out in a pet boarder's place with a total of 50 participants. The participants were randomly selected from the existing pet boarder's customers and non-customers with ages ranging from 18 to 64. An initial demographic back- ground was surveyed before proceed with the evaluation task. Participants with the following criteria were selected: a pet boarder who owns and offers boarding service, regular visitors to the pet store, a customer to pet border, and those pets. A set of tasks for the different scenario was identified for the evaluation. There were two types of scenarios representing the pet border (web-based) and customer role (mobile-based). A total of 8 task scenarios were prepared for the pet border, and 7 for the customer's role. All participants will act in different roles and access the application on both platforms with two appointed facilitators' assistance. All evaluations used the re- searchers' laptops and mobile devices. Participants were allocated 4 minutes to ac- complish each task scenario. The task scenarios for each role are portrayed in Table 2. 108 http://www.i-jim.org Paper—Performing Usability Evaluation on Multi-Platform Based Application for Efficiency and… Table 2. Evaluation task scenario on both platforms Task No. Web-Based Pet Boarder Task Scenario Mobile-Based Customer Task Scenario 1 The user clicks the signup button and will be redirected to the registration page. User input username, name, email address, system role, password, confirm password, check the term, and policy agreement checkbox. Finally, click the Register button. 2 User input username, password, and click sign in button. 3 User clicks the create announcement menu section, input the message, and click the create button. The user clicks the announcement menu section and the view list of the announce- ment. 4 User clicks the create notification menu section, input the message, and click create button. Users click the notification menu section, and the list of all notifications posted will be listed. 5 User clicks the create package menu section, input the package name, slot available, price, reward points, upload multiple pictures, and add one addon service. User clicks the package menu section whereby all packages will be listed, choose one package, click the book now button, input pet name, select start date and end date, select available addon service check- box button and click create button. 6 User click booking menu whereby all booking requests made by all customers will be listed, select one booking request and click the update button, view booking status information, select approved from the dropdown list, and click the update button. User view list of booking requests made by clicking on the booking request menu section, and the list of all booking requests made by the user is listed. 7 The user clicks the report card menu whereby all booking requests made by each customer will be listed, choose one booking request and click the create report button, input notes, upload media files by clicking the add media icon or choose file, select a picture and click the insert button. The user clicks the report card menu section whereby all report card submitted is displayed; the user can view the pet name and report date, click to view the notes button, and click to view the media files button. 8 User clicks the user menu whereby all users will be listed, select one user, click the update button, scroll down to the user status information, the approved user from the dropdown list, and click the update button. N/A For collecting numerical data, a survey has been conducted as a second evaluation. There were two sets of distributed questionnaires, consisting of the respondent back- ground information and application review from the end-user. The usability question- naires include the following: 1) I think this application is easy to use, 2) It is easy to find the information needed, 3) I think this application has a pleasant interface, 4) The font utilized by the application is pleasant, 5) This application can speed up task com- pletion, 6) This application provides error messages that inform how to solve the problems, 7) I find the steps to execute tasks in this application is simple, 8) This application provides clear and descriptive information, 9) I feel that it is easy to navi- gate through the pages, 10) This application has no broken link/menu/page, 11) The iJIM ‒ Vol. 15, No. 10, 2021 109 Paper—Performing Usability Evaluation on Multi-Platform Based Application for Efficiency and… application provides all the functions that I expected it to have, 12) This application needs to be improved, 13) I have the intention to use this application shortly, 14)I will recommend this application to others, and 15) Overall, I am satisfied with this appli- cation. 4 Results and Discussion The study was carried out by issuing questionnaires to the 50 participants after each respondent had done the set of tasks required for the usability evaluation. The purpose of choosing participants from different backgrounds is to have a broader perspective of the study. Fig. 2. Distribution of respondents by profession As shown in Figure 2, respondents were categorized according to their occupation status, working in the private sector, government servant, self-employed, retiree, and student. The highest number was those who worked in the private sector reflected the ability to own pets, requiring cost and maintenance. Task Completion Time: 100% of participants completed task 1 until four on both platforms, representing sign up, sign in, and create / view announcement. Only 41 out of 50 users can complete task 5, create a package by admin on the web (82.0%) with seven errors, while 90.0% can complete the view/apply package on mobile with four errors. No issues were reported on task 6 on mobile for viewing customer's booking request, but three errors have been identified on the web application for approve booking request scenario with a 94% completion rate. Task 7, which focuses on up- dating the report card, has encountered four errors and a 92% completion rate, while the mobile application has a 100% completion rate with zero error encountered. The final task, which is approved user registration, is only available on the web, and this task has triggered five errors with an 86.0% completion rate. This is because the user's Student 20% Government 24% Private 36% Self- Employed, 14% Retired 6%, 110 http://www.i-jim.org Paper—Performing Usability Evaluation on Multi-Platform Based Application for Efficiency and… name did not appear in the list even after several pages reloads. Task completion time is a measure of the efficiency of the app. Table 3 displays the time taken by each user per task compared to the allowed 3 minutes for the task completion—more time spent on Web-based applications than mobile applications due to a more user-friendly look at mobile. The time allocated for each task was 4 minutes. However, not all the tasks can be completed within the allocated time: tasks 3, 5, 6, 7 for web-based applications and task 5 for mobile applications. The lowest completion time was spent on task 5, which was 4.7 minutes reflecting that the application lacks efficiency. Table 3. Summary of task completion, errors, and time on task Task No. Web-Platform Mobile-Platform Task Completion Errors Average Time on Task Task Completion Errors Average Time on Task 1 50 0 2.1 min 50 0 1.6 min 2 50 0 2.4 min 50 0 2.3 min 3 50 0 3.2 min 50 0 2.9 min 4 50 0 3.0 min 50 0 2.5 min 5 41 7 4.7 min 45 4 2.7 min 6 47 3 3.5 min 50 0 3.4 min 7 46 4 4.6 min 50 0 3.1 min 8 43 5 2.4 min - - - Success rate of task completion: Task completion or success rate is a measure of an application's effectiveness in use. Table 3 shows that only tasks 1 – 5 achieved a 100% completion rate while the rest scored below 94%, which is equal to 47 partici- pants and below. Therefore, this application is not sufficient enough. The age might be the factor of this result as 3 participants were retirees with age above 55. Elderly participants took a long time getting familiar with the interface and understanding the steps even though they are smartphone users. Table 4 summarizes details on partici- pants' error and feedback, classified according to ISO 9124-11 usability attributes; efficiency, effectiveness, and satisfaction. A set of recommendations was proposed to promote the application's enhancement, leading to higher efficiency and effectiveness levels. iJIM ‒ Vol. 15, No. 10, 2021 111 Paper—Performing Usability Evaluation on Multi-Platform Based Application for Efficiency and… Table 4. Errors, Comments, and Recommendations for Improvements Attributes Errors / Comments Recommendations Efficiency Task 5 (Web) • “It takes me quite a while to figure out what to put for as addon.” Task 6 (Web) • "It is confusing because there are so many users to go through (navigating)." Task 5 (Mobile) • "I have to spend a long time as it is tricky for me to figure out how to input the date with the proper format because I kept getting an error message." • "Menu section to make a booking is label as History. Hence I am confused." Task 3 and 4 : • Distinguished the interface design for announcement and notification to show each serves a different purpose. • Create a text input for the text field whereby it gives an example of an ap- propriate message for an announcement or notification. Task 5 : • Place a pop-up containing a few sugges- tions on what kind of addons services are commonly offered in the pet industry. This will help the user to make a faster decision. • A visible pop calendar icon at the end of the text field can easily pick the dates. • Create a text input for the text field whereby it alerts the user to click the icon nearby. • Each package available on display should show written information such as any addons available or reward points. • Menu section History should be changed to Booking and contains a subsection called My History. Task 6: • Place a filter button whereby the user can utilize this function to filter out the cus- tomer's pending registration status. • It is better to provide a help needed button whereby if it is clicked, it displays a video on how to approve the customer. Effectiveness Task 5 (Web) • “I have no idea what to put for addons for the package, so I did not put any- thing extra." Task 6 (Web) • "It just shows the status, and there is just an update button instead of approv- ing button." Task 5 (Mobile) • "I cannot input the dates for booking." • “I was unaware of the aid provided by the calendar date selection.” Satisfaction Task 3 and 4 (Web) • "There is no problem, but I cannot differentiate between what to post an announcement or notification." Task 5 (Web) • "I am moderately satisfied." Task 5 (Mobile) • "The interface is too technical," "sim- ple." • The information provided on the menu section and each package is unclear and written poorly." Satisfaction: Besides evaluating the applications using the prepared tasks list, par- ticipants must complete a usability questionnaire using a five-point Likert scale, rang- ing from 1 to 5, from strongly disagree to agree strongly. The questions were to eval- uate the participants' satisfaction over the completed tasks performed earlier. Table 5 summarizes the findings. 112 http://www.i-jim.org Paper—Performing Usability Evaluation on Multi-Platform Based Application for Efficiency and… Table 5. Satisfaction feedback General Strongly Agree Agree Neutral Disagree Strongly Disagree % Agree I think this application is easy to use. 17 21 11 1 0 76% It is easy to find the information needed. 15 19 13 3 0 68% I think this application has a pleasant interface. 6 15 17 10 2 42% The font utilized by the application is pleasant for my eyes. 8 15 21 6 0 46% This application can speed up my task comple- tion 13 24 11 2 0 74% This application provides error messages that tell me how to solve the problems. 13 20 15 2 0 66% I find the steps to execute tasks in this application is simple. 9 23 16 2 0 64% This application provides clear and descriptive information. 1 20 25 2 2 42% I feel that it’s easy to navigate through the pages. 1 19 22 6 2 40% This application has no broken link/menu/page. 11 22 14 3 0 66% The application provides all the functions that I expected it to have. 13 19 16 2 0 64% This application needs to be improved. 13 12 23 2 0 50% I have the intention to use this application shortly 5 17 25 3 0 44% I will recommend this application to others. 7 15 22 6 0 44% Overall, I am satisfied with this application. 12 23 14 1 0 70% Fig. 3. Overall Mean Analysis Overall, most of the participants were satisfied with the application, with an aver- age mean of 3.68. The agreed notch counts the agree and strongly disagree score. 70% of participants agreed that they were satisfied using the application on both platforms. The highest score was on the ease of use, with 76% agreement, followed by the appli- cation's ability to speed up participants' task with 74%, while the lowest score accord- ing to participants was on the page navigability, 40%. Due to several crucial errors during the evaluation, half of the respondents thought this application needs modifica- tion. Suggestions for modification have been identified in Table 4 to increase the users' satisfaction by further adopting the formal requirements technique [34]. The use 4.08 3.92 3.28 3.5 3.96 3.88 3.78 3.32 3.22 3.82 3.86 3.72 3.48 3.46 3.92 0 2 4 6 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15M e a n R a ti n g Question Number Mean Data Value iJIM ‒ Vol. 15, No. 10, 2021 113 Paper—Performing Usability Evaluation on Multi-Platform Based Application for Efficiency and… of the think-aloud protocol was proven to help obtain a comprehensive understanding of usability issues [35] related to the user and multi-platform application interactions, where 12 crucial errors from 8 assigned tasks were successfully discussed. 5 Conclusion This study has explored the potential use of the think-aloud protocol in determining the usability of the multi-platform framework to optimize its efficiency, effectiveness, and satisfaction. This study's proposed approach offers new insights in the usability evaluation context, particularly for mobile and multi-platform applications. The eval- uation involved both platforms, web and mobile applications, and the users were re- quired to evaluate the application in both platforms using a think-aloud protocol. This study involved measuring the success rate of task completion, task completion time, task errors, participants' feedback, and a satisfaction assessment through a question- naire with the involvement of 50 participants. The results indicated that 100% task completion rate was only applied to task 1 until task 4 for both platforms, and an additional task 7 for mobile application. The remaining tasks discovered errors, and comments from participants were recorded. Therefore, this application seems to lack efficiency and effectiveness. However, satisfaction has a good score, with an average mean of 3.68. To further increase the satisfaction rate, efficiency, and effectiveness, the suggestion for improvements were made accordingly. It is believed that fixing the errors as highlighted by participants will likely increase the satisfaction, efficiency, and effectiveness of the application on both platforms. This study's limitation in- volved a biased sample that focuses on a specific pet border, and user satisfaction based on human needs behavior was not recorded. Due to a severe need for useful tools [36], future studies are expected to produce comparable outcomes after specific improvements were made on the user interface based on the results of this study [37, 38] and present the model requirements for the reliable modeling of actual user be- havior to extract usage patterns. 6 References [1] Nik Ahmad, N. A., and Syed Zamri, S. A. (2014). The cross-platform application devel- opment adapted the Spring framework to support front-end tendering services. 2014 Inter- national Conference on Computer, Communications, and Control Technology (I4CT), Langkawi, 2014, pp. 58-62, https://doi.org/10.1109/i4ct.2014.6914145 [2] Delia, L., Galdamez, N., Thomas, P., Corbalan, L., and Pesado, P. (2015). Multi-Platform Mobile Application Development Analysis, 2015 IEEE 9th International Conference on Research Challenges in Information Science (RCIS), https://doi.org/10.1109/rcis.2015.712 8878 [3] Papadakis, S.; Vaiopoulou, J.; Kalogiannakis, M.; Stamovlasis, D. (2020). Developing and Exploring an Evaluation Tool for Educational Apps (ETEA.) Targeting Kindergarten Children. Sustainability, 2020, 12, 4201. https://doi.org/10.3390/su12104201 [4] S. Papadakis, and M. Kalogiannakis, “A research synthesis of the real value of self- proclaimed mobile educational applications for young children.,” Mobile learning applica- 114 http://www.i-jim.org https://doi.org/10.1109/i4ct.2014.6914145 https://doi.org/10.1109/rcis.2015.7128878 https://doi.org/10.1109/rcis.2015.7128878 https://doi.org/10.3390/su12104201 Paper—Performing Usability Evaluation on Multi-Platform Based Application for Efficiency and… tions in early childhood education, 1-19, 2020. https://doi.org/10.4018/978-1-7998-1486- 3.ch001 [5] El-Kassas, W. S., Abdullah, B. A., Yousef, A. H., & Wahba, A. M. (2017). Taxonomy of Cross-Platform Mobile Applications Development Approaches. Ain Shams Engineering Journal, 8(2), 163–190. https://doi.org/10.1016/j.asej.2015.08.004 [6] Martinez, M., & Lecomte, S. (2017). Towards the Quality Improvement of Cross-Platform Mobile Applications. 2017 IEEE/ACM 4th International Conference on Mobile Software Engineering and Systems (MOBILESoft). https://doi.org/10.1109/mobilesoft.2017.30 [7] Kadhim KJ, Odhaib MF, Hadi YH, Ameen HM, Muhdi AA, et al. (2018) Developing a Multi Platforms Web Applications for Mobile Device Using HTML5. J Inform Tech Softw Eng 8: 225. https://doi.org/10.4172/2165-7866.1000225 [8] Saswadkar, V., Paygude, V., Dudhe, P., Garad, P., and Gaikwad, R. (2018). Pet Care Sys- tem Based on Android Application. International Journal of Trend in Scientific Research and Development (IJTSRD), pp. 1488-1491. https://doi.org/10.31142/ijtsrd14415 [9] Kolandaisamy, R., Subaramaniam, K., Kolandaisamy, I., and Siew, L. (2016). Stray Ani- mal Mobile App. Conference Paper UCSI University, pp. 1-7. [10] Santy, Karuna, R., and Budiman, A. (2018). E-dopt: A Mobile Application for Pet Adop- tion in Indonesia. TELKOMNIKA. pp. 2137-2143. https://doi.org/10.12928/telkomni ka.v16i5.8074 [11] SENER, R., and TURK, T. (2019). Developing an Android-based Mobile Application for Temporary Animal Shelter Activities. Eurasian Journal of agricultural research. pp. 1-8. [12] ISO 9241-11:2018 (en) Ergonomics of human-system interaction — Part 11: Usability: Definitions and concepts. [13] C. Coursaris & W. Van Osch, (2016). A Cognitive-Affective Model of Perceived User Sat- isfaction (CAMPUS): The complementary effects and interdependence of usability and aesthetics in IS design. Information and Management, 53 (2), 252–264. https://doi.org/10. 1016/j.im.2015.10.003 [14] Sulaiman, S., Mahmood, A., & Khan, M. (2015). Understanding factors influencing user experience of interactive systems: a literature review. Journal of Engineering and Applied Sciences, 10 (23), 18175–18185. [15] Benmoussa, K., Laaziri, M., Khoulji, S., Larbi Kerkeb, M., and El Yamani, A. (2019). Evaluating the Usability of a Moroccan University Research Management Web Platform. Procedia Manufacturing. Pp. 1008-1016. https://doi.org/10.1016/j.promfg.2019.02.315 [16] Erik, B. (2014). Usability testing of e-shop software. Institute of Computer Science Infor- matics Curriculum. Pp. 1-50. [17] Hussain, A., E. O. C. Mojiogu, Jamaludin, N.H, Somia, and T. L. Moh (2017). A usability evaluation of Lazada mobile application. The 2nd International Conference on Applied Science and Technology. Pp. 1-6. https://doi.org/10.1063/1.5005392 [18] Groth, A., and Haslwanter, D. (2016). Efficiency, effectiveness, and satisfaction of respon- sive mobile tourism websites: a mobile usability study. Information Technology and Tour- ism. Pp. 201-228. https://doi.org/10.1007/s40558-015-0041-0 [19] Ali-Shahid, M.M., and Sulaiman, S. (2015). A Case Study on Reliability and Usability Testing of a Web Portal. Malaysian Software Engineering Conference. Pp. 31-36. https://doi.org/10.1109/mysec.2015.7475191 [20] Parsazadeh, N., Ali, R., Rezaei, M., and Tehrani, S. Z. (2018). The construction and vali- dation of a usability evaluation survey for mobile learning environments. Studies in Educa- tional Evaluation. Pp. 97- 111. https://doi.org/10.1016/j.stueduc.2018.06.002 [21] Hujainah, F., Dahlan, H., and Al-Haimi, B. (2013). Usability Guidelines of Mobile Learn- ing Application. Journal of Information Systems Research and Innovation. Pp. 70-77. [22] Madan, A., and Kumar Dubey, S. (2012). Usability Evaluation Methods: A Literature Re- view. International Journal of Engineering Science and Technology (IJEST). Pp. 590-599. iJIM ‒ Vol. 15, No. 10, 2021 115 https://doi.org/10.4018/978-1-7998-1486-3.ch001 https://doi.org/10.4018/978-1-7998-1486-3.ch001 https://doi.org/10.1016/j.asej.2015.08.004 https://doi.org/10.1109/mobilesoft.2017.30 https://doi.org/10.4172/2165-7866.1000225 https://doi.org/10.31142/ijtsrd14415 https://doi.org/10.12928/telkomnika.v16i5.8074 https://doi.org/10.12928/telkomnika.v16i5.8074 https://doi.org/10.1016/j.im.2015.10.003 https://doi.org/10.1016/j.im.2015.10.003 https://doi.org/10.1016/j.promfg.2019.02.315 https://doi.org/10.1063/1.5005392 https://doi.org/10.1007/s40558-015-0041-0 https://doi.org/10.1109/mysec.2015.7475191 https://doi.org/10.1016/j.stueduc.2018.06.002 Paper—Performing Usability Evaluation on Multi-Platform Based Application for Efficiency and… [23] Azhari, S.A., Redzuan, F., and Abdul Majid, R. (2015). Usability Evaluation of Academic Record System at the International Islamic College (IIC). IEEE Conference on e-Learning, e-Management and e-Services. Pp. 1-6. https://doi.org/10.1109/ic3e.2015.7403493 [24] Hussain, A., and Omar, A.M. (2020). Usability Evaluation Model for Mobile Visually Im- paired Applications. International Journal of Interactive Mobile Technologies (iJIM). Vol. 14, No. 05:95. https://doi.org/10.3991/ijim.v14i05.13349 [25] Joo, S., Lin, S., and Lu, K. (2011). A Usability Evaluation Model for Academic Library Websites: Efficiency, Effectiveness and Learnability. Journal of Library and Information Studies. Pp. 11-26. [26] Zhao L, Loucopoulos P, Kavakli E, and Letsholo K (2019) User Studies on End-User Ser- vice Composition, ACM Transactions on the Web, Online Publication, Vol. 13, Issue 3, pp. 1-46. https://doi.org/10.1145/3340294 [27] Riihiaho, S. (2001). Experiences with Usability Evaluation Methods. Helsinki University of Technology. [28] Zhang, T.; Booth, R.; Jean-Louis; R., Chan, R.; Yeung, A.; Gratzer, D.; Strudwick, G. (2020). A Primer on Usability Assessment Approaches for Health-Related Applications of Virtual Reality, JMIR Serious Games 2020; Vol. 8, Issue 4: e18153. https://doi.org/ 10.2196/18153 [29] Devyatkov, V. V., and Tipsin, E. A. (2020). User Behavior Model Development in The Tile Logic for Automating Usability Evaluation. International Conferences Computer Graphics, Visualization, Computer Vision and Image Processing 2020, 23 – 25 July 2020, Russia. https://doi.org/10.33965/cgv2019_201906l032 [30] Macleod, M., Bowden, R., and Bevan, N. (1998). The MUSiC Performance Measurement Method. National Physical Laboratory. Pp. 2-26. [31] MacLeod, M., & Rengger, R. (1998). The Development of DRUM: A Software Tool for Video-assisted Usability Evaluation. Computer Science. [32] Nik Ahmad, N.A., and Syed Dzulkarnain, S.N.S. (2020). Utilization of Gardner’s Multiple Intelligence Theory for School Counselling System with Usability Testing. International Journal of Recent Technology and Engineering (IJRTE) ISSN: 2277-3878, Volume-8 Is- sue-6, pp. 2253 – 2260. https://doi.org/10.35940/ijrte.e6058.038620 [33] Hussain, A., E. O. C. Mkpojigou, and Babalola, E.T. (2020). Using Mobile Educational Apps to Foster Work and Play in Learning: A Systematic Review. International Journal of Interactive Mobile Technologies (iJIM). Vol. 14, No. 18. Pp. 178-194. https://doi.org/ 10.3991/ijim.v14i18.16619 [34] Ahmad, N. A. N., and Kasirun, Z.M. (2011), ‘Elicitation strategies for Web application us- ing activity theory. Journal of Advances in Computer Research. Volume 2, Number 3 (5), pp. 1-13 [online]. Available: https://www.sid.ir/en/Journal/ViewPaper.aspx?ID=260575 [35] Cho, H., Powell, D., Pichon, A., Kuhns, L. M., Garofalo, R., & Schnall, R. (2019). Eye- tracking Retrospective Think-aloud as a Novel Approach for a Usability Evaluation. Inter- national Journal of Medical Informatics. https://doi.org/10.1016/j.ijmedinf.2019.07.010 [36] Papadakis, S. (2020). Tools for evaluating educational apps for young children: a system- atic review of the literature. Interactive Technology and Smart Education, Vol. ahead-of- print No. ahead-of-print. https://doi.org/10.1108/itse-08-2020-0127 [37] Drolia, M.; Sifaki, E.; Papadakis, S.; Kalogiannakis, M. (2020). An Overview of Mobile Learning for Refugee Students: Juxtaposing Refugee Needs with Mobile Applications’ Characteristics. Challenges,11(2), 31. https://doi.org/10.3390/challe11020031 [38] Papadakis, S., & Kalogiannakis, M. (2020). A Research Synthesis of the Real Value of Self-Proclaimed Mobile Educational Applications for Young Children. In Mobile Learning Applications in Early Childhood Education (pp. 1-19). IGI Global. https://doi.org/10.4018/ 978-1-7998-1486-3.ch001 116 http://www.i-jim.org https://doi.org/10.1109/ic3e.2015.7403493 https://doi.org/10.3991/ijim.v14i05.13349 https://doi.org/10.1145/3340294 https://doi.org/10.2196/18153 https://doi.org/10.2196/18153 https://doi.org/10.33965/cgv2019_201906l032 https://doi.org/10.35940/ijrte.e6058.038620 https://doi.org/10.3991/ijim.v14i18.16619 https://doi.org/10.3991/ijim.v14i18.16619 https://doi.org/10.1016/j.ijmedinf.2019.07.010 https://doi.org/10.1108/itse-08-2020-0127 https://doi.org/10.3390/challe11020031 https://doi.org/10.4018/978-1-7998-1486-3.ch001 https://doi.org/10.4018/978-1-7998-1486-3.ch001 Paper—Performing Usability Evaluation on Multi-Platform Based Application for Efficiency and… 7 Authors Nik Azlina Nik Ahmad is a lecturer at the Department of Software Engineering in Universiti Kuala Lumpur. She has experience working as a system analyst for a li- brary system development. Her research interests lie in software testing, usability, user experience, requirements engineering, software maintenance, and mobile UX design. She is certified with Certified Professional for Requirements Engineering (CPRE) and Certified Tester Foundation Level (CTFL). She had experience working on Fundamental Research Grant and Short-Term Research Grant in collaboration with other universities. She was also appointed as a technical program committee for sev- eral international conferences. She held the Program Coordinator position, Final Year Project Coordinator, became the External Examiner for the Lincoln University Col- lege, Programme Advisory Committee for Malaysian Polytechnic, and a Panel for Technical Vocational Education and Training (TVET) workshop. Nur Iman Mardhiah Hamid is a student at Universiti Kuala Lumpur. She suc- cessfully delivered her project, which focused on mobile application solutions. Her research interests include software engineering, software development life cycle, and software usability evaluation. She attained various kinds of awards, such as first place in a programming competition, aerial arts beginner level award, and a gold standard of the Duke of Edinburgh's award. Hence, she portrayed a well-balanced state of mind in different aspects of life. Anitawati Mohd Lokman is a professor at Universiti Teknologi MARA (UiTM). Her expertise is in Kansei Engineering, Emotional User Experience, and Emotion & Design. She is the pioneer of Kansei Engineering in Malaysia and currently serves as the president of the Malaysia Association of Kansei Engineering (MAKE) and the Research Initiative Group for Emotion, Kansei, and Design Engineering (RIG EKDE) at UiTM. Article submitted 2020-12-10. Resubmitted 2021-01-25. Final acceptance 2021-03-16. Final version published as submitted by the authors iJIM ‒ Vol. 15, No. 10, 2021 117