International Journal of Interactive Mobile Technologies (iJIM) – eISSN: 1865-7923 – vol. 15, No. 09, 2021 Paper—A Usability Testing of a Higher Education Mobile Application Among Postgraduate… A Usability Testing of a Higher Education Mobile Application Among Postgraduate and Undergraduate Students https://doi.org/10.3991/ijim.v15i09.19943 Nik Azlina Nik Ahmad (), Muhammad Hussaini Universiti Kuala Lumpur, Kuala Lumpur, Malaysia nikazlina@unikl.edu.my Abstract—The advancement in technology has produced innovative applica- tions for educational purposes. Numerous higher education applications have aroused in recent years. Therefore, it is essential to strive for a high usability application. This study presents a usability evaluation of a higher education mo- bile application in Universiti Kuala Lumpur. The study was conducted using a usability testing methodology based on ISO 9241-11 standard to measure three usability factors: effectiveness, efficiency, and satisfaction. Sixty-four partici- pants from postgraduate and undergraduate programs were subjectively selected. The result reveals that the application is effective, efficient, and meet the users' satisfaction on its usability, with a satisfaction rate of 82.15%. However, several issues were highlighted by the respondents during the usability test. Therefore, despite positive results, several recommendations regarding improving the usa- bility of the application were proposed in this study. Keywords—Usability, usability testing, mobile usability, higher education app 1 Introduction Owing to the strong adoption of technology by new generations, the educational en- vironment is swiftly evolving. A mobile application for teaching and learning is getting more popular in various education levels [1,2,3]. Higher education institutions are re- sponsible for providing convenient infrastructure for all students and should focus more on higher-level interactions that entail a significant change in information and commu- nication-based society. The exponential advancement of mobile technology has been generally recognized at all educational levels [4] and given an excellent chance for an outside-classroom engagement [5]. However, the factors that contribute to a mobile educational application's effectiveness are subject to the application itself's usability, its capability in attracting the users’ interest [6], and its appropriateness [7]. Therefore, with the increasing number of educational applications, a better user experience while using the application is also increased. The application's usability must be evaluated with suitable usability evaluation methods [8]. 88 http://www.i-jim.org https://doi.org/10.3991/ijim.v15i09.19943 mailto:nikazlina@unikl.edu.my Paper—A Usability Testing of a Higher Education Mobile Application Among Postgraduate… Evaluating the functionality of mobile applications alone is not sufficient. Our argu- ment that examining an application's usability is crucial even after the development team has fully functionally tested it. The real effects of using mobile apps in higher education will depend on students' points of view and whether they are usable. It is essential to select the most suitable measurement methods [9]. A proper usability eval- uation should be conducted by assessing its effectiveness, efficiency and meet the users' satisfaction [10]. This study focuses on evaluating a higher education application, namely UniKL Link, which was developed specifically for Universiti Kuala Lumpur (UniKL) students for information retrieval, service assistance, and keeping them up- dated with the latest news or announcement. The usability testing was carried out to determine the application's usability level to provide some recommendations to im- prove the UniKL Link application for the university with the effort to increase the user experience. Mobile learning applications can improve the students' responsibility and self-direction, which contributes significantly to their involvement [11]. To disclose the influence of students' application, it is essential to perform comprehensive assessments and evaluations with students. These findings make a considerable contribution to the improvement and promotion of the application, thereby motivating students to use the app service and substantially increase the service's standard. Providing a handy appli- cation that is accessible to students will also help the university community to grow. This article aims to outline the usability evaluation of a mobile application used in higher education. The main focus is to identify its usability level according to ISO standards; effectiveness, efficiency, and satisfaction. The study's methodology is ex- plained in section 3, along with results and discussion in Section 4, while conclusions and recommendations are eventually proposed at the end of the article. 2 Background 2.1 Usability A software application's usability relies on the user-friendly interface and how much convenience the user feels when using the application [12]. Evaluating usability in- volves looking into the minimum accomplishment degree of various usability factors, which can also be characterized as the degree to which a product or service can guar- antee maximum user satisfaction, efficiency, effectiveness [13, 14, 15, 16], learnability, and ease of use [14] for various type of users. Usability includes the ease with which the customer can access a product or service with minimal complexity and optimum satisfaction while meeting the desired goals [16]. Usability is a widespread issue in human-computer interaction and web technology, in particular [17]. International Organization for Standardization (ISO) 9241-11 standard described us- ability as a degree of quality in which specified users can use the system with effective- ness, efficiency, and satisfaction [10]. The two most regular metrics are effectiveness, which identifies how the user can execute the tasks accurately and efficiently and how easily the user can reach the goal using the available services and tools [18]. There is a high correlation between efficiency and effectiveness, and when the interface is simple, iJIM ‒ Vol. 15, No. 09, 2021 89 Paper—A Usability Testing of a Higher Education Mobile Application Among Postgraduate… it is easier to communicate, resulting in better job performance. Both efficiency and effectiveness will then influence user satisfaction. The assessment of usability is spe- cifically involved in evaluating the core concepts of web accessibility features. Distin- guished models are used to calculate and test several usability factors of a software application interface [19]. Usability usually requires an impartial view of consistency. The usability testing approach's cornerstone is mainly focused on evaluation or meas- urement as participants engage with the product or service. Traditional usability evalu- ation concepts mainly concentrate on user activities and their efficiency in task com- pletion, that is, the functional dimensions of the interaction between user and product [20]. Among the challenges of software interface design involved a broad range of us- ability aspects such as navigation, content usefulness, and user experience. Having a comprehensive usability evaluation is an approach that can be used to identify user in- terface design problems. Usability evaluation or research depends on the measurement of how people communicate with a specific product or service. Therefore, it can be concluded that usability evaluation which also known as usability testing, is mainly used for assessing the users' interaction with a particular application or product [21]. 2.2 Usability characteristics for educational-based application Usability evaluation can be conducted in numerous ways: pluralistic walkthrough, cognitive walkthrough, heuristic evaluation, and guideline reviews [12]. The most com- mon measures for usability are effectiveness, efficiency, and satisfaction. Different measurement criteria should be used to assess different usability factors, where the measurement for effectiveness can be made through assessing the task completion rate. Simultaneously, efficiency can be measured through the users' time to complete the tasks, and satisfaction measurement involves paying attention to the users' confidence level on how the application fulfills their expectations [12,14]. Usability evaluation re- quires user involvement to make it a success. Research on usability testing was conducted by [14]. This study conducted a usabil- ity evaluation on M-learning application to test the effectiveness, efficiency, and user satisfaction using two usability evaluation methods; a Formal experiment and a System Usability Scale (SUS) questionnaire with 100 participants. The findings of this study indicated the application as effective, efficient, and user-friendly. Another study was conducted by [22], who evaluated a Brazilian university's mobile social application to improve the interaction between students and the university. The study collected quan- titative and qualitative data, adapted from the Computer System Usability Question- naire (CSUQ) and Questionnaire for User Interface Satisfaction (QUIS). The latter used the observation technique to identify the usage of the application among participants and semi-structured contextual interviews. In [23], a usability study on university web- sites in the Kyrgyz Republic was conducted. The study looked into the aspects of visi- bility, usability, and accessibility. The usability results showed that most of the web sites took longer upload time, while more than half of the web sites provided inacces- sible links, and all of them have browser compatibility problems. The study suggested that universities in the Kyrgyz Republic need to improve their websites' visibility, usa- 90 http://www.i-jim.org Paper—A Usability Testing of a Higher Education Mobile Application Among Postgraduate… bility, and accessibility. Another study conducted by [24] used questionnaires to eval- uate the user interface's navigability and its design elements in the MyLA Usability instrument. The feedback of the students was committed via an online questionnaire on a five-point Likert scale and eye-tracking. Another usability study involved assessing the mobile counseling application based on Gardner's multiple intelligence theory [25]. The study noted positive results on the application’s usability and quality requirements. Despite many studies conducted in assessing an educational app's usability, most of the studies focus on the scope of teaching and learning, and it is minimal to find such eval- uation that focuses on campus service-based application. 2.3 Testing the usability A usability test is one of the essential activities in software development as it helps to produce better software with maximum user satisfaction. Developers can use the outcome of the usability test for software improvements. Conducting a usability test may involve task completion metrics, observing the errors, time spent or preparing sub- jective satisfaction questionnaires. [26] characterized effectiveness as the consistency of users in completing their tasks without mistake, while performance is best described as how easily users can reliably, efficiently, and thoroughly complete their tasks, and satisfaction can be defined as the positive attitude of users towards using a system along with the ease with which they execute their tasks on the system. Therefore, the presence of users is very critical in performing usability testing for the accuracy of results. 3 Methodology 3.1 Data collection method and analysis As described by ISO 9241-11, usability testing should be conducted with specified users by assessing the application's degree of effectiveness, efficiency, and satisfaction [10]. The implementation of usability testing for the UniKL Link application was con- ducted with the following procedures: Fig. 1. Procedures for usability testing iJIM ‒ Vol. 15, No. 09, 2021 91 Paper—A Usability Testing of a Higher Education Mobile Application Among Postgraduate… • Step 1 Involved preparing a usability test plan, test script (usability test tasks), and recruit the test participants. • Step 2 Was conducted by preparing for the test session and providing a briefing to participants before distributing the test scripts (tasks). Observation technique was used to record the task completion rate and time, and the think-aloud protocol was used to gather feedback. • Step 3 Implied analysis of the findings where the outcomes of the questionnaire from participants were statistically analyzed, presented as the mean calculation to repre- sent the satisfaction rate. In contrast, participants' issues/feedback were investigated, and justifications and recommendations were provided accordingly. 3.2 Testing environment The evaluation of the UniKL Link application was conducted with a total of 64 par- ticipants. Seven of them were from the Master of Computer Science program, while the remaining 57 were undergraduate students from various disciplines ranging from 20 to 27. All participants performed the tasks using their own mobile devices, which run on the Android or iOS platform. While laboratory research may be helpful, it is not prac- tical for such tests to measure real user actions in a real environment. Therefore, actual mobile devices are highly suggested to be used during the assessment [27]. It is neces- sary to assess users' actual experiences and actions in the real-world using actual tool or devices, as these experiences lead to more accurate results. With this approach, the results may capture issues caused by various devices or platforms too. All participants were informed about the test earlier, and two appointed facilitators monitored the session. The tests were separated into several sessions due to participants' availability. Each test took about 1 hour. During the session, the facilitator described the test activities and let every participant understand the implementation. Participants were asked to answer a brief background questionnaire, which includes questions re- garding the program, semester, and the frequency of using mobile phones for educa- tional purposes. The evaluation of gathered data was carried out in the computer labs of the university. 3.3 Task criteria for testing Participants were provided with a list of tasks that need to be tested, grouped accord- ing to the module. A total of 30 task criteria needs to be accomplished. The participants read the task criteria and worked on the required information in the application while being appointed facilitators. 92 http://www.i-jim.org Paper—A Usability Testing of a Higher Education Mobile Application Among Postgraduate… Table 1. Task criteria for the Usability Testing Module Task Criteria Login Task 1: Log in to the app using student email and their password. Task 2: View the home screen (contains menus, students' name, ID, photo, and email). Task 3: Select task to do based on modules. Notification Task 4: Able to receive notifications from their phone. Task 5: View notifications. Logout Task 6: Logout from the app. Academic Task 7: View their subjects that have been registered Task 8: Access academic calendar Task 9: View the past class attendance. Task 10: Check on students’ results. Task 11: Check on students' timetables. Attendance Task 12: Click open the Attendance Task 13: Click the Self-Attend Button Task 14: Scan the QR Code provided by the Lecturer Task 15: Apply for leave application by courses Financial Task 16: View the total outstanding fee. Task 17: View invoices and statements. Task 18: Download invoices and statements in PDF format. Residential Task 19: View students’ residential information. Task 20: Chat with the service desk for inquiries. Activities Task 21: Able to view the "coming soon" error message displayed on the screen. Disciplinary Task 22: View students’ disciplinary case. Welfare Task 23: Able to view the "coming soon" error message. FAQ Task 24: View FAQ by category. Task 25: Search questions using keywords. Service Desk Task 26: Submit enquiry / suggestion / complaint / request. Task 27: Receive notification after submitting inquiry/suggestion/complaint/re- quest to show that the service desk has received it. Task 28: View history of enquiry/ suggestion/ complaint/ request Task 29: View details & solution of enquiry/ suggestion/ complaint/ request Task 30: Rate resolved request. 3.4 Usability evaluation After interacting with the app through the first task (Section 3.3), the participants were asked to proceed with the post-task, completing a questionnaire by rating the ap- plication using a 5-point Likert Scale. The measurement ranges from Strongly Disagree with point value 1 to Agree using value 5 Strongly. The post-task scenario subjective measures include: 1. Do you think this application is easy to use? 2. Do you feel comfortable using this application on your smartphone? 3. Do you think it is easy to find the information needed? 4. Does the information provide in this application clear and understandable? 5. Do you think this application has a pleasant interface? iJIM ‒ Vol. 15, No. 09, 2021 93 Paper—A Usability Testing of a Higher Education Mobile Application Among Postgraduate… 6. Do you feel that learning to use this application is easy? 7. Do you find it easy to view academic-related information? 8. Does this application able to speed up your task completion? 9. Does this application display error messages instead of fixing the problems faced? 10. Do you think that organization of information in this application is clear? 11. Do you find it easy to perform academic-related features? 12. Does this application provide all the functions you expected to have? 13. Does this application provide useful information to help you complete the tasks? 14. This application has no broken link/menu/page. 15. Do you find the steps to accomplish tasks in this application is simple? 16. Does the application provide clear and descriptive information/instruction? 17. Do you feel that navigating through the pages is easy? 18. Do you have the intention to use this application in the future? 19. Will you recommend this application to others? 20. Overall, are you satisfied using this application? 4 Results 4.1 Task completion rate and time: There were thirty tasks used in the study. Table 2 shows the time taken to complete each task. The time to complete each task was observed and recorded by facilitators. The participants were allocated a particular time to finish each task. Table 2. Time spent on task criteria completion Time (seconds) / Total number of participants (64) Tasks 1-5 6-10 11-15 16-20 21-25 26-30 31-35 36-40 41-45 46-50 51-55 56-60 >60 Average Task 1 34 11 8 5 4 2 0 0 0 0 0 0 0 8.11 Task 2 57 3 1 1 2 0 0 0 0 0 0 0 0 3.61 Task 3 56 3 0 2 1 0 0 0 0 0 0 0 2 3.02 Task 4 51 6 3 1 0 0 0 0 0 0 0 0 3 13.51 Task 5 57 2 1 0 0 2 0 0 0 0 0 0 2 3.68 Task 6 51 9 1 1 2 0 0 0 0 0 0 0 0 4.44 Task 7 58 4 0 0 1 1 0 0 0 0 0 0 0 3.51 Task 8 53 5 4 0 0 1 1 0 0 0 0 0 0 4.66 Task 9 53 8 0 2 1 0 0 0 0 0 0 0 0 4.08 Task 10 56 5 1 0 0 1 1 0 0 0 0 0 0 4.61 Task 11 60 3 0 0 1 0 0 0 0 0 0 0 0 2.65 Task 12 59 4 1 0 0 0 0 0 0 0 0 0 0 2.53 Task 13 54 5 3 1 0 0 0 0 0 1 0 0 0 4.41 Task 14 39 14 3 1 1 0 1 1 0 0 0 1 3 14.49 Task 15 57 4 3 0 0 0 0 0 0 0 0 0 0 3.36 Task 16 46 9 3 4 1 1 0 0 0 0 0 0 0 5.91 Task 17 34 19 3 1 1 0 1 1 2 0 1 0 1 18.20 94 http://www.i-jim.org Paper—A Usability Testing of a Higher Education Mobile Application Among Postgraduate… Task 18 59 3 1 1 0 0 0 0 0 0 0 0 0 2.7 Task 19 39 5 2 2 1 0 0 0 2 2 0 6 5 18.01 Task 20 60 3 1 0 0 0 0 0 0 0 0 0 0 1.97 Task 21 61 1 1 0 0 1 0 0 0 0 0 0 0 2.55 Task 22 62 1 0 1 0 0 0 0 0 0 0 0 0 2.07 Task 23 52 7 1 1 2 0 1 0 0 0 0 0 0 4.42 Task 24 44 7 2 2 3 2 0 1 1 1 0 1 0 8.23 Task 25 36 7 3 0 0 2 0 0 0 0 1 5 10 23.09 Task 26 53 6 0 1 0 0 0 0 0 1 0 0 3 7.11 Task 27 56 5 0 2 1 0 0 0 0 0 0 0 0 3.81 Task 28 58 3 1 0 2 0 0 0 0 0 0 0 0 3.70 Task 29 50 8 1 0 0 2 1 1 0 0 0 0 1 6.42 Task 30 59 3 1 1 0 0 0 0 0 0 0 0 0 2.45 Based on the finding, it is clear that all participants were completed all tasks, with a 100% completion rate. Therefore, the effectiveness of the application is high. Most of the tasks took less than 10 seconds to be accomplished by most participants. However, the average is not persistent because some respondents take a longer time to complete the task (more than 60 seconds) due to having difficulties or getting confused with some features. The longest time spent was on tasks 17, 19, and task 25; view invoices and statements, view students' residential information and search questions using keywords. For task 17, some students, especially those who were using iOS unable to download the invoice even after several tries. For task 19, those who were not staying at the hostel found it confusing when they could not find their residential information, while for task 25, the search function did not return any matches or feedback. Fig. 2. Total time taken by the users in each module Figure 2 shows the total time taken by five users in each module. Data of user 17, 18, 19, 20, and 21 were highlighted in this table. Five random students from 64 were chosen. From the table, the highest total time taken to accomplish the task is 217 sec- onds from the service desk module, and the lowest total time taken to accomplish the task is 9 seconds from login module by user 17, attendance module by user 20, financial 9 15 13 34 81 33 142 204 68 217 14 15 13 9 159 15 12 9 2230 18 26 10 29 0 50 100 150 200 250 Login Academic Attendance Financial Service Desk User 17 User 18 User 19 User 20 User 21 iJIM ‒ Vol. 15, No. 09, 2021 95 Paper—A Usability Testing of a Higher Education Mobile Application Among Postgraduate… module by user 19 and 20. Thus, it can be concluded that the efficiency of this mobile application is at a moderate level. 4.2 Task ratings After completing the first task, participants are required to complete a post-task to rate the application's usability to identify the satisfaction of the completed tasks using a five-point Likert scale questionnaire, ranging from 1 (strongly disagree) to 5 (strongly agree). The results of the post-task questionnaire are shown in Table 3. Table 3. Results of Usability Questionnaire Satisfaction Questions Strongly Disagree Disagree Neutral Agree Strongly Agree Q1 0 1 2 29 32 Q2 0 1 4 20 39 Q3 0 3 12 28 21 Q4 1 2 13 30 18 Q5 1 1 13 29 20 Q6 0 0 9 26 29 Q7 0 0 5 33 26 Q8 0 1 16 27 20 Q9 1 8 19 28 8 Q10 0 1 13 37 13 Q11 0 2 10 32 20 Q12 0 14 18 22 10 Q13 0 4 13 36 11 Q14 2 8 14 23 16 Q15 0 1 8 31 24 Q16 0 2 5 36 21 Q17 0 1 7 24 32 Q18 0 1 8 24 31 Q19 1 1 3 22 37 Q20 0 1 3 28 32 Frequency 7 53 195 565 460 The majority of the participants, 60 of them equal to 93.7%, agreed (agree or strongly agree) that they were satisfied using the UniKL Link application. The number corre- sponds to those who agreed to recommend this application to others and those who felt comfortable using it on their smartphones, which is 92.2%. This is also commensurate with the number of participants who thought this application is easy to use, which is 95.3%. A total of 85.9% have the intention to use this application in the future, and the same number of participants found the steps to accomplish tasks in this application is simple. Correspondingly, 87.5% agreed that navigating through the pages is easy, and 89.1% inferred that the application provides clear and descriptive information/instruc- tions, and this lead to 81.3% of the participants found it easy to perform academic- related features using this application. 96 http://www.i-jim.org Paper—A Usability Testing of a Higher Education Mobile Application Among Postgraduate… In terms of information management, 75.0% of the participants thought that the in- formation provided in this application was clear and understandable, 76.6% thought it was easy to find the information they needed, and 78.1% thought that organization of information in this application was straightforward. 73.4% of the participants opined that this application could speed up their task completion as they found it was handy to use a mobile application rather than a web portal. Correspondingly, the same number agreed that the application provided sufficient information to help them complete the tasks. On the other hand, only half of the participants agreed that this application provides all the functions they expected it to have. They suggested some other features to be made available in the app in the future, as highlighted in Table 5. Meanwhile, only 56.3% of the participants agreed that this application displayed error messages with regards of fixing the problems faced. Table 4. Satisfaction Rate Scale Options Strongly Disagree Disagree Neutral Agree Strongly Agree Total Scale number 1 2 3 4 5 Total of Answers 7 53 195 565 460 1280 Maximum Possible Value (Total of Answers * Maximum value of scale) 1280*5 6400 Actual Total Value 1 * 7 = 7 2 * 53 = 106 3* 195 = 585 4 * 565 = 2260 5 * 460 = 2300 5258 Satisfaction Rate (Actual Total Value / Maximum Possible Value) * 100 (5258/6400) * 100 82.15% Table 4 represents the satisfaction rate calculation. In conclusion, from data gather- ing and calculations, the satisfaction rate is equal to 82.15%, which is satisfactorily acceptable. It can be concluded that participants like most parts of the application ex- cept for extra features to be added and error messages organization. 4.3 Issues This evaluation received several positive feedbacks from the participants where some of them pretty happy with how the application can solve their educational activi- ties. However, the evaluation also gathered useful feedback from the participants con- cerning their issues for the improvement purpose. Investigations were made aligned with the feedback received, and justifications and recommendations were provided ac- cordingly. iJIM ‒ Vol. 15, No. 09, 2021 97 Paper—A Usability Testing of a Higher Education Mobile Application Among Postgraduate… Table 5. Issues and Recommendations No. Issues Justifications/ Recom- mendations 1 Unable to download an invoice Investigated: Only happen to iOS users 2 Please add a group number next to the subject in the time table to make it easier to identify which class or group they belong to, for example, Soft- ware Requirements Engineering (L02) Future improvement 3 Make the app able to redirect the user to a webpage after clicking a link such as a survey link for UniKL jackets. Investigated: Network is- sue 4 There is two attendance menu. One in the Academic module and the other in-home screen. The content in these 2-attendance modules is the same. Nothing different. This will lead to confusion as we feel uncertain which Attendance page should we click to scan the QR Investigated: Home screen for short cut / easy access to essential features 5 I cannot go directly to the module that I want to view. For example, cur- rently, I'm in Academic Calendar and want to view my Financial. I have to click the 2 'Back' button until I reach the home screen, then I need to click the Financial and view it. Future improvement 6 In FAQ, there is no question provided for Convocation and Student Ac- tivities. It is not needed to put those sections in the FAQ if there is no question given. Future improvement 7 To view the invoice and student statements, you must download the doc- uments. I cannot directly view the invoice and statement through the Fi- nancial page. Investigated: Invoice is in .pdf format 8 The notification message and logout button do not appear on every page, such as in Financial not appear the notification message and in FAQ page does not appear the logout button. - Inconsistent Future improvement 9 This application should pop up the notification on the phone screen, not when students open the app only because this can make the student less aware or less sensitive about the notification that has been received. Future improvement 10 The invoice and student statement are the same with the same content, but it is placed in different sections with different names. This will lead to confusion. Future improvement 11 I would like to suggest that any activities that have been done in-app will get the notification from the app itself. Future improvement 12 Fix the crash problem the application has when opening finance invoice Investigated: unsupported OS 13 The welfare and activities section has to be made available as soon as possible because those are among the app's main components to be useful for students. Future improvement 14 For subject selection, it is better to list the subject by course. Future improvement 15 I think the Attendance on the home page change to the self-attend which when click it open the scan QR code directly and Attendance in academic just to show the history attendance. Future improvement 16 For an old phone like me to use QR code is a little bit delay to scan the Attendance. Unable to scan the QR attendance when sitting too close or too far from the application Investigated: Happen to only old phone Some issues arose due to the unsupported OS platform and device condition or age span. However, there were also issues related to participant's confusion, which led to dissatisfaction despite many positive comments. The enhancement should take into consideration a more visible user notification, which should be placed next to the app icon rather than inside the application. Having the notification pop up on the phone 98 http://www.i-jim.org Paper—A Usability Testing of a Higher Education Mobile Application Among Postgraduate… screen is more attentive. The app should not let the users see the empty page after per- forming a search but return the 'unable to return any match' message instead to prevent confusion. The application is also suggested to inform users about the minimum re- quirement of using the app. Adapting a think-aloud protocol enables the team to figure out the critical issues from the users' perspective and conduct in-depth analysis. 5 Conclusion This study involved performing a usability assessment of the higher education ap- plication concerning effectiveness, efficiency, and satisfaction. The study was con- ducted with 64 UniKL students, a combination of postgraduate and undergraduate stu- dents as participants. The result affirmed that 100% of participants could complete both task 1 and post-task, which indicates that the app is good in terms of effectiveness. Based on the time taken to complete each task, it can be concluded that this app is efficient for use. The satisfaction rate measured using a post-questionnaire reveals that it is equal to 82.15%, which is satisfactorily acceptable. Overall, the application pro- moted an excellent user experience to both postgraduate and undergraduate students. Nevertheless, the participants highlighted some issues during the test session, which indicates that this application demands severe improvement. This study's limitation is the inexistence of exploring the potential factors that may determine the behavioral in- tention towards the adoption of a campus service application. Future studies are ex- pected to provide comparative results after some interface improvements were made based on this study's results and aimed at identifying the determinant factors of behav- ioral intent to use the higher education mobile application [28-29]. 6 References [1] Pechenkina, E. (2017). Developing a typology of mobile apps in Higher Education: A na- tional case-study. Australasian Journal of Educational Technology, 33(4). https://doi.org/ 10.14742/ajet.3228 [2] Pimmer, C., Mateescu, M., Grohbiel, U. (2016). Mobile and ubiquitous learning in higher education settings. A systematic review of empirical studies. Comput. Hum. Behav. 63, 490– 501. https://doi.org/10.1016/j.chb.2016.05.057 [3] Dorouka, P., Papadakis, S., & Kalogiannakis, M. (2020). Tablets and apps for promoting robotics, mathematics, STEM education and literacy in early childhood education. Interna- tional Journal of Mobile Learning and Organisation, 14(2), 255-274. https://doi.org/ 10.1504/ijmlo.2020.10026334 [4] Papadakis, S. (2016). Creativity and innovation in European education. Ten years eTwin- ning. Past, present, and the future. International Journal of Technology Enhanced Learning, 1, 279. https://doi.org/10.1504/ijtel.2016.082315 [5] Sung, Y.T., Chang, K.E., Liu, T.C. (2016). The effects of integrating mobile devices with teaching and learning on students’ learning performance: a meta-analysis and research syn- thesis. Computers and Education 94, pp. 252–275. https://doi.org/10.1016/j.compedu. 2015.11.008 iJIM ‒ Vol. 15, No. 09, 2021 99 https://doi.org/10.14742/ajet.3228 https://doi.org/10.14742/ajet.3228 https://doi.org/10.1016/j.chb.2016.05.057 https://doi.org/10.1504/ijmlo.2020.10026334 https://doi.org/10.1504/ijmlo.2020.10026334 https://doi.org/10.1504/ijtel.2016.082315 https://doi.org/10.1016/j.compedu.2015.11.008 https://doi.org/10.1016/j.compedu.2015.11.008 Paper—A Usability Testing of a Higher Education Mobile Application Among Postgraduate… [6] Hamidi, H., & Chavoshi, A. (2018). Analysis of the essential factors for the adoption of mobile learning in higher education: A case study of students of the University of Technol- ogy. Telematics and Informatics, Vol. 35, No.4, pp.1053–1070. DOI: https://doi.org/10. 1016/j.tele.2017.09.016 [7] Papadakis, S.; Vaiopoulou, J.; Kalogiannakis, M.; Stamovlasis, D. (2020). Developing and Exploring an Evaluation Tool for Educational Apps (ETEA.) Targeting Kindergarten Chil- dren. Sustainability 2020, 12, 4201. https://doi.org/10.3390/su12104201 [8] Hoehle, H., Aljafari, R., Venkatesh, V. (2016). Leveraging Microsoft’s mobile usability guidelines: conceptualizing and developing scales for mobile application usability. Interna- tional Journal of Human-Computer Studies. Vol.89, pp. 35–53. https://doi.org/10.1016/ j.ijhcs.2016.02.001 [9] Cho, H., Powell, D., Pichon, A., Kuhns, L. M., Garofalo, R., & Schnall, R. (2019). Eye- tracking Retrospective Think-aloud as a Novel Approach for a Usability Evaluation. Inter- national Journal of Medical Informatics. https://doi.org/10.1016/j.ijmedinf.2019.07.010 [10] ISO 9241-11:2018 (en) Ergonomics of human-system interaction — Part 11: Usability: Def- initions and concepts. [11] Volkovitckaia, G., Tikhonova, Y., & Kolosova, O. (2020). Educational Experience in the Mobile Learning Environment: Consumer Behavior Perspective. International Journal of Interactive Mobile Technologies (IJIM), 14(21), pp. 92-106. DOI: https://doi.org/10.3991/ ijim.v14i21.18441 [12] Kumar, B.A., Goundar, M.S. & Chand, S.S. (2019). Usability guideline for Mobile learning applications: an update. Educ Inf Technol 24, 3537–3553 (2019). https://doi.org/10.1007/ s10639-019-09937-9 [13] Abuqaddom, I., Alazzam, H., Hudaib, A., Al-Zaghoul, F. (2019). A measurable website us- ability model: Case Study University of Jordan, 2019 10th International Conference on In- formation and Communication Systems (ICICS). https://doi.org/10.1109/iacs.2019.8809 145 [14] Arain A.A., Hussain Z., Rizvi W.H., Vighio M.S. (2016). Evaluating Usability of M-Learn- ing Application in the Context of Higher Education Institute. International Conference on Learning and Collaboration Technologies. Springer International Publishing Switzerland 2016, pp. 259–268. https://doi.org/10.1007/978-3-319-39483-1_24 [15] Wei, Q., Chang, Z., Cheng, Q. (2015). Usability study of the mobile library App: an example from Chongqing University. Library Hi-Tech, Vol. 33, Issue 3 pp. 340 – 355. https:// doi.org/10.1108/lht-05-2015-0047 [16] Adepoju, S. and Shehu, I. (2014). Usability evaluation of academic websites using auto- mated tools. User Science and Engineering (i-USEr), 3rd International Conference on. IEEE, 2014, pp. 186–191. https://doi.org/10.1109/iuser.2014.7002700 [17] James R. Lewis (2018). Measuring Perceived Usability: The CSUQ, SUS, and UMUX, In- ternational Journal of Human-Computer Interaction, https://doi.org/10.1080/10447318. 2017.1418805 [18] Constantinos K. Coursaris, Wietske van Osch (2016). A Cognitive-Affective Model of Per- ceived User Satisfaction (CAMPUS): The complementary effects and interdependence of usability and aesthetics in IS design. Information and Management. Vol 53, No. 2, pp. 252- 264. https://doi.org/10.1016/j.im.2015.10.003 [19] Dirin, A., & Nieminen, M. (2015). mLUX: Usability and User Experience Development Framework for M-Learning. International Journal of Interactive Mobile Technologies (IJIM), 9(3), pp. 37-51. https://doi.org/10.3991/ijim.v9i3.4446 [20] Olaleye, S., Oyelere, S., Sanusi, I., & Agbo, J. (2018). Experience of Ubiquitous Computing Technology-Driven Mobile Commerce in Africa: Impact of Usability, Privacy, Trust, and 100 http://www.i-jim.org https://doi.org/10.1016/j.tele.2017.09.016 https://doi.org/10.1016/j.tele.2017.09.016 https://doi.org/10.3390/su12104201 https://doi.org/10.1016/j.ijhcs.2016.02.001 https://doi.org/10.1016/j.ijhcs.2016.02.001 https://doi.org/10.1016/j.ijmedinf.2019.07.010 https://doi.org/10.3991/ijim.v14i21.18441 https://doi.org/10.3991/ijim.v14i21.18441 https://doi.org/10.1007/s10639-019-09937-9 https://doi.org/10.1007/s10639-019-09937-9 https://doi.org/10.1109/iacs.2019.8809145 https://doi.org/10.1109/iacs.2019.8809145 https://doi.org/10.1007/978-3-319-39483-1_24 https://doi.org/10.1108/lht-05-2015-0047 https://doi.org/10.1108/lht-05-2015-0047 https://doi.org/10.1109/iuser.2014.7002700 https://doi.org/10.1080/10447318.2017.1418805 https://doi.org/10.1080/10447318.2017.1418805 https://dblp.org/pid/39/10260.html https://doi.org/10.1016/j.im.2015.10.003 https://doi.org/10.3991/ijim.v9i3.4446 Paper—A Usability Testing of a Higher Education Mobile Application Among Postgraduate… Reputation Concern. International Journal of Interactive Mobile Technologies (IJIM), 12(3), pp. 4-20 https://doi.org/10.3991/ijim.v12i3.7905 [21] Venkatesh H., Hoehle, H., Aljafari, R. (2014). A usability evaluation of the Obamacare web- site, Government Information Quarterly 31 (2014) 669–680. https://doi.org/10.1016/j.giq. 2014.07.003 [22] Darin T., Andrade R., Macedo J., Araújo D., Mesquita L., and Sánchez J. (2016) Usability and UX Evaluation of a Mobile Social Application to Increase Students-Faculty Interactions. In: Stephanidis C. (eds) HCI International 2016 – Posters' Extended Abstracts. HCI 2016. Communications in Computer and Information Science, vol 618. Springer, Cham. https://doi.org/10.1007/978-3-319-40542-1_4 [23] Ismailova, R., Kimsanova, G. (2017). Universities of the Kyrgyz Republic on the Web: Ac- cessibility and Usability. Univ Access Inf Soc 16, pp. 1017–1025. https://doi.org/10.1007/ s10209-016-0481-0 [24] Kuhnel, M., Seiler, L., Honal, A., Ifenthaler, D. (2018). Mobile learning analytics in higher education: usability testing and evaluation of an app prototype. Interactive Technology and Smart Education, Vol. 15 Issue: 4, pp.332-347. https://doi.org/10.1108/itse-04-2018-0024 [25] Nik Ahmad, N.A. and Syed Dzulkarnain, S.N.S. (2020). Utilization of Gardner’s Multiple Intelligence Theory for School Counselling System with Usability Testing. International Journal of Recent Technology and Engineering (IJRTE). Vol.8, Issue 6, pp.2253-2260. https://doi.org/10.35940/ijrte.e6058.038620 [26] Hussain, A., Mkpojiogu, E.O.C., Suleiman, K. (2018). A Usability Testing of a Mobile Print Shop Booking and Design Application, Journal of Advanced Research in Dynamical and Control Systems 10(SI), pp.1359-1365. [27] Liu, Y.-C., Chen, C.-H., Lee, C.-W., Lin, Y.-S., Chen, H.-Y., Yeh, J.-Y., & Chiu, S. Y.-H. (2016). Design and usability evaluation of user-centered and visual-based aids for dietary food measurement on mobile devices in a randomized controlled trial. Journal of Biomedical Informatics, 64, 122–130. https://doi.org/10.1016/j.jbi.2016.10.001 [28] Drolia, M.; Sifaki, E.; Papadakis, S.; Kalogiannakis, M. (2020). An Overview of Mobile Learning for Refugee Students: Juxtaposing Refugee Needs with Mobile Applications’ Characteristics. Challenges,11(2),31. https://doi.org/10.3390/challe11020031 [29] Papadakis, S., & Kalogiannakis, M. (2020). A Research Synthesis of the Real Value of Self- Proclaimed Mobile Educational Applications for Young Children. In Mobile Learning Ap- plications in Early Childhood Education (pp. 1-19). IGI Global. https://doi.org/10.4018/ 978-1-7998-1486-3.ch001 7 Authors Nik Azlina Nik Ahmad is a lecturer at the Department of Software Engineering in Universiti Kuala Lumpur. She has experience working as a system analyst for a library system development and other information system projects. Her research interests lie in the area of software testing, usability, user experience, requirements engineering, software maintenance, and mobile UX design. She is certified with Certified Profes- sional for Requirements Engineering (CPRE) and Certified Tester Foundation Level (CTFL). She had experience working on Fundamental Research Grant and Short-Term Research Grant in collaboration with other universities. She was also appointed as a technical program committee for several international conferences. She held the Pro- iJIM ‒ Vol. 15, No. 09, 2021 101 https://doi.org/10.3991/ijim.v12i3.7905 https://doi.org/10.1016/j.giq.2014.07.003 https://doi.org/10.1016/j.giq.2014.07.003 https://doi.org/10.1007/978-3-319-40542-1_4 https://doi.org/10.1007/s10209-016-0481-0 https://doi.org/10.1007/s10209-016-0481-0 https://doi.org/10.1108/itse-04-2018-0024 https://doi.org/10.35940/ijrte.e6058.038620 https://doi.org/10.1016/j.jbi.2016.10.001 https://doi.org/10.3390/challe11020031 https://doi.org/10.4018/978-1-7998-1486-3.ch001 https://doi.org/10.4018/978-1-7998-1486-3.ch001 Paper—A Usability Testing of a Higher Education Mobile Application Among Postgraduate… gram Coordinator position, Final Year Project Coordinator, became the External Ex- aminer for the Lincoln University College, Programme Advisory Committee for Ma- laysian Polytechnic, and a Panel for Technical Vocational Education and Training (TVET) workshop. Muhammad Hussaini is a postgraduate student at Universiti Kuala Lumpur. He was successfully delivered his project on collaborative filtering and content-based rec- ommendation engines. His research interests include machine learning, deep learning, software engineering, and agile software development. Article submitted 2020-11-19. Resubmitted 2021-01-04. Final acceptance 2021-01-11. Final version pub- lished as submitted by the authors. 102 http://www.i-jim.org