International Journal of Interactive Mobile Technologies (iJIM) – eISSN: 1865-7923 – Vol. 14, No. 5, 2020 Paper—Usability Testing of Mobile Flipboard Application on both Non-Users and Novice Users Usability Testing of Mobile Flipboard Application on both Non-Users and Novice Users https://doi.org/10.3991/ijim.v14i05.13341 Marwah Naeem Hassooni Al-Kinani, Sumayyah Bukola Adetunmbi, Azham Hussain () Universiti Utara Malaysia, Kedah, Malaysia azham.h@uum.edu.my Abstract—This paper aims to discuss how journalism in a network has transformed the production and circulation of news and as gate keeping and motivation setting, from the action interdependence of ordinary people, journalists, organizations and advanced innovations (such as blogs, Twitter, Facebook and Flipboard). Finally, we analyze the interactions in the Flipboard mobile app. We carried out a study consisting of 13 participants in UUM Sintok, they were assigned five tasks to perform and questionnaires were distributed to gather their experience with the app. The results of this study showed that participants were satisfied with the application's aesthetic but find the complexity of features provided in the application to perform various functions very frustrating to use. Keywords—News aggregator, usability, testing, magazine, evaluation method, mobile application. 1 Introduction Over the years there has been an enormous amount of effort put into satisfying users and get to know their exact needs and what they look for in a software and application. That’s why there has been a wide range of usability testing done for a products and services. Before the coming of smartphones, usability testing was only limited to software and web applications. But since mobile smartphones was created, mobile applications also came into the picture. As mobile smartphones together with mobile apps becomes more popular and easily accessible, there are more applications being developed with various purposes and similar apps came from different company and developer. This led to competition for the survival of the fittest to see who can create a mobile application that is most efficient and is also able to satisfy the users enough for them to stick with the app without having the need to go for a similar app that also serves the same purpose. This eventually requires the need for usability testing of mobile applications to be able to get the exact requirements of what users expected in an application and also knows the features that is suitable for a group of people with special needs whenever they use the particular application. iJIM ‒ Vol. 14, No. 5, 2020 47 https://doi.org/10.3991/ijim.v14i05.13341 mailto:azham.h@uum.edu.my Paper—Usability Testing of Mobile Flipboard Application on both Non-Users and Novice Users 2 Background Flipboard is a news aggregation application that gives stretched out access to news articles and business stories from a wide scope of productions, including the New York Times, Guardian, Financial Times, and Forbes, among others[1]. It is a news aggregate site and distribution service having branches in New York, Vancouver and Beijing, their headquarter is in Palo Alto, California. The software was initially launched in July 2010, also referred to as Flipboard. It aggregates content from social media, news feeds, photo sharing sites and other websites, presents it in magazine format, and allows users to "flip" through the articles, images and videos being shared. Readers can also save stories into Flipboard magazines. As of March 2016, the company claims there have been 28 million magazines created by users on Flipboard. The service can be accessed through an internet browser, a Windows OS or Mac OS version of the Flipboard application, and iOS, Android and Windows Phone smartphone apps. Customer software is available free of charge and is available in 21 languages. It's a fantastic social app that enables users to share, group and gather content from various sources all in one location. It offers you the versatility to create your own streams, which are labeled 'magazines' to assemble relevant content that you will find online. Readers can also subscribe to general magazines from others or the ones managed by a specific brand. It is easy to use the device that is currently common among people. But the issue is how people are satisfied with their current mobile Flipboard app? In order for a Flipboard mobile app to be able to deliver service to a consumer, things like features, layout and accessibility are essential. Therefore, in order to achieve both successful experiences and customer satisfaction, equal importance must be provided to the usability of such an application and user-centric issues must be at the core of the operation. Flipboard offering news services has limited functionality and this demonstrates that their interfaces are still incomplete and not user-friendly. Flipboard mobile app is a lightweight program that can run on mobile gadgets like smartphones, computers and tablets. Making use of smartphone, online news that can be watched at ease offers a comfortable platform for consumers and being able to readily get it when available. The app which represents the website with name Flipboard was utilized in carrying out usability testing on participants in this study. 3 Methodology 3.1 Setting up a usability laboratory and field study For the usability testing procedure, the evaluation is carried using two main methods of inquiry or environments which is the adoption of usability laboratory and field study to investigate the user’s satisfaction towards Flipboard application in the context of efficiency and effectiveness. For the part of the usability laboratory, various quiet places were set up both inside the UUM library (well known as Perpustakaan Sultanahbahiyah) by using the postgraduate area section of the library 48 http://www.i-jim.org Paper—Usability Testing of Mobile Flipboard Application on both Non-Users and Novice Users which are usually more closed in, private and controlled from outside distractions. Some of the tests are also done at the smart reading rooms that are near the UUM library and also at the Library foyer during the time when it is most quiet; these places are used to serve the purpose of performing the test using the method Field testing. The other place used in achieving the field-testing method is the student residential hall of DPP Sime Darby and the Student Lounge located around there, which is well known to most students. The users were given the permission to perform the tasks assign to them in any position they preferred whether it is by sitting or stand. There were two supervisors, one was entrusted with assigning the designated activities verbally to the users and the other was entrusted with recording the test segment with a smartphone camera. Following agreement between the researcher and the participants on the usability testing, no restrictions were placed on the individual during the test session to create a perfect atmosphere for the respondents. And we make sure the respondents were reassured that they are not being tested but the application they are doing task on. 3.2 Tools The respondents were allowed to use their various mobile devices that can either be android or iOS to perform the tasks so as to make them more comfortable in doing the testing since they are used to their own mobile phones and will prefer it to another type of mobile devices they have never used before. We allowed them to use their various mobile devices also for the purpose to prove that the application to be tested on can easily adjust to various mobile device preferences and operating systems. The main recording done for each task the respondent is doing through screen recording capture on each respondent’s phone. So, before the test started, the respondents have to install both Flipboard application and a screen record app (if they don’t have screen record features on their devices) provided by us on their mobile devices. Apart from phone screen recording how the respondents are doing each task, we also take a picture of every respondents while performing the task to serve as proof of the test. 3.3 Task description The 11 respondents in each session are required to test the Flipboard application by using the touch screen to accomplish the 5 experimental tasks. The researcher records the entire sessions on the phone video of the respondents while they are involved with the activities. The testing is done individually, from a participant following another. After concluding with the tasks, the respondents are required to complete the post-test questionnaire. The following are the list of the five tasks used in the testing:  Register an Account  Add categories of news  Link social media accounts to the app  Follow a magazine page iJIM ‒ Vol. 14, No. 5, 2020 49 Paper—Usability Testing of Mobile Flipboard Application on both Non-Users and Novice Users  Create a collection(playlist) of news in profile account The researcher advises respondents to use the "thinking aloud" technique prior to starting the study while they are in the task execution phase. The participants are also allowed to move effortlessly from one task to another if they choose, and this is intended to observe the actions of users when moving from one component to another. The post-test survey provides the additional data and participants’ response to the Flipboard application. The procedure of this experiment is based on the following steps: (adapted from the works of Lin, 2015)  Step 1: Carrying out of the 5 experimental tasks by the 11 respondents who evaluate the Flipboard app system. The respondents are selected randomly to evaluate the application.  Step 2: The researcher has recorded every experimental session using screen recording video app for mobile phone, comprising of when the errors happened and amount of time it took to do the normal task. The calculation consists of total amount of errors that respondents created when trying to finish a task and the time it took to perform a task is the operation time. With a stopwatch and the screen capturing app, the time to finish every task is documented. Successful tasks (whether a respondent finishes a task efficiently or not) are also documented. To fulfill each task, respondents are allocated up to four minutes (pilot testing was conducted to decide that 4 minutes is appropriate to complete the task).  Step 3: When the total 5 tasks are completed, the respondents are immediately given a post-test questionnaire so as to measure each factor of the usability test.  Step 4: Analysis of the numerical data that includes the operation time, error frequency and successful rate. 3.4 Preliminary evaluation An initial evaluation was carried out to determine whether users find the Flipboard application as being efficient and effective as its help users get various news across the world easily and also able to share it with others. It was heavily based on the users’ own words. This study chose to collect quantitative and qualitative data for the tests to reach in order to deem that the usability testing of Flipboard is successful or otherwise. The acceptability testing of the Flipboard can be obtained in a more in- depth fashion by opting for a qualitative investigation and not quantitative alone. The users had informative opinion and information of the Flipboard after the testing procedure. The evaluation comprised direct questions related to the screen (visibility), terminology and app information, learning (Ease of use), application capabilities (efficiency of the app) and general impressions of the Flipboard App. The Flipboard mobile application was used throughout the evaluation period. This study invited 13 students (10 male and 3 female) coming from different level of study at the Universiti Utara Malaysia. The users are categorized as experienced users in terms of using the application (i.e. on the scale of “3 years and above”), 50 http://www.i-jim.org Paper—Usability Testing of Mobile Flipboard Application on both Non-Users and Novice Users moderate users (i.e. on the scale of “1 - 2 years”), and novice users (i.e. on the scale of “less than a year”) in term of the number of years the users have been using the application. The users are categorized as frequently-use users in term of using the application daily. Table 1 above shows the user profiling of the participants. Though only 11 students were able to do the testing with recording, the other 2 male students just checked the application on their own and then filled the post-test questionnaire. Table 1. User Profiling 4 Results and Discussions After the participants completed the task that was given to them, they were given a post-experiment questionnaire to provide more information based on their thoughts about the efficiency and effectiveness on the ease of use of the kindle. The results of this experiment were analyzed using SPSS V23. In this section, the reliability and descriptive statistics are presented accordingly. Descriptive statistics was used due to its convenient and simplicity to analyse the collected data (Mwaura, K. R, 2013). These statistics took the forms of percentage, mean and frequency distribution The findings are presented by using tables, graphs, mean, frequencies and percentages. Table 2 illustrates the outcome of the task completion. Errors were made mostly in task 3 (Link social media) where the main error was how users is able to find the section where they can link their social media accounts to the app. The other task with most errors is task 5 (Create collection), this is mainly because participants don’t recognize the button icon for creating playlist for news and also what to click. The button was not representing the function it is supposed to perform, the overall completion rate of both task 3 and task 5 is 82%. On the other hand, one of the participants committed an error while carrying out task 2 which is (Add categories), the error was because the participant went to the wrong place and choose wrong User # Gender Study level Experience 1 Male Postgraduate Novice 2 Male Postgraduate Novice 3 Female Undergraduate Moderate 4 Male Postgraduate Novice 5 Male Postgraduate Novice 6 Male Postgraduate Novice 7 Male Postgraduate Novice 8 Female Postgraduate Novice 9 Male Postgraduate Novice 10 Female Undergraduate Novice 11 Male Postgraduate Experienced 12 Male Postgraduate Moderate 13 Female Undergraduate Moderate iJIM ‒ Vol. 14, No. 5, 2020 51 Paper—Usability Testing of Mobile Flipboard Application on both Non-Users and Novice Users option to add categories so, he was unable to do it before he started another task. 6out of all the participants finished every task given and did not encounter any challenges. Table 2. Task Completion Table Participant Task 1 (Register Account) Task 2 (Add Categories) Task 3 (link social media) Task 4 (Follow magazine) Task 5 (Create collection) 1 √ √ √ √ √ 2 √ √ √ √ √ 3 √ √ √ √ √ 4 √ √ √ √ - 5 √ √ √ √ √ 6 √ - √ √ √ 7 √ √ - √ √ 8 √ √ √ √ √ 9 √ √ √ √ √ 10 √ √ - √ √ 11 √ √ √ √ - Success 11 10 9 11 9 Completion Rates 100% 91% 82% 100% 82% Table 3 summarizes the findings, only 7 out of 11 participants accomplished task 3 without any errors in the task steps which are; start flipboard> go to my profile > settings > go to accounts > connect your social media accounts. Due to the complexity of the app interface in which the button does not indicate its purpose, the four participants were not able to locate the button to move ahead to the fourth step smoothly. So, the participant does not understand exactly where to click to progress the other step. This task has the maximum running time of 4 minutes and 48 seconds to be exact. At another part, Task 1 had only one error because of incorrect passwords to register. Only one error was also committed in task 4, which is because of incorrect navigation. The steps to accomplish task 1 includes; start flipboard>get started> go to profile> Sign up> enter email and password> enter full name>enter username. See table 2 for Summary of Completion, Errors, Time on Task. Table 3. Summary of Completion, Errors, Time on Task Task Task Completion Errors Time on Task 1 11 1 3:00 2 10 2 1:00 3 9 4 3:45 4 11 1 0:30 5 9 2 1.20 Screen (Visibility): Table 4 has shown the descriptive statistics for the screen measure. From the below table, the questions answered by the respondents in the 52 http://www.i-jim.org Paper—Usability Testing of Mobile Flipboard Application on both Non-Users and Novice Users present study about the screen visibility of the application, it shows that the majority of the respondents’ answers chosen for the questions ‘Agree’. Results have shown that the value of mean is above 4 and the value of standard deviation is below 1, (highest mean = 4.23, Sd = ~1.068) (lowest mean = 3.85, Sd = ~0.439). Table 4. Screen (SN) Item N Minimum Maximum Mean Std. Deviation Likert Scale Strongly Disagree Uncertain Agree Strongly Agree SN1 13 4 5 4.23 0.439 0 0 0 10 3 SN2 13 2 5 4.00 0.913 0 1 2 6 4 SN3 13 2 5 3.85 1.068 0 2 2 5 4 SN4 13 2 5 3.85 1.068 0 2 2 5 4 SN5 13 2 5 4.00 0.913 0 1 2 6 4 App Terminology (Information): Table 5 shows the descriptive statistics for the app information. From the below table according to the questions answered by the respondents in the present study about the ease of terminology of the application, it shows that the majority of the respondents’ answers chosen for the questions is between ‘Agree’ and ‘Uncertain’. Results have shown that the value of mean is below 4 and the value of standard deviation is below 1, (highest mean = 4.00, Sd = ~1.050) (lowest mean = 3.23, Sd = ~0.577). Table 5. App Terminology (AT) Item N Minimum Maximum Mean Std. Deviation Likert Scale Strongly Disagree Disagree Uncertain Agree Strongly Agree AT1 13 3 5 3.62 0.650 0 0 6 6 1 AT2 13 3 5 4.00 0.577 0 0 2 9 2 AT3 13 2 5 3.46 1.050 0 3 3 5 2 AT4 13 2 5 3.54 0.877 0 1 6 4 2 AT5 13 2 5 3.23 0.927 0 3 5 4 1 AT6 13 3 5 3.62 0.650 0 0 6 6 1 Learning: Table 6 below shows the descriptive statistics for the flexibility of learning how to use the application. From the below table according to the questions answered by the respondents in the present study about the ease of terminology of the application, it shows that the majority of the respondents’ answers chosen for the questions is ‘Agree’. Results have shown that the value of mean is below 4 and the value of standard deviation is below 1, (highest mean = 4.08, Sd = ~0.967) (lowest mean = 3.54, Sd = ~0.801). iJIM ‒ Vol. 14, No. 5, 2020 53 Paper—Usability Testing of Mobile Flipboard Application on both Non-Users and Novice Users Table 6. Learning (LG) N Minimum Maximum Mean Std. Deviation Likert Scale Strongly Disagree Disagree Uncertai n Agree Strongly Agree LG11 13 2 5 3.85 0.801 0 1 2 8 2 LG2 13 2 5 3.77 0.927 0 2 1 8 2 LG3 13 2 5 3.77 0.832 0 1 3 7 2 LG4 13 2 5 3.54 0.967 0 2 4 5 2 LG5 13 2 5 4.08 0.954 0 1 3 4 5 Application Capabilities: Table7 below shows the descriptive statistics for capabilities of the application. From the below table according to the questions answered by the respondents in the present study about the ease of terminology of the application, it shows that the majority of the respondents’ answers chosen for the questions is mainly ‘Agree’. Results have shown that the value of mean is below 4 and the value of standard deviation is above 1, (highest mean = 4.08, Sd = ~1.127) (lowest mean = 3.46, Sd = ~0.494). Table 7. Application Capabilities (AC) Item N Minimum Maximum Mean Std. Deviation Likert Scale Strongly Disagree Disagree Uncertai n Agree Strongly Agree AC1 13 3 5 4.08 0.494 0 0 1 10 2 AC2 13 3 5 3.92 0.760 0 0 4 6 3 AC3 13 2 5 3.46 0.776 0 1 6 5 1 AC4 13 2 5 3.54 1.127 0 3 3 4 3 General Impressions: Table8 below shows the descriptive statistics for general impressions of the application. From the below table according to the questions answered by the respondents in the present study about the ease of terminology of the application, it shows that the majority of the respondents’ answers chosen for the questions is mainly ‘Agree’. Results have shown that the value of mean is below 4 and the value of standard deviation is below 1, (highest mean = 4.08, Sd = ~0.862) (lowest mean = 3.92, Sd = ~0.760). Table 8. General Impressions (GI) N Minimum Maximum Mean Std. Deviation Likert Scale Strongly Disagree Disagree Uncertain Agree Strongly Agree GI1 13 2 5 3.92 .760 0 1 4 9 2 GI2 13 3 5 3.92 .760 0 0 4 6 3 GI3 13 2 5 4.08 .862 0 1 1 7 4 54 http://www.i-jim.org Paper—Usability Testing of Mobile Flipboard Application on both Non-Users and Novice Users 5 Conclusion Thus, in conclusion, this area of research has been enriched by the findings of this research. The Flipboard application has shown a very low satisfaction in use while at the same time, providing aesthetically beautiful interface to the user. The study provides a depth analysis of usability test on four principles, which are ease of use, visibility, enjoyment and efficiency. Quantitative and qualitative methods are used to analyze the data; within the context of this research boundaries the objectives have been achieved, which is to investigate users’ satisfaction towards the usability of the Flipboard application. Even though the Flipboard app has a very advanced and interactive interface and layouts that makes it more attractive to the user, further work is needed to be done on the application, so that anyone of different age or academic background with either little or high knowledge in technology could be able to operate and use easily, so it could wind up satisfying most users and gain more frequent users of the app and could turn to be the best news aggregator app. This is because there were limits to the research effort, which are like the number of participants and furthermore some enhancement is required to be done in order to achieve this goal. 6 References [1] E. F. McGovern, C. Luna-Nevarez, and A. Baruca, “Utilizing mobile devices to enrich the learning style of students,” J. Educ. Bus., vol. 92, no. 2, pp. 89–95, 2017 https://doi.org/10.1080/08832323.2017.1281213. [2] C. Study, “Flipboard partners with UserTesting to understand the full impact of its third generation Case Study Flipboard ’ s mobile app is loved by users .,” 2010. [3] Ahmad, N., Boota, M., & Masoom, A. (2014). Smart phone application evaluation with usability testing approach. Journal of Software Engineering, 7(12), 1045.https://doi.org/10.4236/jsea.2014.712092 [4] Wilson, R., Landoni, M., & Gibb, F. (2002). A user-centred approach to e-book design. The Electronic Library, 20(4), 322–330. https://doi.org/10.1108/02640470210 438865 [5] A. Hussain and M. Kutar, “Usability Metric Framework for Mobile Phone Application,” The 10th Annual PostGraduate Symposium on The Convergence of Telecommunications, Networking and Broadcasting, June 22-23, 2009 https://doi.org/10.1145/1497308. 1497412. [6] C. K. Coursaris and D. J. Kim, “A Meta-Analytical Review of Empirical Mobile Usability Studies,” Journal of Usability Studies, vol. 6, no. 3, pp. 117-171, 2011. [7] Been-Lirn Duh, H., Tan. G. C. B. Usability evaluation for mobile device: A comparison of laboratory and field tests. Proc. 8th International Conference on Human Computer Interaction with Mobile Devices and Services, September 12-15, 2006, Espoo, Finland. [8] Balagtas-Fernandez, F., & Hussmann, H. (2009, November). A methodology and framework to simplify usability analysis of mobile applications. In Proceedings of the 2009 IEEE/ACM International Conference on Automated Software Engineering(pp. 520- 524). IEEE Computer Society https://doi.org/10.1109/ASE.2009.12. iJIM ‒ Vol. 14, No. 5, 2020 55 https://doi.org/10.1080/08832323.2017.1281213 https://doi.org/10.4236/jsea.2014.712092 https://doi.org/10.1108/02640470210438865 https://doi.org/10.1108/02640470210438865 https://doi.org/10.1145/1497308.1497412 https://doi.org/10.1145/1497308.1497412 https://doi.org/10.1109/ASE.2009.12 Paper—Usability Testing of Mobile Flipboard Application on both Non-Users and Novice Users [9] Been-Lirn Duh, H., Tan. G. C. B. Usability evaluation for mobile device: A comparison of laboratory and field tests. Proc. 8th International Conference on Human Computer Interaction with Mobile Devices and Services, September 12-15, 2006, Espoo, Finland. [10] Delamaro, M. E., Vincenzi, A. M. R., & Maldonado, J. C. (2006, May). A strategy to perform coverage testing of mobile applications. In Proceedings of the 2006 international workshop on Automation of software test (pp. 118-124). ACM https://doi.org/10. 1145/1138929.1138952. [11] Jardina, J., & Chaparro, B. (2013). Usability, engagement, and satisfaction of two e-120 textbook applications. Proceedings of the Human Factors and, 57(1), 482–486. https://doi.org/10.1177/1541931213571104 [12] Ahmad, N., Boota, M., & Masoom, A. (2014). Smart phone application evaluation with usability testing approach. Journal of Software Engineering, 7(12), 1045. https://doi.org/10.4236/jsea.2014.712092 [13] Hoehle, H., & Venkatesh, V. (2015). Mobile application usability: conceptualization and instrument development. Mis Quarterly, 39(2), 435–472. https://doi.org/10. 25300/MISQ/2015/39.2.08 7 Authors Marwah Naeem Hassooni Al_kinani is a master student in Information Technology at School of Computing, University Utara Malaysia, 06010, Kedah, Malaysia. Marwah Naeem also a teacher at Secondary School, General Directorate for Education in the province of Maysan, Iraqi Ministry of Education. Email: hasnaseem84@gmail.com Sumayyah Bukola Adetunmbi is a master student in Information Technology at School of Computing, University Utara Malaysia, 06010, Kedah, Malaysia. She interested doing research on Human and Mobile interaction. Email: sumayyah1433@gmail.com Azham Hussain is the Associate Professor of Software Engineering at UUM School of Computing. He is the founder and head of Human-Centered Computing Research Group which is affiliated with the Software Technology Research Platform Center at School of Computing, University Utara Malaysia, 06010, Kedah, Malaysia. Assoc. Prof. Azham Hussain is a member of the US-based Institute of Electrical and Electronic Engineers (IEEE), and actively involved in both IEEE Communications and IEEE Computer societies. Azham is published in the areas of software evaluation and testing, user behaviours, group collaboration, and ubiquitous and mobile technology design. He has authored and co-authored more than 100 refereed technical publications, served as reviewer and referee for refereed journals and conferences on computing as well as the examiner for more than twenty doctoral and postgraduate scholars in his research areas. Email: azham.h@uum.edu.my Article submitted 2020-01-19. Resubmitted 2020-03-03. Final acceptance 2020-03-03. Final version published as submitted by the authors 56 http://www.i-jim.org https://doi.org/10.1145/1138929.1138952 https://doi.org/10.1145/1138929.1138952 https://doi.org/10.1177/1541931213571104 https://doi.org/10.4236/jsea.2014.712092 https://doi.org/10.25300/MISQ/2015/39.2.08 https://doi.org/10.25300/MISQ/2015/39.2.08 mailto:hasnaseem84@gmail.com mailto:sumayyah1433@gmail.com azham.h@uum.edu.my