International Journal of Interactive Mobile Technologies (iJIM) – eISSN: 1865-7923 – Vol. 15, No. 15, 2021 Paper—Examining Users’ Concerns while Using Mobile Learning Apps Examining Users’ Concerns while Using Mobile Learning Apps https://doi.org/10.3991/ijim.v15i15.22345 Senanu Okuboyejo(*), Ooreofeoluwa Koyejo Covenant University, Ota, Ogun State, Nigeria sena.okuboyejo@covenantuniversity.edu.ng Abstract—Mobile learning applications (apps) are increasingly and widely adopted for learning purposes and educational content delivery globally, especially with the massive means of accessing the internet done majorly on mobile handheld devices. Users often give feedback on use, experience, and gen- eral satisfaction via the reviews and ratings provided on the various distribution platforms. The massive information offered through the reviews presents an opportunity to derive valuable insights. These insights can be utilized for multiple purposes by different stakeholders of these learning apps. This large volume of reviews creates significant information overload and reading through could be time-consuming. We analyze these user reviews by combining text mining techniques of topic modeling using Latent Dirichlet Algorithm (LDA). These techniques identify inherent topics in the reviews and variables of user satisfaction while engaging the apps. The content analysis reveals the importance of videos (multimedia) and downloads as integral parts of learning apps. The the- matic analysis identifies features under these headings – financial, technical and design. Going by the values derived from these integral components and features of learning apps, it is worthwhile for app developers to improve on these to create a rewarding learning experience. Keywords—mobile learning, mobile apps, sentiment analysis, topic modeling, user reviews 1 Introduction Smartphones, being a form of handheld devices, have mobile apps that are install- able according to user preference; mobile apps are software applications developed for mobile devices. There are different categories of mobile apps downloadable from the respective distribution platforms, Google play store for android phones and App Store for iOS devices. Users’ feedbacks on these apps appear as reviews on the different platforms for the developers and other interested stakeholders. Hence, user reviews and ratings are essential variables in the development and upgrade of mobile apps. When requests made through user reviews are addressed with utmost satisfaction, the app rating will be equally high [1–3]. Online store reviews are free and fast crowd feedback mechanisms that developers can use as a backlog for the development process [2]. iJIM ‒ Vol. 15, No. 15, 2021 47 Paper—Examining Users’ Concerns while Using Mobile Learning Apps Mobile learning is a two-way communication process between educators and students via various learning facilitators and mobile devices, not confined to time and space barriers [4]. It has been an effective way to support and deliver knowledge to distance learners and part-time students. Mobile learning, characterized by its person- alized and interactive approach to learning, provides “learning at your own pace” as it maximizes the use of all forms of handheld (mobile) devices. These apps differ in their context and scope; some are multi-disciplinary, while some are specific in their content. MOOC (Massive open online courses) primarily have web platforms accom- panied by their mobile app versions, presenting a comprehensive coverage for users. For institutions that have a MOOC platform, the expected benefits include extend- ing the institution’s reach to a broader audience, building, maintaining the institution’s brand, and using MOOCs as a potential source of revenue [5]. User feedback is essential to the continuous development, maintenance, and evolu- tion of software products. User reviews are feedback or requests which can be (i) bugs or issues that need to be fixed [6], (ii) summaries of the user experience with specific features [7] , (iii) requests for enhancements [8], and (iv) ideas for new features [6], [9]. Maleej et al. in [10], [11] also stated four significant types of user reviews in bug reports, feature requests, user experiences, and ratings. User satisfaction could also be about how users are comfortable using the application, how they enjoyed exploring the application, and their overall satisfaction with it [12]. User reviews are significant in the life of an app; developers depend on these reviews to obtain feedback from users, which would inform the next stage of the app. These reviews are voluminous. As reported in an empirical study by [6], mobile apps received approximately 23 reviews per day and those popular apps, such as Facebook, received on average 4,275 reviews per day. These reviews appear in the form of unstructured text that is difficult to parse and analyze. Thus, developers and analysts are faced with the enormous task of reading large amounts of textual data to understand users. In the opinion of [13], a classification model for user reviews could allow developers to (i) filter relevant information in user reviews, (ii) understand more quickly the software maintenance tasks to apply, and, consequently, (iii) be more responsive to users’ requests. Specifically, topic models can be used to cluster sentences, for example, grouping together sentences of an identified category related to the same functionality of a given app. User satisfaction is an essential context in information systems as it checks for the positive impact on users. User satisfaction with other metrics such as system use, orga- nizational performance, and user decisional performance measures the effectiveness of information systems [14]. Extensive research has shown that user satisfaction with apps has a positive impact on customer loyalty, intention to continue to use the apps, willing- ness to pay for and recommend the app to others for use [5], [15–18]. As highlighted in [19], perceived benefit, which entails perceived enjoyment and perceived usefulness, has a significantly positive relationship with user satisfaction. The large volume of online reviews results in significant information overload; reading through all the reviews is time-consuming while making it difficult to draw conclusions and make informed decisions [20]. The reviews and ratings are sig- nificant to improving software quality and addressing missing application features that aid user experience. It is challenging to discover underlying topics from many online reviews and rate them against these topics [21]. Hence, app developers are 48 http://www.i-jim.org Paper—Examining Users’ Concerns while Using Mobile Learning Apps challenged with having a first glance knowledge of users’ opinions from the massive text available in the app stores. Previous works have covered several areas using dif- ferent content and text analysis techniques, sentiment analysis, and deriving insights from user reviews for stakeholders. The use of topic modeling method in supple- menting these techniques will discover features mentioned in user reviews of learn- ing apps for software developers towards software maintenance and evolution. Topic modeling is a type of statistical modeling for finding the abstract “topics” that occur in a collection of documents. Latent Dirichlet Allocation (LDA) is an example of a topic model technique used to classify text in a document to topics. It builds a topic per document model and words per topic model, modeled as Dirichlet distributions. Topic modeling as a statistical method will be applied to the extracted user reviews to classify the texts. Previous research in learning technologies and interventions have investigated the adoption and usage of mobile technologies for learning [22–25] and analyzed the content of mobile learning applications especially for children [23], [26]. Studies have also identified the absence of scientific standards and benchmarks in app design and development [27–29]. While extensive research has been carried out and is ongoing, there is little and insufficient research evidence to confirm the effectiveness and use- fulness of these apps. At the same time, to the best of our knowledge, there is none that reviews of MOOCs apps. Analyzing user reviews will provide actionable insights into the desirable features, benefits, issues, and limitations of these apps. This paper evalu- ates user reviews of 10 top MOOCs apps on Google Play and App Store. We used topic modeling to gain insights into the desirable features of these apps that will influence usage and determine the effectiveness of these apps. We analyzed the reviews classified as “negative” based on the user rating. We conclude with recommendations and sugges- tions for improvement for future updates and design. 2 Methodology In this section, we discuss the various approaches adopted in developing the classification model. We extracted the reviews through a Python-based web scrapping tool. These sets of reviews became the input data for this study. The user reviews were preprocessed to include stop word removal, lemmatization, stemming, part-of-speech tagging (POS), and phrase modeling (n-grams). Subsequently, we transformed the review texts to a standard form in lower case (case folding) and removed all non- text reviews, e.g., emoticons and symbols, etc. Upon completion of the preprocessing stages, the texts are put through the LDA to identify inherent topics that would deter- mine the classification of the subjects in the reviews. Using LDA as the topic model- ing algorithm in this work is based on the premise that the review dataset serves as training data for the Dirichlet distribution in the document-topic distributions. A sample is derived from the Dirichlet distribution, and progress in the analysis is made hereafter. However, since we have a dataset, it allows an efficient and easy-to-use topic model. Also, hidden semantic structures in the body of the review document are discovered with the LDA generated through unsupervised learning. The LDA also comes with its examining and evaluation metric, which is the coherence score calculation. iJIM ‒ Vol. 15, No. 15, 2021 49 Paper—Examining Users’ Concerns while Using Mobile Learning Apps Topic Coherence scores a single topic by measuring the degree of semantic simi- larity between high-scoring words in the topic. High coherence scores show that there is an increased coherence (semantic similarity) within the words categorized in a topic [30], [31]. Topics derived in the processing stage are visualized in diagrams and charts. Python modules such as Word Cloud and pyLDAvis in the Genism package help with the visualizations. Word Cloud gives a word-frequency distribution, where the words with the highest frequency show in big, bold letters in the entire corpus. The pyLDAvis module also provides an inter-topic distance map which gives a global topic overview by distributing the topics into four quadrants. These quadrants depict topic closeness or similarity through the words identified under each topic. Word Chart is another visual- ization that gives the top 10 common words in a bar chart with the frequency of occur- rence. Figure 1 shows the methodology workflow discussed above. Analysis and Classification User reviews extraction from app stores Text pre- processing tasks Lemmatization and Stemming Lowercasing Tokenization Part of Speech tagging Topic Identification Semantic Analysis and Classification Training Model Semantic Analysis and Classification Fig. 1. Methodology workflow 3 Results The data used in this work were extracted from Google Play Store and Apple Appstore (Nigeria). A total of 19709 reviews were extracted from the following apps: Udemy, Coursera, eDx, Khanacademy, Pluralsight, Linkedinlearning, Lynda, Skill- share. These apps are highly rated (top learning apps) in both play stores. The mobile apps selected for use in this work equally exist as Massive Open Online Courses (MOOCs) platforms. Only English reviews were considered in this study. The prepro- cessing task resulted in 4284 reviews for further analysis. A sample of the extracted review features is presented in Table 1. A screenshot of the result from the prepro- cessing stage – case folding, stemming, and lemmatization is presented in Figure 2. Each record in the dataset was labeled using the star ratings given on the apps; hence, star ratings “1” and “2” represent negative reviews, “3” represent neutral or unbiased reviews, while “4” and “5” ratings represent positive reviews. This labeling approach is also adopted by [32–34]. By count, there were 14005 positive, 4285 negative, and 1419 neutral reviews. Since our research focuses on identifying the variables of user satisfaction, the negative reviews are of importance in achieving this. Hence, the negative reviews are further analyzed for sentiments and topics to identify user satisfaction variables. 50 http://www.i-jim.org Paper—Examining Users’ Concerns while Using Mobile Learning Apps Table 1. Sample of the extracted reviews S/N App Rating Review Text 1. Udemy 1 “I use Udemy on iPod and ….” 2. Udemy 2 “No matter what I do, I can’t seem to download…” 3. Udemy 1 “App hangs while downloading…” Fig. 2. Screenshot of an output of preprocessing We run the LDA model on the preprocessed negative reviews dataset. For optimi- zation, the number of topics is set to 5 with an optimal coherence score of “0.54”. The LDA model is built with the variables stated and set the number of topics to 5. The results of the LDA modeling highlight the keywords that are similar enough to be classified under the same topic. The topic naming is done manually by inferring from the keywords ranked together. From the results, we have topics numbered as 0,1,2,3,4; the numbers before the words indicate the weight such words hold within the topic classification. Figure 3 gives the results of the LDA topic model showing the keywords that make up each topic and assigned weight within the topic. Fig. 3. Screenshot of identified topics and keywords iJIM ‒ Vol. 15, No. 15, 2021 51 Paper—Examining Users’ Concerns while Using Mobile Learning Apps Visualizing a document corpus is valuable for graphically illustrating and generating images from large datasets. The identified topics are visualized, and the visualization for Topic 1 is presented in Figure 4, while Figure 5 gives the Word cloud visualization of frequent tokens the overall negative review dataset. Fig. 4. Visualization of topic 1 showing the intertopic distance map and topic-term frequency Fig. 5. Word Cloud visualization showing the negative word distribution in dataset 52 http://www.i-jim.org Paper—Examining Users’ Concerns while Using Mobile Learning Apps 4 Discussion The significance of this work focuses on extracting useful information from a large body of text. The large body of text being user reviews submitted in mobile digital distribution platforms and the mobile applications considered are mobile learning apps. User feedback contains inherent insights within the body of text in the different app dis- tribution platforms important to app developers and vendors. Analyzing these reviews will provide requirements for targeted and focused app improvement and modification. Learning app users value “videos” while engaging content. Research has shown the importance of videos (multimedia) as an effective information delivery and educational tool during learning [35–37]. Videos allow for easy memory recall and knowledge retention, facilitate ubiquitous learning, have a wider reach and acceptance, and cater to different learners. Our research justifies this claim as the highest occurring token in the reviews is “video,” followed closely by “work” (not functional), “download,” and “load. Some of the challenges identified by users include: • small video size (aspect ratio) – “…The only issue with the app is that video does not go full screen. Unreasonable top big header remains and does not stretch edge to edge”. “…Wish full screen used all the screen space on edge-to-edge screens”. • poor video quality/speed – “Needs support for 2x video speed. It’s impossible to watch video tutorials without that basic setting”. “…button part of the video crop off, no video speed setting”. “…I love your courses but often want to watch away from keyboard on mobile. However, the videos don’t seem to be optimised to buffer cleverly, and I end up with a stuttering stop/start video or watching with audio only switched on, or closing the entire app. I would love to have the option of controlling video quality - just like on the desktop website....”. • Video resolution too high to stream thereby consuming more data (cost) - “…The content looks promising; however, videos take a lot of time to buffer and then stream. It is usually running in minutes...”. “way too much buffering. I don’t have this problem with any other streaming app. Only LinkedIn learning app. The functionality is fantastic but every 30 seconds I have to wait 15 seconds for at the buffer. And there is no option available to change the quality of the stream…”. Going by the value derived from this integral component of learning apps, it is worthwhile for app developers to improve on this to create a rewarding learning expe- rience. The identified topics in the reviews after analysis, modeling, and processing show different variables related to user satisfaction as analyzed using LDA. It is crucial to infer from and utilize the insights identified in the reviews given by current and past users of the apps; the information derived can help retain existing users and attract potential users when the negative issues are fixed. iJIM ‒ Vol. 15, No. 15, 2021 53 Paper—Examining Users’ Concerns while Using Mobile Learning Apps Our analysis of negative reviews identified desirable features to aid user satisfac- tion while using mobile learning apps. These features vary along the lines of pay- ments, experience, features etc., and are consistent with the findings in [34], [38]. The topics are discussed under the following broader headings – Financial, Technical and Design issues: 4.1 Financial issues The reviews identified under this topic speak to challenges and feedback on payment facilities offered on the apps, such as subscription and in-app purchases. It ranges from subscription made and not reflecting, in-app purchases of and payment for specific and introductory courses on the apps which are expensive, card payment issues, exchange rates, country/location-specific payment restrictions. Users give some feedback when payment is involved especially, in cases where disparities are observed. For any and every payment made, mobile app users expect to receive total value for all these. Sam- ple comments include: • “…Charged my account for Premium membership after my trial period without giv- ing any prior notices. I barely used the app and haven’t taken any more classes since I first signed up………”. • “ …I use this app to make easy and secure payments. But unfortunately I cannot use the coupon codes I have for the courses and ofc I’m not willing to pay 20 times the price of 10 bucks course so I have to wait until you go on a sale. It’s unac- ceptable. Your app is essentially useless without that feature…”. • Both time I purchased through the app have been a horrible experience. iOS charges me and once the payment clears I don’t have my purchased course. It takes 24 hours or more to straighten the mess out. I am still waiting to receive my latest purchase…” 4.2 Technical issues – non-functional features App users gave feedback on new features added or modification of existing fea- tures not functioning as expected or not suitable for the purpose intended. Typically, all new features should be tested and checked for best fit, suitability, and interoper- ability. Issues would have been raised and fixed, if any, at the point of testing. How- ever, due to time, cost, and other constraints, some of these new features are not duly checked before pushing out to the public for use resulting in complaints on the distri- bution platforms. Users request “fix” features on the apps via the reviews submitted. These requests speak to how the features will aid or enhance user satisfaction on the apps, require some modifications or reset. Mobile app stakeholders harvest these fix requests on features to guide developers and app vendors in their evolution and maintenance practices. Some other reviews are aspect/feature-based, highlighting features that make the user experience not exciting or smooth. Here, users state their challenges with specific or general parts while using the apps, which hinders total contentment with the app’s 54 http://www.i-jim.org Paper—Examining Users’ Concerns while Using Mobile Learning Apps behavior. From these reviews, app stakeholders can identify these aspects to work on them accordingly. Sample comments include: • “…Cannot login with password via Android app. Website login works…” • “I can’t open subjects up after an update.. can’t download a whole subject after an update…” • “…The app is not working since last update. It simply opens, but any keys doesn’t respond. Plz help me out…” • “…I can’t open subjects up after an update.. can’t download a whole subject after an update…” 4.3 Design issues – media defects Users in their reviews also comment on some aspects of the mobile learning apps related to media such as video, audio, resolution, size, video orientation (landscape or portrait) based on device compatibility and aspect ratio on the mobile learning apps. With this information, app owners understand and identify the specific media issues that users face while using the apps. While some might be a function of the type of device that users have and interact with the apps on, some might be the responsibility of the app developers, which requires some further attention and fix. Sample comments include: • “…The new update is terrible. I am trying to take practice tests and the interface is jumbled. The counter overlaps the questions and the visuals are all stacked on each other. Please patch ASAP. Running an iPad with the latest iOS updates…” • “…I find at least 3 problems with this app. First, a video title is always hovering on top and covering the area where applications like MS Excel plus VBA editor, Word and others have their own menu options, thus can’t see directly what the instructor is clicking. Second, once I push “play”, the video controls keeps hovering over and waits for a second touch to disappear. Controls disappearing after 2–3 secs would be nice. Third, I see no option for video resolution seeting. I use an LG-G6 phone…” • “…The video screen is small. I think it’s meant just for mobile phones, not for Chromebooks or large screens…” • “… app on tablet is nearly unusable. Constant issues with orientation being stuck in portrait mode, then when I finally get it to work in landscape, video is displayed at the top of the screen with a huge black bar at the bottom (needs centering vertically). The courses are great, but accessing them, not so much….” 5 Conclusion This work contributes to the aspect of information system that seeks to improve user satisfaction. The results from this study are significant for learning apps development and maintenance. End-users widely use mobile apps and give feedback in the form of reviews; hence, the need to infer how to utilize this feedback for further improvements. iJIM ‒ Vol. 15, No. 15, 2021 55 Paper—Examining Users’ Concerns while Using Mobile Learning Apps The developed model can be deployed to other domains with mobile app platforms and a high influx of user reviews. This act would help identify and derive substantial insights that can positively influence app maintenance and evolution. Consequently, future research can examine the potential relationships between the emotions and tones of the review content and the numerical user ratings and determine the extent of the impact of these variables (design, cost, and technical issues) on continuous usage and retention of the apps. 6 References [1] C. Gao, J. Zeng, M. R. Lyu, and I. King, “Online app review analysis for identifying emerging issues,” 2018, pp. 48–58. https://doi.org/10.1145/3180155.3180218 [2] F. Palomba et al., “Crowdsourcing user reviews to support the evolution of mobile apps,” J. Syst. Softw., vol. 137, pp. 143–162, Mar. 2018. https://doi.org/10.1016/j.jss.2017.11.043 [3] F. Palomba et al., “User reviews matter! Tracking crowdsourced reviews to support evolu- tion of successful apps,” in 2015 IEEE International Conference on Software Maintenance and Evolution (ICSME), 2015, pp. 291–300. https://doi.org/10.1109/ICSM.2015.7332475 [4] S. K. Alzu’Bi and S. Hassan, “Factor Affecting the Success of Mobile Learning Implementation: A study of Jordanian Universities,” Asian J. Inf. Technol., vol. 15, no. 1, pp. 113–121, 2016. [5] N. Chen et al., “AR-Miner: Mining Informative Reviews for Developers from Mobile App Marketplace,” vol. 7, pp. 767–778, 2014. https://doi.org/10.1145/2568225.2568263 [6] D. Pagano and W. Maalej, “User feedback in the appstore: An empirical study,” in 2013 21st IEEE International Requirements Engineering Conference, RE 2013 - Proceedings, 2013, pp. 125–134. https://doi.org/10.1109/RE.2013.6636712 [7] E. Guzman and W. Maalej, “How Do Users Like This Feature? A Fine Grained Sentiment Analysis of App Reviews,” in IEEE 22nd international requirements engineering conference, 2014, pp. 153–162. https://doi.org/10.1109/RE.2014.6912257 [8] C. Iacob and R. Harrison, “10th Working Conference on Mining Software Repositories (MSR) : proceedings, May 18-19, 2013, San Francisco, CA, USA,” in Proceedings of the 10th Working Conference on Mining Software Repositories, 2013, p. 438. [9] L. V. G. Carreno and K. Winbladh, “Analysis of user comments: An approach for software requirements evolution,” in 2013 35th International Conference on Software Engineering (ICSE), 2013, pp. 582–591. https://doi.org/10.1109/ICSE.2013.6606604 [10] W. Maalej and H. Nabil, “Bug Report, Feature Request, or Simply Praise? On Automatically Classifying App Reviews,” in IEEE 23rd international requirements engineering conference, 2015. https://doi.org/10.1109/RE.2015.7320414 [11] W. Maalej, Z. Kurtanović, H. Nabil, and C. Stanik, “On the automatic classification of app reviews,” Requir. Eng., vol. 21, no. 3, pp. 311–331, Sep. 2016. https://doi.org/10.1007/ s00766-016-0251-9 [12] K. Segaran, A. Z. M. Ali, and T. W. Hoe, “Usability and User Satisfaction of 3D Talking-head Mobile Assisted Language Learning (MALL) App for Non-native Speakers,” Procedia - Soc. Behav. Sci., vol. 131, pp. 4–10, May 2014. https://doi.org/10.1016/j.sbspro.2014.04.069 [13] S. Panichella, A. Di Sorbo, E. Guzman, C. A. Visaggio, G. Canfora, and H. C. Gall, “How can i improve my app? Classifying user reviews for software maintenance and evolution,” in 2015 IEEE 31st International Conference on Software Maintenance and Evolution, ICSME 2015 - Proceedings, pp. 281–290, 2015. https://doi.org/10.1109/ICSM.2015.7332474 56 http://www.i-jim.org https://doi.org/10.1145/3180155.3180218 https://doi.org/10.1016/j.jss.2017.11.043 https://doi.org/10.1109/ICSM.2015.7332475 https://doi.org/10.1145/2568225.2568263 https://doi.org/10.1109/RE.2013.6636712 https://doi.org/10.1109/RE.2014.6912257 https://doi.org/10.1109/ICSE.2013.6606604 https://doi.org/10.1007/s00766-016-0251-9 https://doi.org/10.1007/s00766-016-0251-9 https://doi.org/10.1016/j.sbspro.2014.04.069 https://doi.org/10.1109/ICSM.2015.7332474 Paper—Examining Users’ Concerns while Using Mobile Learning Apps [14] R. R. Nelson, P. A. Todd, and B. H. Wixom, “Antecedents of information and system qual- ity: An empirical examination within the context of data warehousing,” J. Manag. Inf. Syst., vol. 21, no. 4, pp. 199–235, 2005. https://doi.org/10.1080/07421222.2005.11045823 [15] M. Kim, Y. Chang, M. C. Park, and J. Lee, “The effects of service interactivity on the satisfaction and the loyalty of smartphone users,” Telemat. Informatics, vol. 32, no. 4, pp. 949–960, Nov. 2015. https://doi.org/10.1016/j.tele.2015.05.003 [16] X. Peng, Y. (Chris) Zhao, and Q. Zhu, “Investigating user switching intention for mobile instant messaging application: Taking WeChat as an example,” Comput. Human Behav., vol. 64, pp. 206–216, Nov. 2016. https://doi.org/10.1016/j.chb.2016.06.054 [17] L. Zhao and Y. Lu, “Enhancing perceived interactivity through network externalities: An empirical study on micro-blogging service satisfaction and continuance intention,” in Decision Support Systems, vol. 53, no. 4, pp. 825–834, 2012. https://doi.org/10.1016/ j.dss.2012.05.019 [18] T. Zhou and Y. Lu, “Examining mobile instant messaging user loyalty from the perspec- tives of network externalities and flow experience,” Comput. Human Behav., vol. 27, no. 2, pp. 883–889, Mar. 2011. https://doi.org/10.1016/j.chb.2010.11.013 [19] G. Wang and J. Song, “The relation of perceived benefits and organizational supports to user satisfaction with building information model (BIM),” Comput. Human Behav., vol. 68, pp. 493–500, Mar. 2017. https://doi.org/10.1016/j.chb.2016.12.002 [20] N. Genc-Nayebi and A. Abran, “A systematic literature review: Opinion mining studies from mobile app store user reviews,” J. Syst. Softw., vol. 125, pp. 207–219, Mar. 2017. https://doi.org/10.1016/j.jss.2016.11.027 [21] X. Dai, I. Spasic, and F. Andres, “A Framework for Automated Rating of Online Reviews Against the Underlying Topics,” 2017. https://doi.org/10.1145/3077286.3077291 [22] M. Kalogiannakis and S. Papadakis, “Evaluating pre-service kindergarten teachers’ inten- tion to adopt and use tablets into teaching practice for natural sciences,” Int. J. Mob. Learn. Organ., vol. 13, no. 1, pp. 113–127, 2019. https://doi.org/10.1504/IJMLO.2019.096479 [23] P. Dorouka, S. Papadakis, and M. Kalogiannakis, “Tablets and apps for promoting robotics, mathematics, STEM education and literacy in early childhood education,” Int. J. Mob. Learn. Organ., vol. 14, no. 2, pp. 255–274, 2020. https://doi.org/10.1504/IJMLO.2020.106179 [24] B. Eschenbrenner and F. Fui-Hoon Nah, “Mobile technology in education: uses and bene- fits,” Int. J. Mob. Learn. Organ., vol. 1, no. 2, pp. 159–183, 2007. https://doi.org/10.1504/ IJMLO.2007.012676 [25] A. A. Economides and A. Grousopoulou, “Mobiles in education: students’ usage, preferences and desires,” Int. J. Mob. Learn. Organ., vol. 4, no. 3, pp. 235–252, 2010. https://doi.org/ 10.1504/IJMLO.2010.033553 [26] M. Meyer et al., “How educational are ‘educational’ apps for young children? App store content analysis using the Four Pillars of Learning framework,” 2021. https://doi.org/10.10 80/17482798.2021.1882516 [27] S. R. Stoyanov, L. Hides, D. J. Kavanagh, O. Zelenko, D. Tjondronegoro, and M. Mani, “Mobile app rating scale: a new tool for assessing the quality of health mobile apps.,” JMIR mHealth uHealth, vol. 3, no. 1, p. e27, Mar. 2015. https://doi.org/10.2196/mhealth.3422 [28] C. Caldeira, Y. Chen, L. Chan, V. Pham, Y. Chen, and K. Zheng, “Mobile apps for mood tracking: an analysis of features and user reviews,” AMIA ... Annu. Symp. proceedings. AMIA Symp., vol. 2017, pp. 495–504, 2017. [29] N. Aslam, W. Y. Ramay, K. Xia, and N. Sarwar, “Convolutional Neural Network BasedClas- sification of App Reviews,” IEEE Access, 2020. [Online]. Available: https://ieeexplore.ieee. org/stamp/stamp.jsp?arnumber=9217455. [Accessed: 30-Apr-2021]. iJIM ‒ Vol. 15, No. 15, 2021 57 https://doi.org/10.1080/07421222.2005.11045823 https://doi.org/10.1016/j.tele.2015.05.003 https://doi.org/10.1016/j.chb.2016.06.054 https://doi.org/10.1016/j.dss.2012.05.019 https://doi.org/10.1016/j.dss.2012.05.019 https://doi.org/10.1016/j.chb.2010.11.013 https://doi.org/10.1016/j.chb.2016.12.002 https://doi.org/10.1016/j.jss.2016.11.027 https://doi.org/10.1145/3077286.3077291 https://doi.org/10.1504/IJMLO.2019.096479 https://doi.org/10.1504/IJMLO.2020.106179 https://doi.org/10.1504/IJMLO.2007.012676 https://doi.org/10.1504/IJMLO.2007.012676 https://doi.org/10.1504/IJMLO.2010.033553 https://doi.org/10.1504/IJMLO.2010.033553 https://doi.org/10.1080/17482798.2021.1882516 https://doi.org/10.1080/17482798.2021.1882516 https://doi.org/10.2196/mhealth.3422 Paper—Examining Users’ Concerns while Using Mobile Learning Apps [30] R. Annisa and I. Surjandari, “Opinion Mining on Mandalika Hotel Reviews Using Latent Dirichlet Allocation,” Procedia Comput. Sci., vol. 161, pp. 739–746, 2019. https://doi.org/ 10.1016/j.procs.2019.11.178 [31] J. Qiang, Z. Qian, Y. Li, Y. Yuan, and X. Wu, “Short text topic modeling techniques, appli- cations, and performance: A survey,” arXiv. arXiv, 13-Apr-2019. https://doi.org/10.1109/ TKDE.2020.2992485 [32] M. Bahja and M. Lycett, “Identifying Patient Experience from Online Resources via Senti- ment Analysis and Topic Modelling,” 2016. https://doi.org/10.1145/3006299.3006335 [33] J. Gebauer, Y. Tang, and C. Baimai, “User requirements of mobile technology: Results from a content analysis of user reviews,” in Information Systems and e-Business Management, 2008, vol. 6, no. 4, pp. 361–384. https://doi.org/10.1007/s10257-007-0074-9 [34] T. Olagunju, O. Oyebode, and R. Orji, “Exploring Key Issues Affecting African Mobile eCommerce Applications Using Sentiment and Thematic Analysis,” IEEE Access, vol. 8, pp. 114475–114486, 2020. https://doi.org/10.1109/ACCESS.2020.3000093 [35] D. Zhang and L. Zhou, “Enhancing e-Learning with interactive multimedia,” Inf. Resour. Manag. J., vol. 16, no. 4, pp. 1–14, 2003. https://doi.org/10.4018/irmj.2003100101 [36] D. Zhang, L. Zhou, R. O. Briggs, and J. F. Nunamaker, “Instructional video in e-learning: Assessing the impact of interactive video on learning effectiveness,” Inf. Manag., vol. 43, no. 1, pp. 15–27, Jan. 2006. https://doi.org/10.1016/j.im.2005.01.004 [37] M. Ljubojevic, V. Vaskovic, S. Stankovic, and J. Vaskovic, “Using Supplementary Video in Multimedia Instruction as a Teaching Tool to Increase Efficiency of Learning and Qual- ity of Experience,” Int. Rev. Res. Open Distrib. Learn., vol. 15, no. 3, pp. 275–291, 2014. https://doi.org/10.19173/irrodl.v15i3.1825 [38] O. Oyebode, A. Felwah, and R. Orji, “Using Machine Learning and Thematic Analysis- Methods to Evaluate Mental Health AppsBased on User Reviews,” IEEE Access, vol. 8, pp. 111141–111158, 2020. https://doi.org/10.1109/ACCESS.2020.3002176 7 Authors Senanu R. Okuboyejo is an Assistant Professor of MIS in the Department of Computer and Information Science, Covenant University, Nigeria. Her research inter- ests include Health Informatics, Technology Adoption and Use in LDCs, and Applied Machine Learning. She has publications spanning these interests. Dr. Okuboyejo is a member of the Association of Information Systems (AIS). Ooreofeoluwa I. Koyejo graduated with a first-class bachelor’s and master’s degrees in Management Information System (MIS) from Covenant University. She is an information security practitioner currently working as the information security man- ager with a fast-growing startup company in Lagos, Nigeria, where she maintains and manages the Information Security Management System (ISO27001 standard) within the organization. Article submitted 2021-02-27. Resubmitted 2021-05-26. Final acceptance 2021-06-08. Final version published as submitted by the authors. 58 http://www.i-jim.org https://doi.org/10.1016/j.procs.2019.11.178 https://doi.org/10.1016/j.procs.2019.11.178 https://doi.org/10.1109/TKDE.2020.2992485 https://doi.org/10.1109/TKDE.2020.2992485 https://doi.org/10.1145/3006299.3006335 https://doi.org/10.1007/s10257-007-0074-9 https://doi.org/10.1109/ACCESS.2020.3000093 https://doi.org/10.4018/irmj.2003100101 https://doi.org/10.1016/j.im.2005.01.004 https://doi.org/10.19173/irrodl.v15i3.1825 https://doi.org/10.1109/ACCESS.2020.3002176