PAPER MLUX: USABILITY AND USER EXPERIENCE DEVELOPMENT FRAMEWORK FOR M-LEARNING mLUX: Usability and User Experience Development Framework for M-Learning http://dx.doi.org/10.3991/ijim.v9i3.4446 A. Dirin1 and M. Nieminen2 1 Haaga-Helia University of Applied Science, Helsinki, Finland 2 Aalto University, ESPOO, Finland Abstract—in this paper, we propose the mLUX framework, a model based on the user-centered design (UCD) frame- work, which is specifically for the development of m- learning applications. We present the results of four case studies conducted to develop m-learning applications in which the proposed mLUX framework was applied. The main goal of the proposed mLUX framework is to ensure that the stakeholders, especially students, recognize that m- learning applications are learning media that fulfill their essential educational requirements. We begin by reviewing the literature on the contributions to mobile learning usabil- ity made in various conferences and journals from 2002 to 2010. This review helped identify and recognize the method- ology used to develop mobile learning applications during this period. Based on the literature review, as the empirical case studies, four m-learning applications are examined to demonstrate the performance of the proposed framework for the development of m-learning applications. We apply three distinct measurement criteria to assess the perfor- mances of the mLUX framework. This paper also argues that emotional factors, such as the user’s enjoyment, adjust- ability, and reliability, are significant design issues in m- learning. Index Terms—mobile learning usability, user experience, user-centered design I. INTRODUCTION Contemporary smartphones are complex devices that use a wide range of advanced technologies. In addition to their communication capabilities, these devices are able to execute many other functions and capabilities, even multi- tasking. M-learning applications are examples of multi- tasking applications that enable learners to learn and ac- cess learning resources regardless of time and place both during classroom training and in off-site learning. The functionality of m-learning applications is expand- ing to wide range of new possibilities in addition to im- proved video, audio, animation, and synchronization ca- pabilities in internal or external applications, such as cal- endar and email. As Traxler [1] stated, mobile learning using handheld computers is an obviously immature ped- agogical technology; nevertheless, it is developing rapid- ly. However, mere technology is not enough. There is an obvious need to design these applications so that they are user-centered way and have high usability. However, is usability sufficient to motivate users to use the mobile learning application regularly? In the era of thousands of mobile apps available in platform-specific application stores, m-learning applications must engage students in using them. This paper reviews the extant literature to determine how the expanding possibilities for m-learning develop- ment have been addressed in previous studies. Additional- ly, this paper proposes enhancements to the development framework of user-centered m-learning applications. Changes in students’ media consumption present chal- lenges to learning activities. Therefore, it is essential that students be emotionally engaged with the learning appli- cation, which is achieved by making the application en- joyable to use. Students then will be motived to use the application. The m-learning application should be able to compete with other entertainment applications on the smart phone [2], such as games, for the students’ atten- tion. II. OBJECTIVE AND SCOPE The objective of this paper is to present the mLUX framework and demonstrate how it has been used to pro- duce acceptable, usable, and error-free m-learning appli- cations. A qualitative approach [3] was used to create the concept and assess of the m-applications usability and performance. In order to increase our understanding of the develop- ment process of m-learning and activities, we use the case study approach [4][5][6]. Regarding technology, from the broad range of different information appliances that can be used in m-learning (e.g., e-readers, tablet computers, and laptops), this study is limited to smartphones. III. METHODS AND DATA This paper is based on review of the literature on mo- bile learning usability. The review collected data mainly from papers published in m- and e-learning journals and conferences from 2002 to 2010. This paper also refers to a study [2] on the contempo- rary expectations of students regarding learning environ- ments that are facilitated by contemporary smartphones. The study was conducted as a comparative usability eval- uation [7], and it used the “thinking-aloud” method. Moreover, this study uses data gathered from four suc- cessful m-learning development projects that were con- ducted according to the mLUX framework process. These four projects are the case studies examined in this paper. The framework focuses on usability and the user’s ex- perience throughout the development process [8]. In the process of designing the mLUX framework, the designers collected the user data, analyzed it, and proposed a design based on processed data. In all phases of the design and development, the lead users [9] were directly involved. In developing the mLUX framework, we utilized appropriate iJIM ‒ Volume 9, Issue 3, 2015 37 PAPER MLUX: USABILITY AND USER EXPERIENCE DEVELOPMENT FRAMEWORK FOR M-LEARNING methods in each development phase, the details of which are described in the following sections. IV. RELATED RESEARCH In addition to the ISO 9241-11’s definition of usability, Nielsen[10] defined it as a quality attribute that assesses how easy user interfaces can be used. The definition of usability is based on how efficiently users are able to learn to use the system and how easy it is to memorize the es- sential interaction steps, perform tasks without errors, demonstrate error-free activity, and experience satisfaction with the use the application. In the field of human- computer interaction (HCI), researchers have developed sets of usability guidelines to help designers produce usa- ble systems [11]. The development of mobile learning usability has evolved over time through different paths, which are addressed briefly in the following paragraphs. The usability of mobile learning applications often re- quires the assessment of multi-tier usability. Vavoula [12] considered mobile learning applications on three different levels: the micro level (concerned with the evaluation pf usability), the meso level (concerned with the evaluation of learning), and the macro level (focused on the evalua- tion of integration with the existing educational contexts). The importance of usability for mobile applications was noted as soon as mobile phones became capable of execut- ing third party applications. In the early 2000s, mobile devices had small screens, low processing power, and low memory capacity; moreover, their data exchange rates were expensive. However, solutions to these issues have increased the use and development of m-learning applica- tions, which have fostered the importance of usability of mobile learning applications has increased [13][14][15][16][17]. The analysis of published papers on m-learning from 2002 to 2010 revealed that seminal work on usability assessments was produced during this period. Initially, the concept of m-learning was a research topic confined to research at universities . However, this trend rapidly changed as phones became smarter, and the cost of mobile data communication dropped significantly. This trend is also reflected in studies on m-learning usability from 2002 to 2005. Kukulska-Hulme [18][19] inspected a dozen mobile learning applications from the perspecitive of usability, which led to the identification of failures in the general usability of mobile learning applications. During this period (2002–2005), methods used to assess general usability, such as heuristic evaluation and usability testing, were applied. This trend, however, changed in 2004. Usability as- sessments are no longer sufficient. An increasing number of methods and guidelines for developing m-learning applications have been proposed. These include Syvänen’s [20] recommendations on pedagogical usability as well as Keinonen’s [21] argument for the changing role of mobile devices. As they become more personal and portable, it becomes increasingly difficult to predict what users might do. Kukulska-Hulme [18] recommended that education practitioners utilize new technologies in educational set- tings. Soloway [22] focused on learner-centered design, and Mostakhdemin-Hosseini [23] investigated a frame- work for a scenario-based mobile learning application. Pehkonen and Malliou [24][25] advocated a user-centered design in m-learning application development. Adaptive mobile learning application development was also pro- posed during this period by Vainio [26], Malliou [25], and Jäppinen [27]. This development was furthered by Seong’s guidelines for mobile learning usability, specifi- cally m-learning [28]. Koole’s FMRE framework [29] was based on the context of information. In 2007, Magal-Royo [30] and Nikkoi [26] pointed out that there were no specif- ic or suitable criteria to evaluate m-learning platforms. They recommended that traditional usability methods should be combined with in-situation methods in order to assess m-learning applications. Many attempts have been made to improve the quality of m-learning applications. Uden [31] utilized activity theory [32][33][34][35] to overcome the complexity of the relationships involved in designing a mobile learning environment. Uden utilized the activity theory by clarify- ing the purpose of the activity (i.e., exploration of the motives and goals of the activity system) and used it as the unit of analysis. The activity system is the unit of analysis, which gives a context and meaning to seemingly random individual events. Despite the transition in the focus of the research on us- ability of m-learning applications, there was little indica- tion of specific issues related to the factors in user experi- ence that deal with the emotional side of application usage and the engagement of users. Nielsen and Norman [36] defined user experience as something that encompasses all aspects of the end-user’s interaction with the company, including its services and its products. Hassenzahl [37] defined user experience as a consequence of the user’s internal state (e.g., predispositions, expectations, needs, motivation, mood, etc.), the characteristics of the designed system (e.g., complexity, purpose, usability, functionality, etc.), and the context (or the environment) within which the interaction occurs (e.g. organizational/social setting, meaningfulness of the activity, voluntariness of use, etc.). The case studies presented in this paper are based on Brusilovsky’s classification of educational components [38]. Brusilovsky classified the courses of virtual universi- ties into four main components: presentation, activities, communication, and administration. Presentation refers to functions that are related to the delivery of new materials. Activities refer to activities that students need to perform. Communications refers to interactions between teacher and students or between students. Finally, administration refers to activities related to registration, record keeping, and so on. V. USER-CENTERED DESIGN PRINCIPLES AND PROCESSES User-centered design (UCD) and the development of interactive systems and devices have gained increasing importance in service and product development [39]. Gould [40][41] argued that in order to design a usable system, users should be continuously involved in the de- velopment process and the design should be modified based on their feedback. User-centered design reduces costs and improves usability because it continually focus- es on the customer’s needs as early as possible in the de- velopment process. User requirements are the focus of all stages of development cycle. ISO 9241-210 [42] defined the human-centered design processes for interactive sys- tem design as follows: 1) cooperative design: designers and user involved in all stages; 2) participatory design: users’ occasionally participate in the design process; 3) 38 http://www.i-jim.org PAPER MLUX: USABILITY AND USER EXPERIENCE DEVELOPMENT FRAMEWORK FOR M-LEARNING contextual design: design based on the actual context. User-centered system design (UCSD) focuses on usability not only in the development process but also throughout the system’s life cycle [8]. Human-centered design [43] and interactive interaction design [44] are aspects of the same issue, but in practice, they are used synonymously for UCD [45]. Figure 1 presents Wallach and Scholtz User-Cenetered design activities [46]. A. Scope In the scoping phase, the goals and the constraints of the projects are defined. The goal of the design should be addressed in advance, such as designing a new function- ality for the existing application or redesigning the exist- ing functionality. Moreover, in this phase, the roles of the players are analyzed and a common ground between exist- ing product concept and research results is created. Scope in m-learning: Regarding the scoping activity in m-learning, there is a need to define the aim and role of the developed system. In addition to defining the roles of the players in the m- learning system, the analysis addresses the extent to which the m-learning system covers the learning activities by asking the following questions: 1) Is there a need to de- sign a standalone application? 2) Should the m-learning application enhance the existing e-learning platform? 3) Is the aim is to support the traditional classroom-based edu- cation offering? Similarly, the context of use of m- learning needs to be defined. This includes, for example, the analysis of where the application is used the most and the need for an internet connection. B. Analyse In the analysis phase, the main aim is to reveal the at- tributes of the user, such as the tasks that they currently or expect to handle in their future applications. This infor- mation is often collected through usability inspections or by various user-study methods. 1) Analyse in m-learning The students and teachers are the main users of the m- learning application. In the analysis, we learned the cur- rent ways that students and teachers use to deal with edu- cational tasks and their expectations of the forthcoming application. The results of the case studies revealed that the user-study methods, such as web questionnaires and semi-structured interviews, were adequate to collect in- formation from the participants. The web questionnaires enabled us to collect information about the ways in which students and teachers deal with their educational tasks. This method was preferable because the users could pro- vide the information at their convenience. The semi- structured interviews were used to learn about the partici- pants’ current and future expectations. Moreover, to analyze the collected data, we used tran- script coding and users and task analysis methods. Tran- script coding helped identify details about the actions and the environment in which these actions were often took place. C. Design The main goal of the design is to transform the findings into concepts and tangible prototypes. Design in m-learning: The concept of the potential application is presented as a learning activity scenario. A learning scenario describes the learning activities that the students and teachers carry out in specific real-life learning environments. For early feedback from the users, we asked students and teachers to review the scenario and share their remarks about the concept with designers. This yielded information about users’ further demands for the proposed concept of a mo- bile learning application. Finally, based on a revised sce- nario, a low and high fidelity prototype was designed and developed accordingly. D. Validate Validation concerns applying various usability methods to ensure that the design follows the initial requirements and supports the required functionalities. Validate in m-learning The specific validation of m-learning is carried out with students and teachers. They were asked to assess the low and high fidelity learning prototypes in a usability lab. Controlled validation enabled the explicit delivery of feedback from students and teachers to the developers. M- learning specific tasks included the following: teachers upload course and lecture material for students’ use; stu- dents are asked to view, download, and use the learning material; and students are asked to submit their assign- ments through the m-learning application. Nielsen’s heu- ristic evaluation guidelines were used to assess the con- sistency of the m-learning application. Heuristic evalua- tion was used to evaluate all applications that students and teachers used for learning purposes. This enabled the analysis and evaluation of the performance, usability, and user experience in the use of the combined m-learning application. Figure 1. User-centered design activities according Wallach and Scholz [46]. These activities are the basis of the mLUX model. iJIM ‒ Volume 9, Issue 3, 2015 39 PAPER MLUX: USABILITY AND USER EXPERIENCE DEVELOPMENT FRAMEWORK FOR M-LEARNING E. Deliver In the delivery activity, the implementation team han- dles the outcome of the design and the related documenta- tion. 1) Deliver in m-learning The design and development documentation of m- learning application consists of the following: 1) results and documents of user studies (i.e., students and teachers); 2) detailed list of user requirements (students and teach- ers); 3) the learning scenarios and the proposed learning prototype concepts; 4) the usability assessment test report. VI. THE MLUX MODEL: CONSTRUCTING A USER- CENTERED DESIGN MODULE FOR M-LEARNING In this section, we explore the mLUX, which is the framework used to develop the m-learning application. In addition to the model presented by Wallach and Scholz [46], the ISO 9241 standard (parts 11, 1998 and 210, 2010) is used to define the important characteristics of this framework, including the stakeholders involved in m- learning application development and the definition and analysis of the context of use. The proposed mLUX framework consists of three con- ceptual parts: 1) the role-players in the m-learning applica- tion; 2) the context of use of the m-learning application, and 3) the m-learning application development process. Figure 2 illustrates the mLUX framework. A. Actors and roles in M-learning applications Our previous studies [47] [ICAICTE, 2014] found that numerous role-players (e.g., students, teachers, adminis- trators, educational institutes, etc.) are involved in the design, development, and usage of m-learning applica- tions. It is essential to identify the needs and expectations of the role-players because they directly or indirectly influence the use of the application. The mLUX frame- work includes the direct involvement of the role-players in the application design and development phases. Subse- quently, the role-players are directly or indirectly involved in the design and development of the m-learning applica- tion. The students, teachers, and administrators are the key users of m-learning applications. The needs, expectations, and the motivations of these key players affect the features of the m-learning application For example, a teacher pro- vides course materials through uploading assignments. Similarly, students download the material, or they may submit their answers to the assignments through m- learning applications. Moreover, the administrative per- sonnel support students and teachers in their educational activities in various ways, such as guiding them through- out course registration activities and providing course certificates through the m-learning application. In addition, many invisible role-players, individuals, organizations, systems, and technical systems are involved in the design and development of m-learning applications. These invisible role-players are not the actual users of the application, but they have a significant influence on the design and usage of the m-learning application. The mobile learning application enhances the existing e-learning platforms, which are a significant role-player in the design and development of m-learning applications. Figure 2. The proposed framework for developing m-learning applica- tions (mLUX) Therefore, the designers have to determine the features that must be implemented in the m-learning application. The designers and developers of m-learning applica- tions are invisible role-players. They create the concept and select appropriate development technologies. Moreo- ver, the design and development of m-learning applica- tions often incorporates various experts, such as user study experts, concept designers, and software developers, whose design activity affects the overall m-learning appli- cation. The network operator’s role in the design and develop- ment of m-learning applications is that of an intermediary between the content provider (teachers) and the content users (students). Therefore, the network operator's con- straints and regulations, such as the format of the ex- change data, are compulsory in m-learning application development. The network operator is an invisible role- player and has a direct influence on the usage of the m- learning application. Therefore, network operators’ sup- port in m-learning content deliveries is vital, especially when the number of m-learning application subscrip- tions increases. Educational institutes are the primary agents in the practice and usage of m-learning applications, which mak- ing them a strong role-player in the design and develop- ment of m-learning applications. The educational insti- tutes’ demands and expectations in m-learning applica- tions need to be addressed properly. An important exam- ple is the influence of the cost of the m-learning utilization on the course. However, the institution’s decision to use m-learning is based on its overall institutional aims, its access to the infrastructure required by the m-learning application, and its appreciation of the pedagogical trends that support m-learning and [18]. In addition to providing financial support for m-learning applications and content development, educational institutes have the important role of promoting and encouraging their staff to use m- learning applications in their courses. Mobile device/terminal manufacturer. This invisible role-player is indirectly involved in formulating the usage and development m-learning application. M-learning applications target the mobile devices of students and teachers, but the latter affect the usage of the application. Despite some convergence, the technological features of 40 http://www.i-jim.org PAPER MLUX: USABILITY AND USER EXPERIENCE DEVELOPMENT FRAMEWORK FOR M-LEARNING the mobile devices still vary significantly. For example, some devices fully support HTML5 specifications, where- as others only partially support them. Therefore, the de- signers and developers have to consider the inconsisten- cies in technological and operating systems in the mobile devices used by the students and teachers. B. The context of the use of m-learning The main role-players of m-learning applications such as teachers and students carry out their educational tasks in different times and environments. Therefore, the mobile learning is not just about learning with a portable device, but across contexts[48]. These include the following con- texts. 1. The social context in this study refers to social acceptance of the m-learning application as a learning medium by students and teachers. In addition, knowledge transformation is a social process, which occurs when there is interaction among learners. M-learning provides social presence and awareness as the learner is situated in different learning contexts such as formal location e.g. classrooms, semi-formal locations e.g. libraries, and in- formal locations e.g. residences[49]. M-learning enables and supports communication between students and teach- ers regardless of the physical location. 2. The physical context refers to a non-human as- pect of the context, the accessibility and the performance related to the physical aspects of the environment. Mobile learning applications aim to overcome the location and time constraints, which occur in traditional classroom and e-learning platforms. In a traditional classroom, knowledge is shared and transformed inside the class- room: Students and teachers share and exchange the in- formation inside the classroom to create knowledge. However, students may be physically located in different places where teachers provide mobile learning content and services. Therefore, m-learning applications overcome the location constraints; Students and teachers have access to educational resources regardless of the location through their mobile devices. 3. The educational context is about the surround- ings where the learning happens. These surroundings are composed of the learning outcomes, the subject matters of the learning topics, learning assignments and material, as well as the pedagogical approaches. C. m-learning application development process In the mLUX framework, the process used to develop the application is divided into four main phases: 1) The elicitation phase, in which user studies are conducted with students, teachers, and administrators; 2) The data analysis phase, in which the collected data from students and teachers are analyzed. In this phase, the most important needs and features of the students, teachers, and adminis- trators are identified. Various methods are used to analyze the data, such as transcript coding, User and environment analysis is conducted to classify the needs and features; 3) The idea creation or the concept designs phase, in which the potential m-learning application features are presented as scenarios, followed by the design of the prototype of the potential application; 4) the product concept design phase, in which a usability test is conducted on the pro- posed m-learning application prototype after which the prototype is revised based on the latest feedback, after which is the product is ready to implement. Figure 3. Role-players involvement in m-learning application design and development 1) User Study In the user study phase, the primary goals of the m- learning application and the characteristics of each stake- holder are identified. Moreover, in this phase, the user study is conducted with the role-players who are directly or indirectly involved with the design and usage of the application. Furthermore, the main goal of the user study is to determine the needs and expectations of the students and teachers regarding the m-learning application. User representatives, such as students, teachers, and administra- tive personnel, are selected based on the context of use of the application. Therefore, it is essential to determine whether the application should be a new “standalone” or whether it should enhance existing e-learning offerings. Eason [50] classified users as primary, secondary, and tertiary. Primary users use the system constantly, second- ary users use the system occasionally, and tertiary users make decisions about the system. In m-learning applica- tion design and development, the role-players are classi- fied as: • Direct involvement in the elicitation phase: o Primary users, such as students, instructors, and teachers o The secondary users, such as administrative per- sonnel • Indirect involvement (resourcing, supporting, con- straining) o Tertiary users, such as representatives of the ap- plication’s owners and commissioners • Application designers Figure 3 presents the summary of the role-players in- volved in m-learning application development at various phases. In this phase, the designers learn how students and teachers carry out their educational tasks and the context in which these tasks are performed. The following are examples of the questions that aimed to reveal the basic needs and expectations: • How is the target course currently implemented? • Is the lecture offered as an online course or face-to- face? iJIM ‒ Volume 9, Issue 3, 2015 41 PAPER MLUX: USABILITY AND USER EXPERIENCE DEVELOPMENT FRAMEWORK FOR M-LEARNING • How do students and teachers currently carry out ed- ucational tasks, such as the reading material, the as- signments mode, and type of assignments? • What smart devices do student and teachers have ex- perience with? • What is the overall knowledge that students and teachers have regarding smart devices? • How often do students and teachers commute be- tween school and home? • Where do students and teachers often need to have access to course content? • How would users like to change the current imple- mentation of the course? • What are the preferred ways to learn and handle edu- cational tasks and activities? • How can they use their mobile devices in their educa- tional tasks? Beyer [51] recommends that six to twenty users should participate in user studies, which this study defines as the number of users. In this study, we often asked six to eight students and teachers to participate in the user study phase. Six to eight users are recommended because of the diversity of the mobile devices used by stakeholders and the complexity of the application content, which is used by learners. 2) Data Analysis In this phase of the mLUX framework process, the de- signers interpret the data collected in the previous phase. The designers apply various data analysis methods, such as interview transcript coding. The user and task analysis reveals the ways in which students and teachers currently handle educational tasks. Moreover, the designers deter- mine the environments in which these tasks are carried out. This information is vital for developing a robust m- learning application. Figure 4 presents the analysis of the tasks and environment. Interview transcript coding helped to identify the ac- tions and activities of students and teachers in current learning and teaching formats. Moreover, the data analysis phase revealed users’ expectations of the potential m- learning application. The results of the analysis were used to classify the tasks and the environments in which the students and teachers performed the educational tasks. Kujala [52] also promoted the application of a user and task matrix when different stakeholders used the same application. 3) Idea/Co-Creation Design In the concept design phase, the data categorized in the previous phase are expressed as learning and teaching scenarios. The scenarios contain the functionality and features of the potential m-learning application. Kyng [53] recommended that the design be easily understood by all potential stakeholders in the elicitation phase. Therefore, in a possible concept design method, scenarios are select- ed because students and teachers are able to understand the functionalities and deliverable features of the targeted application more clearly. The scenarios are written so that the proposed user experience factors are expressed in different educational components. The content of the scenarios is divided into four educational components: Figure 4. Analysis of users, tasks, and environments in m-learning application development presentation, activities, communications, and administra- tion. In each component, the requirements of the students and teachers are expressed as a short story. The following are examples of the features that are written in the scenarios: • The kind of courses that the m-learning application supports • The preferred material formats in the m-learning ap- plication • Lab/home assignments and their formats • The m-learning application is specific to the type of communication, such as chat, email, and so on. • Locations where the application is used To validate the learning application, three to five stu- dents and teachers share the prepared scenarios. After the students and teachers review the scenarios, the designer conducts a semi-structured interview to collect their feed- back on the proposed concept. This feedback is analyzed, and the designer revises the scenarios according to the results. After validating the m-learning application con- cept, low-fidelity or high-fidelity application prototypes are designed based on the revised scenarios. 4) Product Concept The application prototype is assessed by the students and teachers. The assessments are often carried out in the usability lab. The goal is to assess the functionalities and performance of the application from the perspective of learning. We apply Nielsen’s Heuristic evaluation guide- lines, post-interviews, and think-aloud methods in the assessment. Students and teachers go through sets of pre- defined tasks during the sessions. For example, the teach- ers may be asked to upload the lecture presentation, and the students may be asked to upload the presentation. The designers analyzed the collected feedback and modified the prototype accordingly. The revised prototype of the m- learning application was then ready for development. Figure 5 presents the proposed mLUX framework process that applied and evaluated in m-learning application de- velopment. 42 http://www.i-jim.org PAPER MLUX: USABILITY AND USER EXPERIENCE DEVELOPMENT FRAMEWORK FOR M-LEARNING Figure 5. The mLUX framework applied in the case studies The proposed mLUX framework process was designed and developed so that the m-learning application would be usable, efficient, and error-free. Moreover, the framework aims at ensuring that the students, teachers, and adminis- trators would have a positive experience because the ap- plication is enjoyable, adjustable, reliable, and secure. These factors of user experience ensure that the users of the m-learning application, such as students and teachers, are emotionally engaged with the application so that, for instance, students feel that the learning is enjoyable. Moreover, by ensuring that the user interface is adjustable and customizable, the students feel that they are engaged in the application’s design. Therefore, students consider the application a familiar learning environment. Addition- ally, because the application is secure and reliable, stu- dents and teachers trust it, which reduces concerns about losing important data, such as submitted assignments, and the teachers trust that the submitted feedback reaches the intended students. VII. FRAMEWORK EVALUATION CRITERIA The proposed mLUX framework has been assessed by conducting four case study projects. The criteria for the mLUX framework assessments [54] are as follows: 1. The m-learning applications' usability and user expe- rience are assessed from the perspectives of learners and teachers. The assessment is based on Brusilov- skys' educational components. Brusilovsky divided the course delivery in a virtual university into four main categories: presentation, activities, communica- tions, and administration. In each category, the ad- justability, delightfulness, reliability, and satisfaction of the features were assessed. Table 1 presents the framework measurement criteria based on education- al components and factors of usability and user expe- rience. 2. The assessment of usability of the m-learning appli- cation by students and teachers was based on tradi- tional usability assessment methods. The evaluation of m-learning usability was based on ISO 9241 part 11 and part 304, which recommend that the measures of usability should include effectiveness, efficiency, and satisfaction. A general metric was used to meas- ure the usability as a quality metric, such as the suc- cess rates. Students’ and teachers’ performance was also measured in completing the tasks. The meas- urements are based on the time required to perform the tasks or the number of errors. In addition, the stu- dents’ and teachers’ subjective satisfaction with the application performance and functionality was meas- ured. 3. Semi-structured interviews were conducted with the developers, designers, and the owners of the applica- tion. The aim of this assessment was to collect fur- ther information about the performance of the mLUX frameworks from people who had applied the framework as a method of application development. Examples of questions are the following: • Did you find the mLUX framework helpful in the design and development of the m-learning appli- cation? How easy was it to create the concept by applying the mLUX framework? • How many errors were identified in the scenario phase of your m-learning application design? • What are your overall impressions of the mLUX framework as method for application develop- ment? iJIM ‒ Volume 9, Issue 3, 2015 43 PAPER MLUX: USABILITY AND USER EXPERIENCE DEVELOPMENT FRAMEWORK FOR M-LEARNING TABLE I. CRITERIA OF FRAMEWORK MEASUREMENT Educational Components Usability and User Experi- ence Factors Criteria Presentation Adjustability Supports learning medium / personal-ization Delightfulness Facilities and acceptance of the tool Reliability Supports communication, and tech- nology adequacy Satisfaction Performs the required task Activities Adjustability Supports the learner to perform the basic learning activities Delightfulness Supports the learner to perform tasks without difficulties Reliability Supports the learner to perform his task securely Satisfaction Successfully performs the educational activities Communica- tions Adjustability Supports the performance of basic communication related tasks Delightfulness Facilities and acceptance of the tool Reliability Supports communication and tech-nology securely Satisfaction Successfully performs the task secure- ly Administra- tion Adjustability Supports the learners requirements Delightfulness Supports communication and tech- nology securely Reliability Successfully performs tasks securely Satisfaction Successfully performs tasks For example, to assess the presentation component, we measured the degree to which users were able to personal- ize and customize the m-learning application user inter- face and content based on personal preferences. Therefore, the assessment indicates whether the application supports the adjustability element in the presentation component. In this phase, we evaluated the user experience and usability by asking users to perform predefined tasks using the application. The case studies used different contexts, such as Java applications, adaptive learning, context-aware nursing applications, and so on. Our evaluation assessed only educational components that were relevant to the application’s context. For example in the educational game application, the testing was mainly focused on the presentation and communication components, whereas in driving the m-learning application we assessed and evalu- ated all the layers. VIII. FRAMEWORK EVALUATION: CASE STUDIES The mLUX framework was applied in four different case studies. The aim was to assess the performance of the proposed mLUX. The mLUX assessments composed of three distinic evaluations 1.The application usability as- sessment 2. The case study application assessment from educational components perspective, and 3. The designers and developers feedback. A. Case 1: M-learning application for introduction to Java programming course In this case study project, we applied the mLUX framework to design and develop a mobile learning appli- cation prototype for a Java programming course at the HAMK University of Applied Science. The case study was initiated by conducting a user study with students and teachers to learn their needs and re- quirements. Several methods were applied, such as a dia- ry, questionnaire, and semi-structured interviews, and six students (n = 6) and two teachers (n = 2) were used. The user study begin with collecting background information about the users, such as the type of mobile phones, how often they used their mobile phone, and where and where did they use their mobile the most often. We then created a user profile based on the collected background infor- mation. We the invited the students and teachers to partic- ipate in short semi-structured interviews. The discussions in the interview sessions were recorded for further analy- sis. We applied transcript coding, user and task methods to explore and classify the users’ needs and requirements In the next step, two scenarios were created—one for the students and another for the teachers. The scenarios con- tained the requirements and the features that we identified in the data analysis phase. The scenario was then shared with the participant students and teachers for their review and assessment. In this round, we asked three students and one teacher to review the scenario. Finally, the scenarios were revised based on the reviewer’s feedback. The poten- tial high-fidelity application was designed and implement- ed based on the revised scenarios. The prototype was assessed by conducting a usability test with the potential students and teachers at the media lab at the HAMK Ap- plied Science University. In the usability test, the users were given several pre- defined tasks to carry out one by one. A semi-structured interview was conducted after the user performed all the predefined tasks. The objective of the semi-structure in- terview was to learn about the experiences of the students and teachers in evaluating the prototype of the mobile learning application. Figure 6 presents the application’s home screen. Figure 6. M-learning application for the Java course The application assessment process and the assessment results are discusses as follows: 1) Usability and user experience assessment We conducted different usability test sessions for the students and teachers in the media lab at the Hamk Uni- versity of Applied Science. Six students (n = 6) were asked to participate in the first test session, and two teach- ers (n = 2) were asked to participate in the second test session. The users were given distinct sets of tasks to carry 44 http://www.i-jim.org PAPER MLUX: USABILITY AND USER EXPERIENCE DEVELOPMENT FRAMEWORK FOR M-LEARNING out within a given time. Examples of the given tasks are as follows: • Login to the application with the given credentials. • Upload and download today's lecture notes. • Upload the assignments, submit your answers, and provide feedback for students assignments. The assessment results indicated that the users were able to carry all the given tasks successfully. These results were obtained although this test was the users' first experi- ence in using a third party mobile application in their devices. Moreover, the test results demonstrated that the students and teachers were excited about have the applica- tion in all courses at the university even though the data exchange cost was relatively high for students. Interest- ingly, the users’ initial mental model of the m-learning application was that they had to carry out all the educa- tional activities via short messages (SMS). However, this perception changed as soon as they experienced the appli- cations on the test devices. Additionally, the users ex- pressed appreciation of the simplicity of the user interface on the mobile learning application: "I liked the applica- tion, it was easy to find the needed feature, you just need to go the categories.” Similar responses were made by several students and teachers during the post-interview sessions. The users, however, complained that the input medium, that is, the keypad was not convenient for writ- ing a long text. 2) Educational activities assessment In this evaluation process, the students and the teachers were asked to perform a predefined task to assess the educational components of the application. Table 1 pre- sents the assessment criteria and the process. The results of detailed assessments of each educational component are beyond the scope of this paper. The following are exam- ples of the assessment process and the results. For exam- ple, in the presentation component, the students were asked to download lecture materials, such as lecture notes. The students demonstrated that they were able to down- load the lecture notes successfully, although the technolo- gy did not support many proposed functions, such as vid- eos. The teachers uploaded assignments in the activity component, and the students were asked to upload the assignment and submit their answers through the m- learning application. The teachers provided feedback on the students’ assignments, such as accepted or failed. The assessment analysis revealed that the students did not enjoy using the functionality in the activity component because of the keypad was inconvenient. On the other hand, the teachers enjoyed providing instant feedback on the student’s assignments even when they were walking. In the communication component, the students and teach- ers were asked to use the following features (chat, SMS, email) to communicate with students or other classmates. The students and the teachers found the communication functionalities interesting, especially the capability of direct and secure communication with their peers and course staff. Furthermore, the students and the teachers found the email functionality in communication compo- nent particularly efficient because the course participants’ emails were accessible on the m-learning application. Therefore, they believed that sending an email to a class- mate or teacher was simple and fast because it involved on a few clicks. Similarly, in regarding the administration component, the students and the teachers were asked to perform predefined tasks related to this component, such as registering to the next available course. The students and the teachers responded favorably to the functionality of administration component. 3) mLUX: The developers experiment The design and development of this case study project was carried out by two groups of students at the HAMK University of Applied Science. The first group consisted of four students (n = 4) who had previously studied usabil- ity engineering related courses. In the project, they had the role of designers. This group conducted the user study with the students and the staff. They carried out the data analysis and designed the application concept, which resulted in the scenarios. The second group consisted of two students (n = 2) who had previous experience in ap- plication development. They had the developers’ role in this case study. A workshop was conducted to discuss the mLUX development process with both groups. The team especially appreciated the systematic approach of the framework used to achieve the goal. Additionally, the participants recognized that the divisions were an asset in this framework. Moreover, the participants believed that having the scenario of a design methodology helped the students, teachers, designers, and developers to conceptu- alize the overall application clearly. They showed their wiliness to utilize the methodology in future projects in responses such as “I liked it, clear and helpful to come up with the prototype,” which were repeated by the develop- ers and the designers. B. Case 2: Adaptive m-learning Application for driving school The second case study was designed to develop an m- learning application for driving school candidates. In this project, we applied the proposed mLUX framework pro- cess to designing and developing an adaptive m-learning application. The project was conducted at the Haaga-Helia University of Applied Science during the autumn of 2012, and it continued during 2015. The application was origi- nally commissioned by the Haaga Driving School in Hel- sinki. The early version of the concept and implementa- tion of the application were published in CSEDU 2014 [55]. This mobile learning application enabled the driving license candidates to fulfil the driving schools’ mandatory theory requirements. Additionally, the instructors at the driving school were able to obtain instantly full reports on the students’ performances in driving and theory. Fur- thermore, the application notified the instructors if the students made critical mistakes during the driving ses- sions. In Case Study 1, the application was considered an enhanced tool for existing face-to-face lectures. In Case Study 2, however, the application was a standalone m- learning application. The students and instructors first participated in various user study methods, such as web questionnaire and semi-structured interviews. The stu- dents in this case study were mainly driving school appli- cants who aimed to get a driving license. We conducted the user study with seven users (n = 7) four of which (n = 4) were male and three (n = 3) were female. In the next phase of the project, we analyzed the collected data by applying methods, such as transcript coding, to explore the real needs and expectations of the students and instruc- tors. Figure 7 presents the role-players who were in- volved in the case study 2. iJIM ‒ Volume 9, Issue 3, 2015 45 PAPER MLUX: USABILITY AND USER EXPERIENCE DEVELOPMENT FRAMEWORK FOR M-LEARNING Figure 7. Users’ involvement in different phases Two different scenarios were written for the next phase. The first scenario reflected the implementation of the application based on the driving license candidates’ re- quirements, and the second scenario contained the features required by the instructors. The scenarios were then shared with six students and instructors (n = 6) of whom three users were new and had not been consulted in the elicitation phase. The users were asked to read the scenario and then to participate in a short semi-structured interview. After all the users reviewed the scenarios, the designer analyzed the collect feedback and revised the scenario accordingly. We proceeded with this case study by designing a high fidelity prototype based on the revised scenario. Finally, we conducted a usability test with potential users. Figure 8 presents a screenshot of candidates application user inter- face. Figure 8. The m-learning application adapted for driving license candidates 1) Usability and user experience assessment In this case study, we conducted a usability test with five potential test users (n = 5) and two instructors (n=2) in the media lab at the Haaga-Helia University of Applied Science. The users were asked to perform ten (n = 10) predefined tasks using the application. After the test ses- sions were completed, short semi-structured interviews were conducted to collect data on the users’ experiences with the application. We also recorded all the actions that the users performed in the usability test session for further analysis. Unlike in the previous case study, the users had a mental model of the mobile learning application because they were already familiar and had experience with third- party mobile applications in their smart devices. The anal- ysis of the test sessions indicated that the users were able to perform all predefined tasks. The users found the appli- cation easy to use, and believed that the application con- tain the essential features. Therefore, the application con- cept matched the user’s mental model of a mobile applica- tion in their smart device. The users specifically enjoyed having the content about their driving license on their mobile phones. Responses such as “great, this application is like a permanent reference book in my pocket all the time” were repeatedly given [46]. 2) Educational activities assessments The criteria for the educational components test are shown in Table 1. As in the usability test, the students and instructors were given set of predefined tasks. The tasks were prepared exclusively for each component. The presentation component, the students, and the instructors were able to carry out the tasks properly although some components were not implemented during the test period. Nevertheless, the application content and the User Inter- face (UI) were customizable based on student's perfor- mance, such as in the theory lessons and practical driving session. Moreover, the test showed that the application support- ed multi-formatted content based on the students’ needs and application requirements. However, because of the technological inadequacy and lack of development time, the presentation component was not fully implemented. In the activity component, the test users showed their satisfaction with the clarity of the features: “the applica- tion helped me what to do next.” However, the reliability factor for this component could not be tested because the prototype was not fully implemented. The communication component was not explicitly implemented, in which the students and teachers were instructed to use SMS or any other existing methods. In the administration component, the test users were able to perform the required task, such 46 http://www.i-jim.org PAPER MLUX: USABILITY AND USER EXPERIENCE DEVELOPMENT FRAMEWORK FOR M-LEARNING as printing out the student’s driving activities during the last ten days. The results of the analysis revealed that the application supported the administration component fea- tures and fulfilled the criteria defined in Table 1. 3) Methodological assessments The interviews conducted with designers and develop- ers indicated that they were satisfied with the resulting application. They were especially satisfied that the mLUX framework reduced the number of the design errors before implementing the high fidelity prototype: “We managed to find most of the errors already in scenario review phase.” The designers were specifically happy with the involve- ment of potential users in the design phases, which helped identify the new feature. In the elicitation phase, not all the designers found all the required features: “We identi- fied additional features both in the scenario and in low- fidelity prototype testing.” The other role-players, such as those who conducted the user studies, were also consulted about their motivation for using the mLUX framework: “In the beginning using the framework work was difficult, did not know what to do, but after the transcript coding everything changed.” The main negative feedback re- ceived from developers was that this methodology re- quired extensive consultation with users, which would consume a great deal of time. For example, one stakehold- er, who had commissioned the application, expressed overall satisfaction with the m-learning application: “Very surprised everything works with any errors; we tested the application with students and instructors.” C. Case 3: Business guide mobile game application development In the third case study, a customer briefing m-learning application was developed based the on mLUX frame- work process. The application owner provided us with the potential requirements of the application. Hence, in this case study, no user study was conducted to define the potential features of the application. Therefore, the con- cept design began with three different scenarios based on the given list of requirements. These scenarios represented different pedagogical approaches to developing target application: 1) m-learning application as an educational game; m-learning application as a choice question; and m- learning applications where theoretical content is present- ed to users. The application owners initially assessed the scenarios by confirming that we evaluated the proposed scenario by testing it on potential users of the applications. We shared the scenarios with five (n = 5) students and teachers. The students and teachers read the scenarios and then partici- pated in short semi-structured interviews. In the interview session, we consulted the students and teachers about the reviewed scenarios, asking questions, such as “What do you think about the scenarios?” and “Which scenario would you prefer as an application in your phone?” We recorded the interview sessions with students and teachers for further analysis of the responses. The results revealed that the majority of the students and teachers recommend- ed the educational game scenario as appropriate for the potential m-learning application. Hence, the educational game scenario transformed into a low-fidelity paper proto- type with the help of a wire-framing tool. The paper proto- type was shared once again with the students and teachers to collect further design-related feedback. The prototype evaluation was carried out with five (n = 5) students and teachers, three of which had already been consulted in the scenario review sessions. Two users (n = 2) were new test users. The feedback collected from the evaluations resulted in modifications to the design concept. The analysis of the collected feedback indicated that the users were dissatis- fied with the application’s starting style, the non- responsiveness of the soft keys (buttons) in the user inter- face (ui) and the lack of essential functionality. The func- tional and “proof-of-concept” prototype was then imple- mented by using technologies, such as PhoneGap, HTML5, CSS3, and various JavaScript libraries. The server-side of the application was implemented using Ruby to build the API and Redis for persistent data stor- age. When the application was ready and functional, the us- ability tests were conducted by applying the Nielsen heu- ristic evaluation guidelines [56]. Unlike in the usability assessment of the paper prototype, where the users carried out predefined sets of tasks, in this assessment, the users had the freedom to play the game as they wished. In order to gather feedback, the users were asked to pause the game only when they reached a specific stage. The main goal of this assessment was to measure the consistency and aesthetics of the proposed educational game. A secondary goal was to evaluate the usability of the product. The users were asked to participate in a short interview after the test session. Figure 9 presents screenshots of the user interface in the business game application. Figure 9. Screenshots of the business game application 1) Educational activities assessments In this prototype, the presentation component was not fully implemented although the users were able to select the game character. Moreover, the adjustability factors, such as setting the color and components of the game, were not appropriate for the proposed game. Nevertheless, the users were able to create a profile and select a role image. In this case study, the activity component required students to play and explore the game’s features, which all users did properly. Furthermore, the users demonstrated their enjoyment in responses such as “Wow, this is fun.” In this application prototype, we also did not implement the communication component. Therefore, we ignored this feature in the test session. The personnel administrators showed their satisfaction with the implemented features, such as create and delete, and with the ability to print users’ profiles and scores. The users also showed satisfac- iJIM ‒ Volume 9, Issue 3, 2015 47 PAPER MLUX: USABILITY AND USER EXPERIENCE DEVELOPMENT FRAMEWORK FOR M-LEARNING tion with the administrator component in responses: “I can see who played and what points they gained.” 2) Methodological assessments Because the owner of the application was in a different city, Skype meetings were conducted after each develop- ment phase, such as scenario design, low-fidelity proto- type design, and high-fidelity evaluations. Therefore, the owner was considered a stakeholder in the application. The designers considered the mLUX framework an ap- propriate development model to achieve their goal. The designers particularly appreciated the scenario design in comments, such as “Scenario saved many development times. I had initially planned to design an application and then check with users, now I know that users want game and not any other mobile application types” and “Scenar- io helped me see the kind of game users preferred and wanted me to implement.” In the starting phase of the project, the designers considered the scenario design an unnecessary step. However, during the project, the de- signers considered the framework a robust approach to development. The designers, developers, and customers expressed satisfaction with the concept of the application concept and the product. The negative feedback from the designers and developers focused on the extensive user consultancy required, which consumed a great deal of time during the design and development of the applica- tion. D. Case 4: Context-aware nursing application (CANA) The objective of this case study was to test the proposed mLUX framework process. Several mHealth applications exist, but none combines the essential professional tools for nursing. We tackled this problem in a Finnish elderly care facility. We applied the mLUX framework in a user study. The profiles of 12 nurses and their expectations of work-related mHealth application functionalities were identified. The results were utilized in the conceptual design of the context-aware nurse assistant (CANA) ap- plication, which combines the identified functionalities and provides context-sensitive services to consolidate nurses’ work activities. The results of this case study are published in the ICUM 2015 conference proceedings [57]. Figure 10 presents sample of application user interface. Figure 10. Screenshot of the context-aware user interface The framework assessment conducted in this case study is briefly discussed in the following sections. 1) Usability and user experience assessment The prototype was evaluated by four nurses (n = 4) and a doctor (n = 1) ranging age between 21 to 60 years (four females and a male), who also participated in the initial user study phase. We conducted the evaluations in one-to- one sessions with five users at elderly care facilities. The evaluation process was as follows: We gave a set of pre- defined tasks to the users to perform by using the applica- tion individually. We asked the users to think aloud when they carried out the tasks. After the users completed the tasks, we conducted semi-structured interviews with them to collect data on their evaluation experiences. The objective of this usability evaluation was to meas- ure the prototype’s overall design and to assess the con- cept. The results of the analysis showed that all test users successfully performed the predefined tasks within the given time. The test users considered the application unique and excellent, which would ease their work signif- icantly. Several responses expressed that “I never ever thought that I could perform my work related tasks with my mobile.” The test users were surprised at the simplici- ty of the application. No significant usability or functional failures occurred in this usability assessment. 2) Educational activities assessments The application in this case study was not designed for educational purposes. Hence, the assessment of the educa- tional components features is not applicable in this pro- ject. Nevertheless, we assessed the communication and administration components, which are similar to the m- learning application communication and administration components. The users were delighted to see that through the application, they could access other coworkers in the same department: “Nice feature; I don't have to go to the office and search for the other nurse in the department” and “I can see who I can contact if I need help, great.” Similarly, the results of the assessment demonstrated that the users were able to carry their activities properly and without encountering difficulties. The further assessment of the administration component showed that that users were able to view, search, and print, in addition to adding and removing users. 3) Methodological assessments A meeting with designers and developers was orga- nized to discuss the process of applying the ppmLUX framework process. The designers considered the process of developing the application straightforward: “It is an excellent method for designing and developing an applica- tion without benchmarking.” Moreover, the application developers considered the methodology very effective although it involved many role-players. However, the designers complained about the mLUX framework’s lack of design guidelines: "I wish we had the design guidelines to help us get the better user experience for the design.” Furthermore, the designers found that the continuous feedback from users was a motivating factor: “The sce- nario evaluation feedback showed that I am doing the right thing.” Furthermore, the designers found the scenar- io design to be a helpful method for users to conceptualize the final application easily. The designers promoted this scenario as a strength of the proposed mLUX framework process. The discussion notes indicated that all stakehold- ers of the application were satisfied with the outcome: “Good model to design a mobile application; I'll apply the UCD framework in my next project.” 48 http://www.i-jim.org PAPER MLUX: USABILITY AND USER EXPERIENCE DEVELOPMENT FRAMEWORK FOR M-LEARNING IX. CONCLUSION AND DISCUSSION A. How to Design M-Learning with Good UX The designs and development of application used in smart gadgets, such as smartphones, require special design considerations. Application designs require many iterative phases in order to make the application appealing yet hide the complexity of the application’s functionalities. M- learning applications are complex because they deal with learning and learners, who have diverse needs and re- quirements. Therefore, the design and development of the m-learning applications studied here required the partici- pation of the role-players in design and in development. This ensured that the m-learning application would be efficient and easy to use by both students and teachers. Nevertheless, the literature review showed that tradi- tional usability assessments were not sufficient for a ro- bust mobile learning application. M-learning applications should be able to compete for students’ time and dedica- tion among the entertainment applications in the phone. The results of the four case studies in this project demon- strated that mLUX framework was acceptable for use in m-learning applications. The performance of the mLUX frame was evaluated by role-players, such as students, teachers, and the administrative personnel, designers, and developers. Moreover, the applications tested in the four case studies were assessed based on the educational com- ponents of user experience criteria from educational per- spective, this assessment helps to evaluate the m-learning application as an educational medium. Furthermore, the results of the case study evaluations revealed that the students and teachers directly influenced the design and functionality of the m-learning application. This study demonstrated that the successful structure of the mobile learning application did not depend solely on the requirements of the user and the usability of the appli- cation. A successful mobile learning application, fulfil the educational components, which include the usability and user experience. Based on the results, we believe that the development of acceptable mobile learning application should use the following three main components, which are shown in Figure 11: 1. The application’s features, functionalities, and deliv- erables enabled the users to perform their essential educational tasks through their mobile learning ap- plication. 2. The development technology facilitates usage with a wide variety of the latest wireless and application de- velopment technologies across several platforms. Figure 11. Components in the development of m-learning applications The results of this study showed that the most prevalent factors in the usability of m-learning applications were adjustability, Delightfulness, reliability, and satisfaction [50]. Smartphones are supported by diverse development technologies and divergent application features. Hence, the design and development of m-learning applications must include their support across various smartphones platforms. Therefore, designers should ensure the compat- ibility of the m-learning application with the platforms of different smart devices [2]. Hence, the compliance of m- learning applications with smart device-supported tech- nology [2] would result in positive user experiences with the application [40]. The mobile learning application is often used for long periods. In addition, students often need to pay extra atten- tion to learning how to use the application as well as the assigned topic. Therefore, m-learning applications must provide positive experiences for both students and teach- ers. The experiences that students and teachers gain from using the mobile learning application will ensure the ap- plication’s sustainability and its long-term utilization in the educational process. Furthermore, entertainment appli- cations, such as games, compete for the time that students spend using their smartphones. These factors increase the importance of ensuring that the m-learning application user has a positive experience. In this study, the design of the m-learning application was such that the students and teachers were entertained by using it. However, the design and function of the application must avoid complexity and help users focus on the content. Factors of user experi- ence, such as adjustability, helped the students and teach- ers to customize the application based on their own design preferences. The case study applications' evaluations re- vealed that the adjustability and customization factors are important to gain positive user experience. The adjustabil- ity helps students and teachers to consider the m-learning application as their own design. Other attributes of the application, such as delightfulness, indicated a significant effect on the user’s retention and sustainability in using the application. In the context of mobile learning, a de- lightful and enjoyable experience means that the students and teachers are not just satisfied but even entertained by using the m-learning application. The result indicates that the delightfulness as an im- portant factor in m-learning. Delightfulness is achieved when the target application users, such as teachers and students, create the content of the learning application. Furthermore, when the user interface in the m-learning application is designed to be entertaining, it helps students to avoid complexity in the learning environment and yield an easy comprehension of the learning topics. In the future, we aim at evaluating the mLUX frame- work from the design perspective based on the responses to questions, such as “How to create an enjoyable experi- ence?” and “What are the m-learning application sustaina- ble use factors?” The answers to these questions will pave the way towards more adaptive learning, personally ad- justed learning experiences, and broader learning contexts. REFERENCES [1] J. Traxler, “Defining mobile learning,” in IADIS International Conference Mobile Learning, 2005, pp. 261–266. [2] A. Dirin, M. Nieminen, and M. Kettunen, “Student capabilities to utilize m-learning service in new smart devices,” in Proceedings iJIM ‒ Volume 9, Issue 3, 2015 49 PAPER MLUX: USABILITY AND USER EXPERIENCE DEVELOPMENT FRAMEWORK FOR M-LEARNING of the 2013 International Conference on Advanced ICT, 2013. http://dx.doi.org/10.2991/icaicte.2013.89 [3] N. Golafshani, “Understanding Reliability and Validity in Qualita- tive Research,” Qual. Rep., vol. 8, no. 4, pp. 597–606, 2003. [4] R. K. Yin, Case Study Research: Design and Methods, vol. 5, no. 5. 2009. [5] K. M. Eisenhardt, “Building Theories from Case Study Re- search.,” Academy of Management Review, vol. 14, no. 4. pp. 532–550, 1989. [6] I. Benbasat, D. K. Goldstein, and M. Mead, “The Case Research Strategy in Studies of Information Systems,” Manag. Inf. Syst. Q., vol. 11, no. 3, pp. 369–386, 1987. http://dx.doi.org/10.2307/ 248684 [7] R. Molich, M. R. Ede, K. Kaasgaard, and B. Karyukin, “Compara- tive usability evaluation,” Behaviour & Information Technology, vol. 23, no. 1. pp. 65–74, 2004. http://dx.doi.org/10.1080/ 0144929032000173951 [8] J. Gulliksen, B. Göransson, I. Boivie, S. Blomkvist, J. Persson, and Å. Cajander, “Key principles for user-centred systems de- sign,” Behaviour & Information Technology, vol. 22. pp. 397– 409, 2003. http://dx.doi.org/10.1080/01449290310001624329 [9] G. He and J. Yu, “Identify Lead Users by Customer Competence,” in IEEM2010 - IEEE International Conference on Industrial Engi- neering and Engineering Management, 2010, pp. 1305–1308. http://dx.doi.org/10.1109/ieem.2010.5674396 [10] J. Nielsen, “Usability 101!: Introduction to Usability,” All Usabil- ity, vol. 9, pp. 1–10, 2003. [11] J. L. Mcgrenere, “Design: Educational electronic multi-player games - A literature review,” Science (80-. )., vol. 25, no. June, p. 2003, 1996. [12] G. Vavoula and M. Sharples, “Meeting the challenges in evaluat- ing mobile learning: a 3-level evaluation framework,” Int. J. Mob. Bended Learn. 1(2), pp54-57., 2009. http://dx.doi.org/10.4018/ jmbl.2009040104 [13] M. Ziefle, “The influence of user expertise and phone complexity on performance, ease of use and learnability of different mobile phones,” Behaviour & Information Technology, vol. 21, no. 5. pp. 303–311, 2002. http://dx.doi.org/10.1080/0144929021000048538 [14] J. Gong and P. Tarasewich, “Guidelines for handheld mobile device interface design,” Proc. DSI 2004 Annu. Meet., pp. 3751– 3756, 2004. [15] D. Zhang and B. Adipat, “Challenges, Methodologies, and Issues in the Usability Testing of Mobile Applications,” International Journal of Human-Computer Interaction, vol. 18, no. 3. pp. 293– 308, 2005. http://dx.doi.org/10.1207/s15327590ijhc1803_3 [16] D. Corlett and M. Sharples, “Tablet technology for informal collaboration in higher education,” in Mobilelearning any- timeeverywhere, 2004, vol. Bracciano, pp. 59–61. [17] A. Hussain and E. Ferneley, “Usability metric for mobile applica- tion: a goal question metric (GQM) approach,” in Proceedings of the 10th International Conference on Information Integration and Web-based Applications and Services (iiWAS ’08), 2008, pp. 567–570. http://dx.doi.org/10.1145/1497308.1497412 [18] A. Kukulska-Hulme, “Mobile learning: a handbook for educators and trainers.” 2005. [19] A. Kukulska-Hulme, “Mobile usability in educational contexts: What have we learnt?,” Int. Rev. Res. Open Distance Learn., vol. 8, 2007. [20] P. Syvänen, Antti., & Nokelainen, “Digitaalisen oppimateriaalin mobiilikäytettävyyden arvioinnin kriteerit, 2004, Saarinen Jorma (toim). 5/2004 Evaluator. Hämeenlinna: Hämeen AMK. 2004,” 2004. [21] C. Lindholm and T. Keinonen, Mobile Usability: How Nokia Changed the Face of the Cellular phone. McGraw-Hill, Inc. New York, NY, USA ©2003, 2003. [22] E. Soloway, M. Guzdial, and K. E. Hay, “Learner-centered design: the challenge for HCI in the 21st century,” interactions, vol. 1. pp. 36–48, 1994. [23] A. Mostakhdemin-hosseini, “Mobile learning framework,” in Mobile Learning Framework. Proceedings IADIS International Conference Mobile Learning 2005, 2005, vol. 3, pp. 203–207. [24] M. Pehkonen and H. Turunen, “Preliminary guidelines for the design of the mobile learning activities and materials.,” in Euro- prix Scholars Conference, 2003. [25] E. Malliou, F. Maounis, A. Miliarakis, S. Savvas, S. Sotiriou, and M. Stratakis, “The motfal project - Mobile technologies for Ad- hoc learning,” in Proceedings - IEEE International Conference on Advanced Learning Technologies, ICALT 2004, 2004, pp. 910– 911. [26] V. Teija and A. Mikko, “A critical approach to an adaptive user interface design. Learning with mobile devices – research and de- velopment - a book of papers. Learning Skills Development Agency: (mlearn 2003),” in mlearn 2003, 2003. [27] A. Jäppinen, M. Ahonen, T. Vainio, and E. Tanhua-Piiroinen, “Adpative Mobile Learning System: The essential issues from the design perspective. In J.Attewell & C.Savill-Smit (Eds). M- learning: an educational perspective,” in MLEARN 2004 Mo- bilelearning anytimeeverywhere, 2004, pp. 73–75. [28] D. S. K. Seong, “Usability guidelines for designing mobile learn- ing portals,” Proc. 3rd Int. Conf. Mob. Technol. Appl. Syst. - Mo- bil. ’06, p. 25, 2006. http://dx.doi.org/10.1145/1292331.1292359 [29] M. Koole and M. Ally, “Framework for the Rational Analysis of Mobile Education (FRAME) model: Revising the ABCs of educa- tional practices,” in Proceedings of the International Conference on Networking, International Conference on Systems and Interna- tional Conference on Mobile Communications and Learning Technologies,ICN/ICONS/MCL’06, 2006, vol. 2006. http://dx.doi.org/10.1109/icniconsmcl.2006.103 [30] T. Magal-Royo, G. Peris-Fajarnes, I. Tortajada Montañana, and B. Defez Garcia, “Evaluation Methods on Usability of m-Learning Environment: Interactive Mobile Technologies,” October, vol. 1, no. 1, pp. 32–39, 2007. [31] L. Uden, “Activity theory for designing mobile learning,” Interna- tional Journal of Mobile Learning and Organisation, vol. 1. p. 81, 2007. http://dx.doi.org/10.1504/IJMLO.2007.011190 [32] L. S. Vygotsky, Mind in society: The development of higher psychological processes, vol. Mind in So. 1978. [33] B. Nardi, “Activity theory and human-computer interaction,” in Context and consciousness: Activity theory and human-computer interaction, 1996, pp. 7–16. [34] K. Kari, “Activity Theory as a potential framework for human- computer interaction research,” MIT Press, 1995. [Online]. Avail- able: file:///C:/Users/hayes/Downloads/Activity Theory and HCI.pdf. [35] Y. Engeström, “Activity theory and individual and social trans- formation,” in Perspectives on activity theory, 1999, pp. 19–38. [36] J. Nielsen and D. A. Norman, “The Definition of User Experi- ence.” . [37] M. Hassenzahl and N. Tractinsky, “User experience - a research agenda,” Behaviour & Information Technology, vol. 25, no. 2. pp. 91–97, 2006. http://dx.doi.org/10.1080/01449290500330331 [38] P. Brusilovsky and M. Philip, “Course Delivery System for Virtu- al University,” 2001. [39] M. Nieminen, “Information Support for User-Oriented Develop- ment Organisation. Considerations based on the Construction and Evaluation of Knowledge Storage,” Tesis Dr., p. 228, 2004. [40] J. D. Gould and C. Lewis, “Designing for usability: key principles and what designers think,” Communications of the ACM, vol. 28. pp. 300–311, 1985. http://dx.doi.org/10.1145/3166.3170 [41] J. D. Gould, S. J. Boies, and J. Ukelson, “How to Design Usable Systems,” in Handbook of Human Computer Interaction, M. G. Helander, T. K. Landauer, and P. V. Prabhu, Eds. Elsevier Sci- ence, 1997, pp. 231–254. [42] ISO, “Ergonomics of human-system interaction - Part 210: Hu- man-centred design for interactive systems (ISO 9241- 210:2010(E)),” ISO. pp. 1–32, 2010. [43] ISO, “ISO 9241 Part 210: Human-centred design for interactive systems,” in ISO 9241 Ergonomics of humansystem interaction, 2010. [44] J. Preece, Y. Rogers, and H. Sharp, Interaction Design: Beyond Human-Computer Interaction, vol. 18. 2002. [45] P. Friedrich, Web-based co-design. 2013. 50 http://www.i-jim.org PAPER MLUX: USABILITY AND USER EXPERIENCE DEVELOPMENT FRAMEWORK FOR M-LEARNING [46] D. Wallach and S. C. Scholz, “User-Centered Design: Why and How to Put Users First in Software Development,” in Software for People, 2012, pp. 11–39. [47] A. Dirin and M. Nieminen, “Managing m-learning application development!: Roles and Responsibilities,” in International Con- ference on Advanced ICT (ICAICTE-2014)., 2014. [48] M. Sharples, “Big issues in mobile learning,” 2006. [49] R. M. Kekwaletswe and D. Ng’ambi, “Ubiquitous social presence: Context-awareness in a mobile learning environment,” in Proceed- ings - IEEE International Conference on Sensor Networks, Ubiq- uitous, and Trustworthy Computing, 2006, vol. 2006 I, pp. 90–95. http://dx.doi.org/10.1109/sutc.2006.142 [50] K. Eason, Information technology and organizational change. London: Taylor and Francis, 1987. [51] H. Beyer and K. Holtzblatt, Contextual design: Defining custom- er-centered systems, vol. 32, no. 2. 1998. [52] S. Kujala, M. Kauppinen, L. Lehtola, and T. Kojo, “The role of user involvement in requirements quality and project success,” 13th IEEE Int. Conf. Requir. Eng., 2005. http://dx.doi.org/10.1109/re.2005.72 [53] M. Kyng, E. Nielsen, and M. Kristensen, “Challenges in designing interactive systems for emergency response,” Proc. 6th Conf. Des. Interact., 2006. http://dx.doi.org/10.1145/1142405.1142450 [54] Dirin Amir and N. Marko, “Assessment of User-Centered Design Framework for m-learning Application Development,” in 17th In- ternational Conference on Human-Computer Interaction, 2015. [55] A. Dirin and M. Casarini, “Adaptive M-Learning Application for Driving Liceness Candidates Based on UCD Framework for M- Learning Application Development,” in 6th International Confer- ence on Computer Supported Education, 2014. [56] J. Nielsen, “How to Conduct a Heuristic Evaluation,” Useitcom. pp. 1–11, 2002. [57] M. Dirin, A. Dirin, and T. H. Laine, “User-Centered Design of a Context-Aware Nurse Assistant ( CANA ) at Finnish Elderly Houses,” in The 9th International Conference on Ubiquitous In- formation Management and Communication, 2015. AUTHORS A. Dirin is with Haaga-Helia University of Applied Science /BIT/, Helsinki, Finland. M. Nieminen is with Aalto University, Department of Computer Science, ESPOO, Finland. Submitted 03 February 2015. Published as resubmitted by the authors 18 May 2015. iJIM ‒ Volume 9, Issue 3, 2015 51 iJIM – Vol. 9, No. 3, 2015 mLUX: Usability and User Experience Development Framework for M-Learning