Paper Title (use style: paper title) EVALUATION METHODS ON USABILITY OF M-LEARNING ENVIRONMENTS Evaluation Methods on Usability of m-Learning Environments T. Magal-Royo, G. Peris-Fajarnes, I. Tortajada Montañana, B. Defez Garcia University Polythecnic of Valencia/Research Center in Graphic Technology, Valencia, Spain Abstract—Nowadays there are different evaluation methods focused in the assessment of the usability of telematic methods. The assessment of 3rd generation web environments evaluates the effectiveness and usability of application with regard to the user needs. Wireless usability and, specifically in mobile phones, is concentrated in the validation of the features and tools management using conventional interactive environments. There is not a specific and suitable criterion to evaluate created environments and m-learning platforms, where the restricted and sequential representation is a fundamental aspect to be considered. The present paper exposes the importance of the conventional usability methods to verify both: the employed contents in wireless formats, and the possible interfaces from the conception phases, to the validations of the platform with such characteristics. The development of usability adapted inspection could be complemented with the Remote’s techniques of usability testing, which are being carried out these days in the mobile devices field and which pointed out the need to apply common criteria in the validation of non-located learning scenarios. Index Terms—Wireless environments, usability, inspections methods, criteria-based evaluation, interfaces analysis. I. II. INTRODUCTION M-learning is, regarding Sharples, the type of learning characterised by the usage of wireless technology, through the personal control of the learning time and place, under an autonomy level and limitations determined by the device (Sharples, 2005)[1]. Nowadays, m-learning employs the same pedagogical methods as any other conventional learning method, telematic or not. Nevertheless, the real problem is still the efficient and suitable adaptation of the contents to a means with clear restriction levels (Avellis, 2003) [2]. The former restrictions are basically visual (reduced screen, colours…etc), technological (memory, variety and compatibility between models…etc), and social (SMS emission and reception costs, acquisition, devices access and appropriate use…etc), all of them treated in several research papers, which measure the impact of the wireless devices and/or “small devices” in our society (Oulasvirta, 2004) [3]. The present paper tries to show the possible use of “usability inspection methods” as a evaluation techniques, to evaluate too essential aspects: the contents based in the type of information managed through the means, and the interface, as the man-machine communication environment (Sharples, 2005) [1]. M-LEARNING CONTENTS Nowadays, and as it happens in other kinds of telematic learning, the contents in general are a key part of the formative environments, that allow to access to different learning formats, and also facilitate the integrated management of the same in complex virtual platforms. These educative platforms can offer a wide range of interactivity based on the set of formative functionalities implemented telematically. Therefore, there can be found from totally self- formative environments, to b-learning techniques, which introduce combined aspects within presential and telematic approaches. Purely m-learning contents are focused in self-learning methodologies addressed to brief and accurate learning (e.g. traffic/road education experiences through contents designed for mobile phones) [4]. Complex contents management is employed in mixed b-learning, when there is an interaction between teacher and student (on/off-line), and a visual interface similar to those used in conventional e-learning. The limit is established by the available technology and the features of the means. It is still a determinant factor, in such technologies, the transmission cost, which affects the communication logistic between teacher and student, and has a direct repercussion on the learning type. In general, the use of contents in m-leaning environments can be regarded as reduced against the transmission of more traditional contents, which have been object of many studies. The most usual contents are: -Casual contents -Protocol and courtesy contents -Contents of social, relational or communicative message -Specific contents -Control contents The employment of this kind of contents expands the immense range of possibilities of wireless environments towards new sociological researches, but involves few advancements in the field of the traditional learning. The same happens with the rest of the telematic platforms. Actually, it is important to reinforce that there is no real evaluation of the formative effectiveness of the contents, but the traditional quantitative-qualitative evaluation. In iJIM – International Journal of Interactive Mobile Technologies – www.i-jim.org 1 EVALUATION METHODS ON USABILITY OF M-LEARNING ENVIRONMENTS the ambit of usability, this topic is many times evaluated in terms of formal aspects rather than content. INTERFACES AS COMMUNICATION TOOLS III. Within the different interfaces there are two main tendencies: traditional interfaces based on the applied operative system, and interfaces based on the development of new languages “ad-hoc”. In both cases, it is fundamental to use protocols, rules, standards…etc., related to the logistic and the computing processes for the creation of these environments. It is also necessary to apply usability evaluation methods in the creation, development and validation phases, which could help to verify the functional aspects in the employment of the elaborated tool or platform. Possibly, the control and verification of an adapted m- learning environment from a more consolidated and validated e-learning platform, can be considered as the most reliable way, thus it is built on premises related to strict language and computing standards. Nevertheless, this fact does not assure expectatives to be accomplished from the formative point of view (Trifonova, 2003) [5]. Clear examples are the interfaces adapted for wireless systems such as PALMS or PDAS, where sometimes the access to the information can be physically complicated (e.g. complementary keyboards and optical pens had been to be designed), visual barriers (too small typography, compressed text, inappropriate colours…etc.) and technological handicaps (connection failure and restricted access). As a consequence, it is necessary to bear, from the conceptualization moment of a new m-learning environment (adapted or not), navigation characteristics, interactivity orientation, information management, and accordingly, the contents to be taught and learnt. On one side, navigation is important because of the need of navigable elements, such us links, icons, symbols or menus that allow the user to know the environment, and to get familiar with the educative ways raised in knowledge acquisition and features management. On the other side, the orientation in a virtual environment is the cognitive situation of the user in front of a task to be carried out. Regarding wireless devises, this could become a serious usability problem due to the sequential nature of the interface, which force to memorised access routes. A highly developed interface with several elements of navigation is achievable. However, there are no mechanisms to permit users to recognise their situation in a particular instant inside the created environment, or when a complex educative task is being developed. Figure 1: M-learning environment IV. M-LEARNING USABILITY The employment of usability evaluation methods of any telematic interactive tool is fundamental in order to validate it against potential users (Karoulis, 2003) [6]. Currently, the most common usability definition can be found in the international standard ISO/IEC 9126-1, where six guidelines are described for the creation of any kind of telematic application, extensible, without a doubt, to other programs and applications developed for mobile devices. They are functionality, reliability, usability, efficiency, maintainability and portability. Usability is also described as the quality of an application to be understood, learned, used and attractive by/to the user, when employed under specified conditions or in context of use conditions. The evaluation criteria related with usability is mostly concentrated in the assessment of the efficiency with which the user is able to manage the tool, and the effective of itself when performing a certain task. Different expert and inexpert evaluators interact with the platform and expose their points of view globally or specifically. Moreover, usability inspection is useful to find running problems, although it can also be applied while the design and building phase. The most widely used methods are: -Heuristic evaluation. Is a method proposed in the 90s by Nielsen (Nielsen &Molich, 1990) [7], where an expert applies some principles of usability on a certain program, tool or environment telematically developed. The different methods and experiences performed during the next decade caused the development of many categorised criteria, reaching up to 294 typical problems. Currently, the need of faster and more operative strategies has made the criteria to come to 10 levels, which many times are organised with regard to the platform to evaluate. -Cognitive walkthroughs evaluation. Is a method developed by Lewis (Lewis et Al. 1990) [8], where some user problems are simulated in detail and step by step, especially analysing each task from a cognitive point of view. Expert users’ profiles are employed, with special care in the first formalization stages of the telematic program, tool or environment prototype. iJIM – International Journal of Interactive Mobile Technologies – www.i-jim.org 2 EVALUATION METHODS ON USABILITY OF M-LEARNING ENVIRONMENTS -Conventional user test, following the diverse analysis methodologies of telematic platforms created or expressly adapted, regarding the need of valuation (functionality, visual ergonomics…etc.), whose profile is quite vast where sometimes previous knowledge is required. The validation or an m-learning environment needs the combined use of usability techniques due to the involved factors in a platform of such characteristics. In any case, each technique can have a specific value, more important depending on the development phase. In the following table, the most suitable evaluation methods are introduced in relationship to each development phase. Complementary to the existent evaluation methods and due to the non-location implications of the mobile devices, usability methods are being employed in real contexts, where the final user can be more comfortable to assess an m-learning platform. The usability tests can be carried out in artificial laboratories or in real scenarios. The most of the experiences are lab-performed in order to prevent data from being contaminated by external factors. Lab-tests need equipment and staff able to assume specialised tasks and essays, executed under human and technical supervision. This method seems to be the most viable in usability evaluation, since the impediments related to the user location and to the different interface models in wireless devices are avoided. However, there are studies that point out that there are no significant variations between lab- and non-lab-tests in the web application field (Tullis et Al) [9]. The concepts and raisings of Remote Web Usability Testing could be treated as a starting point in the real usability validation, extrapolated to m-learning environments, since it permits to obtain quantitative data about the user behaviour in environments external to the researcher. Table 1: Usability Evaluation methods against m-learning platforms development phases Remote usability testing is described as “usability evaluation wherein the evaluator, performing observation and analysis, is separated in space and/or time from user” (Hartson et Al. 1996) [10]. Usually, remote usability testing can employ software able to share and/or control the information managed by the user, allowing a remote platform usability evaluation (Waterson et Al.)[11]. This method increases users’ participation and interactivity, because they feel less controlled. In most cases, they ignore the nature of the evaluation and so act freely and concentrated. The most of the current remote applications are focused in program test through the extraction of perceptive (visual, tactile, audible) or computing (website accesses, website types…etc.) data. These systems hold accurate data of users’ movements in a remote environment. Programs such us Noldus, Uzilla, WebRemUsine or WebQuilt are good examples, although it has to be considered that, normally, some additional software is required to process and control remote data. Freeware software such as WebQuilt, used in web environments, has been adapted to wireless environments to analyse webpage access (Hong et Al) [12] and information management frequency (Tara Matthews, 2001) [13]. It is possible to expand this initiative to navigability analysis in a conventional m-learning environment, based on standard languages. CONCLUSION V. Beginning with the initial conditions of the means, m- learning environments offer a wide field of possibilities in for the development of specific methods that will allow functionality validation in a platform of such characteristics. Traditional techniques are combined with iJIM – International Journal of Interactive Mobile Technologies – www.i-jim.org 3 EVALUATION METHODS ON USABILITY OF M-LEARNING ENVIRONMENTS innovative ones, addressed to “in situ” valuation in real contexts A starting point for the assessment of one technique against the rest is based on two factors: first, the capability of the methods to facilitate the contents access, fomenting their learning in these environments; secondly, the platform functionality assessment through its navigability and user orientation. REFERENCES [1] Sharples M. Taylor J. Vavoula G. “Towards a Theory of Mobile Learning”. Proceedings of mLearn 2005 Conference, 2005. [2] Avellis G., Scaramuzzi, A. , Finkelstein, A. ”Evaluating Non functional Requeriments in Mobile learning Contents and multimedia educational software”. Book of abstracts MLEARN 2003. pp. 4-5. [3] Oulasvirta Antti. “Human-computer interaction in móviles context: a cognitive resources perspective”. Licentiate Thesis, Faculty of Behavioral , University of Helsinki.2004 [4] Theory Mobile, BSM Service. Tribal CTAD. www.bsm.co.uk. (Consulted 10 of march 2007). [5] Trifonova A. , Knapp J. Ronchetti M., Gamper J. “ MOBILEDIT: Challenges in the transition from an e-learning to an m-learning system”. Department of Information and Communication Technology. University of Trento., 2004. [6] Karoulis A., Pombortsis A. “Heuristic Evaluation of Web-based ODL programs.” Usability evaluation of online learning programs. Ghaoui, Claude Editor. Idea Group Inc. pp. 88-109, 2003. [7] Nielsen, J. Molich, R. “Heuristic evaluation of user interfaces”. Proceedings ACM. CHI´90 Conference, pp. 249-256.1990. [8] Lewis C. Polson, P. Wharton, C. and Rieman, J. “Testing a walkthrough methodology for theory-based design of walk-up and use-interface”. Proceedings ACM CHI´90 Conference. pp. 235- 242, 1990. [9] Tullis T. S., Fleischman S., McNulty M., Cianchette C. and Bergel M. “An empirical Comparision of Lab and remote usability testing of websites”. Proceedings of usability Professionals Association Conference.http://hci.standford.edu/cs377/nardi- schiano/AW.Tullis.pdf (Consulted 31 of july 2007). [10] Hartson H.P. Castillo J. C. “Remote evaluation for post- deployment usability improvement”. Proceedings of AVI´98. Advance Visual Interfaces. ACM Press, pp. 22-29, 1996 [11] Waterson S., Landay J., Matthews T. “In the Lab and Out in the Wild: Remote Web Usability Testing for Mobile Devices”. Proceedings of ACM CHI 2002, pp. 296-297. [12] Hong J., Heer J., Waterson S., Landay J. “WebQuilt: A Proxy- based Approach to Remote Web usability Testing” [13] Matthews. T. “WebQuilt and mobile devices: a web usability testing and analysis tool for the mobile internet”. GUIR. Group for User Interface Research. 2001. AUTHORS T. Magal-Royo is Associated Research at the University Polytechnic of Valencia in the Research Center in Graphic Technology, Valencia, 46022 SPAIN (e-mail: tmagal@degi.upv.es). G. Peris-Fajarnes., is Director of Research Center in Graphic Technology at the University Polythecnic of Valencia. 46022 SPAIN (e-mail: gperis@degi.upv.es). I. Tortajada Montañana is Associated Research at the University Polythecnic of Valencia in the Research Center in Graphic Technology, Valencia, 46022 SPAIN (e-mail: itortaja@degi.upv.es). B. Defez Garcia is PostDoctoral Student at the University Polythecnic of Valencia in the Research Center in Graphic Technology, Valencia, 46022 SPAIN (e-mail: bdefez@degi.upv.es). Manuscript received 03 August 2007. Published as submitted by the author(s). iJIM – International Journal of Interactive Mobile Technologies – www.i-jim.org 4