International Journal of Internactive Mobile Learning (iJIM) - Volume 3, Issue 4, October 2009 USABILITY TESTING OF A COLLABORATIVE AND INTERACTIVE UNIVERSITY ON A MOBILE DEVICE Usability Testing of a Collaborative and Interactive University on a Mobile Device doi:10.3991/ijim.v3i4.905 T. Monahan, M. Bertolotto and G. McArdle School of Computer Science and Informatics, University College Dublin, Dublin 4, Ireland Abstract—The use of mobile devices for delivering learning tools is an attractive concept. Termed mobile learning (m- learning), this new technology allows people to participate in learning activities without being tied to a fixed location and provides users with convenient and flexible access to learn- ing resources anytime and anywhere. While many m- learning applications have been developed to date, most provide tools to help students’ with specific learning tasks rather than a general interface to online courses. Few sup- port online learning communities or allow users to download multimedia learning content. These features would engage mobile users and enable them to interact with one another, thus allowing them to participate in group learning activities despite their changing location. In this article, we describe an m-learning system which we have developed which aims to incorporate these facilities. This system provides access to multimedia learning resources and supports mobile users in an interactive synchronous learning environment with their desktop peers. Details of the evaluation techniques which we utilised to appraise the system are provided and the results are presented. Feedback suggests that the features offered by our system are benefi- cial for collaborative m-learning. Index Terms—Virtual Reality; M-Learning; Collaborative Learning; Mobile Computing. I. INTRODUCTION The emergence of mobile technologies has made it pos- sible to deliver a diverse range of services to users through their mobile devices. Coupled with wireless networks, these devices have potential to bring powerful applications directly to the user. Their portable nature makes them convenient for many to use while on the move and there- fore extends the boundaries where people can access digi- tal data. A wide variety of services are now available for tablet computers, Personal Digital Assistants (PDAs), smartphones and mobile phones. The mobility of these devices mean that they are particularly well suited to loca- tion-based applications such as tourist guides and naviga- tion aids. In addition, WebPages are being designed spe- cifically for these devices so that it is now easier than ever to access facilities such as online banking and online retail while not in a fixed location. Mobile technologies also show great potential within the area of education. Through mobile devices it is now possible to access learning services and content anytime and anywhere. The term mobile learning (m-learning) has been introduced to describe this concept and much re- search is ongoing examining the various ways that these devices can be used to effectively aid students in their learning. Firstly, the provision of administrative services has been explored which allows students to remain up-to- date with course information. Through their mobile de- vices, students can access information regarding course registration, timetables, contact details of lecturers, as- signment deadlines and assessment grades for example. Applications have been developed for the PDA platform which deliver this information [1], [2] while Short Mes- sage Services (SMSs) have been widely employed to reach students through their mobile phones [3], [4]. Some universities have also examined the use of SMSs to reduce dropout rates from their courses by sending messages of encouragement to students [5]. Feedback regarding these facilities has been very positive and, in particular, students found the use of SMSs convenient and liked the personal content they provided. Much research has also examined the potential of these mobile platforms for delivering learning content. Course material can be presented in a variety of ways whether it is in the form of short revision notes or quizzes sent via SMS, more detailed Multimedia Message Service (MMS) or course notes downloaded across a wireless network. The M-learning project (www.m-learning.org) developed a wide range of learning tools for mobile phones which delivered learning content to users through interactive SMS quizzes and Java-based games [6]. These tools were tailored for use on PDAs and were primarily used to de- velop aspects of literacy and numeracy in young adults who were no longer taking part in formal education. Learning modules to teach additional topics such as lan- guages, health and safety, the driver theory test and first aid were also developed. The MOBIlearn project (www.mobilearn.org) explored the possibility of deliver- ing context-based learning content to users via mobile devices [7]. They developed a system architecture capable of delivering content to users which can be personalised according to their preferences, previous interactions with the system and their current location. Learning material is also customised for different device types and network connections. Learning scenarios were developed for use on PDAs, mobile phones and tablet PCs using this archi- tecture. The Swedish telecommunications company, Ericsson (www.ericsson.com), are also involved in m- learning research projects. They focus on the develop- ment of course materials for display on mobile devices [8], [9]. Their work integrates technologies such as web browsing, streaming audio and video, SMS and MMS for presenting learning material to users through their PDAs, Smartphones and mobile phones. Evaluation studies for these research projects show that users responded well to these m-learning services. They showed great interest in the possibilities that mobile devices could offer for learn- ing and found the learning tools a useful and enjoyable way to learn [10], [11]. While these research studies have made major contribu- tions to the area of m-learning, most of the applications that have been developed for mobile devices to date have been tools which support individual learning where users 24 http://www.i-jim.org http://dx.doi.org/10.3991/ijim.v3i4.905� http://www.m-learning.org/� http://www.mobilearn.org/� USABILITY TESTING OF A COLLABORATIVE AND INTERACTIVE UNIVERSITY ON A MOBILE DEVICE access course information or work through learning con- tent alone. Little support for interaction among users is provided and thus collaborative work among students is limited. The benefits of supporting collaboration among students and of creating a learning community for online learners have been highlighted in many studies [12], [13], [14], [15], [16], [17]. Ultimately users feel less isolated in online learning courses when they are aware of others, and feel more comfortable in an e-learning environment when they know that help and support is available if required [15]. In addition, communication among students in an online learning community enables them to participate in group learning activities. This allows students to construct knowledge together, to further their understanding of top- ics through group discussion and to develop their problem solving abilities [16]. Our research examines the possibility of incorporating mobile users into an online learning community with their peers. In particular, we aim to develop an mlearning tool which allows mobile users to participate in synchronous learning activities with others. Additionally, given the popularity of online 3D environments such as Second Life (www.secondlife.com) and Active Worlds (www.activeworlds.com), together with previous research studies into the benefits that this type of environment can provide within the e-learning domain [18], [19], [20], [21], we also examine the use of 3D Graphical User Inter- faces (GUIs) for presenting the learning tools to students. This type of interface supports the incorporation of mul- timedia features into the system design and provides a stimulating and interactive learning experience for users. This article describes mCLEV-R, the mlearning system which we have developed, and details a number of evalua- tion studies which we carried out to ascertain the effec- tiveness of the collaboration and multimedia tools pro- vided. II. MCLEV-R mCLEV-R is the m-learning interface which we devel- oped as part of the Collaborative Learning Environments with Virtual Reality (CLEV-R) research project. This pro- ject examines the use of real-time communication tech- nologies and 3D GUIs for creating interactive online learning communities. A client-server architecture which incorporates a number of different components was devel- oped to support these technologies. This architecture is displayed in Fig.1. Clients can connect to the server side components to access various learning resources and to interact in real-time with other users. An e-learning inter- face was developed which allows users to connect to the system through desktop computers and laptops [22], while our m-learning interface allows users to access the learn- ing facilities through their PDAs. Figure 1. The Client-Server Architecture of CLEV-R The e-learning interface, shown in Fig.2, provides ac- cess to learning resources through a multi-user 3D envi- ronment [22]. Real-time text and audio communication methods are incorporated to allow users to interact with one another and participate in group activities. mCLEVR was developed to act as an accompaniment to this desktop interface, to allow mobile users to participate in learning activities. Similar to the desktop interface, mCLEV-R utilises 3D and synchronous communication technologies to present the m-learning tools to users. The mCLEV-R interface is presented to users through a series of web- pages. It consists of 2 main components; the 3D environ- ment and the communication interface. These components provide access to learning resources and allow mobile users to participate in real-time communication with other members of their course. Figure 2. The Desktop CLEV-R Interface iJIM – Volume 3, Issue 4, October 2009 25 http://www.secondlife.com/� http://www.activeworlds.com/� USABILITY TESTING OF A COLLABORATIVE AND INTERACTIVE UNIVERSITY ON A MOBILE DEVICE Figure 3. The 3D Environment of mCLEV-R A. The 3D Environment The 3D environment provided in the desktop CLEV-R interface is a large scale university setting with many rooms where lectures, group meetings and social activities can take place. Many users can be present in this envi- ronment at the same time and each user is displayed as an avatar [22]. While PDA's are currently not capable of sup- porting a multi-user large scale 3D environment like this, we wished to examine the use of 3D GUIs on the mobile platform. This would provide greater consistency between the desktop and mobile applications and also provide some valuable insight into the use of 3D GUIs on mobile devices. Thus the 3D environment provided in mCLEV-R is a single-user office style environment which provides access to learning resources. Users can navigate around the environment and interact with a number of 3D objects to access learning material, communication facilities and course announcements. The 3D environment provided in mCLEV-R is displayed in Fig.3. Fig.3(a) displays the download section of the 3D envi- ronment. From here, students can access course material uploaded by a tutor. They can download available files directly to their device across a wireless network and view them using an external application such as Pocket Acrobat Reader, Conduit Pocket Slides or Pocket Microsoft Word. Music and video files are also supported. The area of the 3D environment shown in Fig.3(b) provides access to other mCLEV-R features including the communication interface, course announcements and help facilities. B. The Communication Interface The communication interface of mCLEV-R provides access to the synchronous communication facilities. It allows mobile users to communicate in real-time with their course tutor and peers, and plays an important role in creating a sense of social presence for mobile users within a learning community. Real-time text and audio commu- nication, which can be used for participating in both social and learning activities, are supported in mCLEV-R. These facilities are linked to the text and audio communication features provided in the desktop CLEV-R system [22], thus mobile users and those connected through the desk- top interface can interact with each other. Figure 4. The mCLEV-R Text Communication Facility The text-chat facility allows mobile users to communi- cate with other users of the CLEV-R system using short text messages. As shown in Fig.4, users can send short messages to one another and the message dialogue is dis- played for all users. The dropdown menu provided is used to indicate the intended receiver of the message. Users can send public messages to all other connected users by se- lecting ’Everyone’ on this menu. Alternatively, they can select a user’s name to send them a private correspon- dence. This list is an important feature because it ensures that all users are aware of the presence of others. The mCLEV-R audio communication feature, dis- played in Fig.5, enables mobile users to broadcast live audio streams from their microphones and to listen to broadcasts of other users. A number of broadcasting 26 http://www.i-jim.org USABILITY TESTING OF A COLLABORATIVE AND INTERACTIVE UNIVERSITY ON A MOBILE DEVICE streams are available which represent various locations in the 3D environment used for the desktop CLEV-R inter- face [22]. When mobile users select an area which they wish to listen or broadcast to, using the dropdown menu provided, a list of all users currently in that part of the 3D environment is displayed so that mCLEVR users are al- ways aware of those listening to their broadcast. mCLEV- R is implemented using half-duplex data transmission. This means that only one user can broadcast to a particular stream at any one time. If a user is broadcasting to a stream, the ’Talk’ button will be disabled and a ’Busy’ status is displayed for all other users connected to that stream. Information regarding the current speaker is also displayed. Fig.5(a) displays the audio communication in- terface when the audio controls are available, while Fig.5(b) shows it in a busy state when the controls are being utilised by another user. The communication inter- face also provides access to other mCLEV-R components. The ’Home’ and ’Help’ buttons in the centre panel of this interface provide access to the 3D environment and the mCLEV-R help files respectively. The button labelled ’Hand’ was added to allow users to signal to the teacher that they have aquestion. Finally, the ’Exit’ button logs users out of the system. Figure 5. The mCLEV-R Audio Communication facility III. EVALUATION In order to determine the overall usability of mCLEVR and its effectiveness for m-learning, we carried out a number of evaluation studies. In particular 2 user evalua- tion studies were employed to gauge user reaction to the concept mCLEV-R presents and the functionality it pro- vides. This section discusses the techniques used in these evaluation studies while the next section presents the re- sults obtained. The 2 user evaluation studies examined the usability of mCLEV-R, users’ attitudes towards the interface and its effectiveness for m-learning. The International Standards Organisation (ISO) defines usability as ’The extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfac- tion in a specified context of use’ [23]. mCLEV-R was evaluated in accordance with this definition. Both user evaluations required the test-subjects to participate in a number of predefined tasks using mCLEV-R and provide feedback by means of a number of different question- naires. The user evaluations were carried out in conjunc- tion with the evaluations of the corresponding desktop interface. Thus, test-subjects were able to communicate with both mobile and desktop users during the evaluation sessions. 4 tasks were designed to ensure users were exposed to all features of mCLEV-R in a variety of different learning scenarios. A. Task 1 - Social Interaction The main aim of this task was to introduce users to mCLEV-R and, in particular, to the communication tools it provides. These communication facilities form the basis of all user interaction and collaboration within mCLEV-R and so it was important that users quickly became famil- iar with them. The task requested users to access the communication interface through the 3D environment and connect to the audio broadcast for one of the social rooms where all desktop CLEV-R users were located. All users participated in an icebreaker game using the text and audio communication facilities to converse with one another. They also accessed the announcements page during this session and collaborated with one another to decide on the topic of a group project which they would present through mCLEV-R in Task 3 of the evaluation study. B. Task 2 - Online Lecture This task requested users to attend a synchronous online lecture using mCLEV-R. Firstly, users were asked to download the lecture material through the 3D envi- ronment and open the lecture slides in an appropriate external application. They then accessed the communica- tion interface and connected to the audio broadcast for the lecture room where the tutor would present the learn- ing material. Test-subjects had to listen to the live lecture and follow the learning material simultaneously. They could ask questions via the text or audio communication facilities as desired. The objective of this task was to evaluate mCLEV-R as a means for accessing and downloading learning content, and to explore its use for attending and taking part in synchronous online learning activities. C. Task 3 - Collaboration Task 3 involved users participating in a group meeting where they presented and discussed the group project cho- sen in Task 1. The objective was to show users the support mCLEV-R provides for group learning activities such as project meetings or discussion groups. All users connected to the audio broadcast for a specified meeting room. Each user presented their findings on their chosen topic using the audio communication facility and listened to the find- ings of others. A group discussion on the presented mate- rial then took place. D. Task 4 - Free Session This final task consisted of a free session where users could further explore the various features of mCLEV-R and interact with others as they wished. The main objec- tives here were to ensure that users experienced all the different features of mCLEV-R and to observe ways in which users would interact informally within the learning environment. E. Evaluation Setup The first user study was carried out on completion of a first working prototype of mCLEV-R. Its aim was to dis- cover any usability issues that had not arisen during the development process. A number of usability problems were highlighted relating, in particular, to the design of the communication interface. These issues were addressed before the second evaluation was carried out. Further de- tails on the first user study and the results obtained can be found in [24]. In this article, we focus our attention on the results of the second evaluation study. In total, 12 users participated in this user evaluation. While a greater num- ber of users would have been desirable, it was difficult to iJIM – Volume 3, Issue 4, October 2009 27 USABILITY TESTING OF A COLLABORATIVE AND INTERACTIVE UNIVERSITY ON A MOBILE DEVICE obtain volunteers to participate in the user trials. The us- ers consisted of 9 males and 3 females and included 9 university students, 2 professionals and 1 secondary school teacher. The majority of these users were univer- sity students since they are the target users of mCLEV-R. Further details regarding the user sample are shown in Table 1. All users had good to excellent computer skills. This level of computer literacy among university students is realistic nowadays as the majority use computers fre- quently throughout their studies. Thus, the sample set is a reasonable representation of the prospective end-users. 2 users also had previous experience of using personal or- ganiser tools, Geographic Information Systems and web browsing on mobile devices. TABLE I. USER PROFILES Male Female No. of Test-Users 9 3 Average Age 24.8 28 Previous Experience with Mobile Devices 2 0 Previous experience of E- Learning 5 2 In order to receive feedback from users about their ex- perience with mCLEV-R, a number of different question- naires were used. Firstly, a task evaluation questionnaire was designed which was filled in on completion of each task. This consisted of the After Scenario Questionnaire (ASQ) and some questions designed specific to mCLEVR and the functionality it provides. The ASQ, which was designed by IBM [25], is a 3 item questionnaire used to determine user satisfaction upon completion of a task. In particular, it examines their satisfaction regarding the ease of completing the task, the time taken to complete the task and the support information available when completing the task. It is designed using a 7-point likert scale an- chored at 1 by Strongly Agree and at 7 by Strongly Dis- agree. [25] and [26] describe a psychometric evaluation of the ASQ which shows that the 3 items can be condensed to one scale through summation, thus giving an overall satisfaction score for the task. The additional questions of the task evaluation questionnaire dealt with individual features of mCLEV-R and user feeling towards their ef- fectiveness for carrying out the task in question. An overall evaluation questionnaire was also distributed to users after their last session with the system. Again, this consisted of a number of different components. The Computer System Usability Questionnaire (CSUQ), also developed by IBM [26], consists of 19 questions which assess user satisfaction with overall sys- tem usability. It can be divided into 3 subscales, namely System Usefulness (questions 1-8), Information Quality (questions 9-15) and Interface Quality (questions 16-18), to obtain feedback on these aspects of the system. The final question relates to overall satisfaction with the sys- tem, and so does not fall into any of these subcategories. Similar to the ASQ, this questionnaire is designed using a 7-point likert scale anchored at 1 by Strongly Agree and at 7 by Strongly Disagree. The CSUQ and its 3 subscales can also be condensed to a single scale score through summation [26]. Another component on which the overall evaluation questionnaire was based was the Microsoft Product Reaction Cards [27]. They were developed by Microsoft as a means of determining the desirability of a product or system. They consist of 118 words of which approximately 60% are positive and 40% are negative. Users are asked to select the words that they feel best de- scribe the system or the feelings they had when using the system. This provides valuable qualitative feedback re- garding users’ reaction to the system in a short amount of time. Finally, some questions specific to mCLEV-R were designed to receive feedback on different aspects of the system including the communication interface, the 3D environment, users’ sense of presence in an online com- munity and general thoughts regarding mobile devices for learning. Additional user comments were also encouraged. IV. RESULTS This section discusses the results obtained using the evaluation questionnaires described. User reaction in rela- tion to the effectiveness of mCLEV-R for completing each of the tasks, the overall usability of the system, their social presence with an online learning community and the no- tion of collaborative m-learning are presented. A. Task Results The average ASQ usability ratings given by participants for each of the evaluation tasks are shown in Table 3. This score can range between 1 and 7, where 1 is positive and 7 is negative. Thus we can see that all 4 tasks were rated positively by the test-subjects. In general, the users were satisfied that the tools pro- vided by mCLEV-R are effective for participating in each of the different learning and social scenarios. Some users did experience difficulty using the device input methods of the PDA. In particular, they found inputting text using the virtual keyboard and stylus slow and cumbersome. These users were, however, inexperienced with mobile devices prior to this evaluation and once they become familiar with this input method, their typing speed should improve significantly. Another issue raised by test- subjects was that they disliked navigating between differ- ent applications to view lecture material and access the communication facilities. However, the screen size of PDAs is a severe limitation on the amount of information that can be displayed at any one time. It is thus not possi- ble to provide the functionality of mCLEVR without al- ternating between different applications. Despite these issues, all users successfully completed each task and ac- tively participated in group activities. TABLE II. PARTICIPANTS’ SATISFACTION WITH MCLEV-R FOR THE EVALUATION TASKS Task Average ASQ Rating Task 1: Social Interaction 2.94 Task 2: Online Lecture 5.28 Task 3: Collaboration 2.62 Task 4: Free Session 1.97 B. Computer System Usability Questionnaire The CSUQ was used to determine the overall system usability. Table 3 shows the average usability rating given by participants for the system, as well as the System Use- fulness, Information Quality and Interface Quality sub- categories. Once again this score ranges between 1 and 7 where 1 is positive and 7 is negative. 28 http://www.i-jim.org USABILITY TESTING OF A COLLABORATIVE AND INTERACTIVE UNIVERSITY ON A MOBILE DEVICE TABLE III. PARTICIPANTS’ USABILITY RATINGS OF MCLEV-R Average Standard Deviation Overall Usability 2.66 1.131 System Usefulness 2.78 1.421 Information Quality 2.65 0.996 Interface Quality 2.47 1.267 mCLEV-R received a positive overall usability rating. However, the high standard deviations displayed in Table 4 suggest that user opinion varied and that the averages were somewhat affected by negative feedback from some users. The participants’ overall satisfaction with mCLEV- R is shown in Fig.6. 11 of the users agreed that they were satisfied with mCLEV-R giving this question a rating be- tween 1 and 3 on the 7-point scale. Only 1 user disagreed strongly with this. This user was unfamiliar with PDAs prior to this evaluation study and strongly disliked the mobile platform. They felt that the screen size was insuf- ficient for effectively displaying information to the user and that the device input methods were restrictive. They found mCLEVR frustrating to use and disliked alternating between different applications. They did, however, com- ment that an experienced PDA user might be better able to use the interface and that the functionality provided by the system would be useful when it is not possible to login on a desktop computer. The 3 subscales of the CSUQ are now further explored for this evaluation study. Figure 6. Users’ Overall Satisfaction with mCLEV-R C. System Usefulness The 12 participants of this evaluation study gave mCLEV-R an average system usefulness rating of 2.78. However, average individual ratings for the 8 questions in this subcategory ranged from 1 to 5.5 as is reflected in the high standard deviation of 1.421. Some results from this section of the questionnaire are analysed in Fig.7. The majority of participants were satisfied with the ease of using mCLEV-R and 10 agreed that it was easy to learn how to use it. While 9 agreed that they could effectively complete their work using the system, only 7 felt that they could complete their work efficiently. The remaining us- ers felt the stylus interaction method was slow, particu- larly for typing messages, and that switching between various applications was time consuming. This is primar- ily due to their inexperience with mobile devices and as they become more familiar with these devices, their inter- action speeds with mCLEVR should improve. Despite this, 9 felt comfortable using the system during their m- learning sessions. Figure 7. Users’ Feedback: System Usefulness D. Information Quality This category is concerned with the quality of user sup- port provided with the system including error notification, user instructions and help files. The average rating given by users for this subscale was 2.65 with a standard devia- tion of 0.996. Error notification proved the most negative feature of the system within this category. As shown in Fig.8, 6 users rated this feature poorly, while 3 did not respond to this question. Many users did not experience any error messages and some encountered problems which they expected to be reported as errors. However, many of the problems encountered by users during the user sessions were not directly related to the system. In- stead they were device related issues such as difficulties navigating between different applications on the PDAs or problems using the stylus for interaction. For example, some users accidentally tapped the stylus on hyperlinks within the 3D environment and as a result were redirected to the webpage of another mCLEV-R component. While it would be inappropriate to report an error in this case, the problem is dealt with in other ways by ensuring that all mCLEV-R webpages have adequate support for navigat- ing between different system components. Despite this, 8 of the participants agreed that they recovered from mis- takes quickly and easily. Figure 8. Users’ Feedback: Information Quality (1) In relation to information provided with the system in- cluding on-screen messages, help files and user instruc- tions, overall feedback was more positive (see Fig.9). 8 users felt the information was clear and 11 considered it easy to understand. The majority (9) found it effective in helping them complete their work, while 11 felt that the organisation of information on the screen was clear. iJIM – Volume 3, Issue 4, October 2009 29 USABILITY TESTING OF A COLLABORATIVE AND INTERACTIVE UNIVERSITY ON A MOBILE DEVICE Figure 9. Users’ Feedback: Information Quality (2) E. Interface Quality An interface quality rating of 2.47 was determined for mCLEV-R. Fig.10 displays the feedback for this category and as shown user response was generally positive. 10 of the test-subjects considered the system interface pleasant with only 1 user stating that they disliked using it. 10 of the users agreed that the system has all the capabilities they would expect it to have. Suggestions from the re- maining users include adding functionality for full-duplex audio communication rather than the ’push to talk’ method currently employed, and providing support for users to share live web-cam broadcasts with one another. Both of these proposals are possible within the mCLEV-R system framework. Figure 10. Users’ Feedback: Interface Quality F. Microsoft Product Reaction Cards Overall user reaction towards mCLEV-R was also re- corded using the Microsoft Product Reaction Cards. Words selected by the users give qualitative feedback of the system, highlighting their thoughts and overall impres- sions. In total, 84 out of the 118 words were selected to describe mCLEV-R. Of these, 74 (88%) were positive and 10 (12%) were negative. Table 4 shows the most highly occurring words selected by users. As shown, these are all positive and were selected by 5 users or more. The nega- tive words selected had a lower occurrence (3 users or less). The words with greatest occurrence included useful, collaborative, creative and fun which are all attributes which we aimed for in our initial system design and speci- fication. G. mCLEV-R Features The communication facilities provided in mCLEV-R play an important role in creating a sense of community among connected users. Through various features of the communication interface, we aim to promote awareness of others and form an online learning community allowing students to interact and learn together. Users were asked if this component was effective in achieving these goals. The majority of the participants stated that they were aware of all other users, both those connected through mCLEV-R and those using the desktop CLEV-R inter- face. Despite some initial problems with the device input methods, the majority of users found the text and audio communication facilities easy to use and effective for communicating with others. When using audio communi- cation, all users found the feature displaying the names of other users in the same location beneficial. 11 users felt part of a group when using mCLEVR (see Fig.11). In ad- dition, 11 of the users agreed that the presence of others enhanced their learning experience and kept them engaged in the learning activities. TABLE IV. MOST FREQUENT WORDS RETURNED FROM THE MI- CROSOFT PRODUCT REACTION CARDS Term Occur. Term Occur. Useful 9 Accessible 5 Collaborative 7 Appealing 5 Creative 7 Easy to use 5 Fun 7 Effective 5 Convenient 6 Efficient 5 Novel 6 Helpful 5 Personal 6 Intuitive 5 Valuable 6 Professional 5 Figure 11. Users’ Feedback: Social Presence within mCLEV-R Feedback was also gathered to determine users’ reac- tion to using a 3D interface for accessing information and system features, and to determine if it was appealing and engaging for the user. 11 users found the 3D interface appealing and 9 found it easy to navigate within it on the PDA. All participants successfully accessed the commu- nication facilities, announcements page and downloaded course material through the 3D interface and 9 of the users rated the intuitiveness of accessing these facilities posi- tively. In comparison to a text-based interface, 10 found the 3D environment more engaging. While users did ex- perience some difficulties with the system during this evaluation study, their response to the system as a whole was very positive. All test-subjects see the benefits in pro- viding learning tools for mobile users. Further feedback regarding the use of mCLEV-R for m-learning is dis- played in Fig.12. 10 agreed that the interface provides a good medium for learning and 11 feel this is especially 30 http://www.i-jim.org USABILITY TESTING OF A COLLABORATIVE AND INTERACTIVE UNIVERSITY ON A MOBILE DEVICE true when it is not possible to be present in a fixed loca- tion. 10 out of the 12 users stated that they would use mCLEV-R if it were provided with an e-learning course and that they had a high enjoyment level when using the system. Figure 12. Users’ Feedback: mCLEV-R for mLearning DISCUSSION AND FUTURE WORK The emergence of mobile devices has led to an interest in the development of applications which provide various services for mobile users. M-learning is one such service and many applications which provide learning tools have been developed for a multitude of mobile platforms. Our research also examines this area. In particular, we aim to provide an m-learning tool which supports mobile users within an online learning community. This article dis- cusses our system, mCLEV-R, which provides facilities for mobile users to access multimedia learning material and allows them to converse and learn together with their peers through a 3D interface and synchronous communi- cation methods. Many different aspects of mCLEV-R have been validated including its overall usability, user reaction to the 3D interface, the effectiveness of the com- munication facilities for learning and social activities, and its value as an m-learning tool. Overall the results are very satisfactory. In general, users enjoyed the available functionality and agreed that mCLEV-R would be a useful system for learn- ing on the move. The synchronous communication meth- ods created a sense of community among the participants and proved sufficient for supporting collaboration among them. Moreover, users responded well to the 3D interface and found it more engaging than a text-based alternative. Navigating between various applications and inputting text via a virtual keyboard and stylus were the main diffi- culties experienced. While such issues cannot be over- looked, they are more related to users’ inexperience with PDAs than to the mCLEV-R application. These results suggest that the functionality provided by mCLEV-R is beneficial for m-learning. Access to learning material while on the move is in itself an extremely useful feature, however, when coupled with support for synchronous communication, the benefits increase significantly. M- learning no longer needs to be a solitary activity. Instead users can participate in group activities and get help from others as it is required. There are a number of ways in which the mCLEVR system can be extended to provide further support for mo- bile users. Since mCLEV-R was developed as a prototype to provide a proof of concept of the benefits that interac- tive and synchronous communication technologies pro- vide for m-learning, our research focused on these aspects of the system development and did not include other fea- tures that are present in current learning management sys- tems. Some of these features such as asynchronous com- munications technologies could be incorporated into the mCLEV-R system design. While the synchronous com- munication methods currently supported provide many benefits, they place time constraints on user interactions as they can only communicate with others when they are simultaneously logged in to the system. Asynchronous communication methods offer greater flexibility in this respect and also their lack of immediacy allows users to contemplate their contributions to the discussion before posting messages. Thus, these communication technolo- gies may be more suitable than synchronous methods for certain learning scenarios. Much research is currently ongoing within other areas of e-learning and m-learning. Techniques such as person- alisation of learning content, adaptive user interfaces and the provision of software agents to support students are being investigated. The incorporation of these features into the CLEV-R system could also be explored. For ex- ample, user profiles detailing student preferences could be stored to allow the mCLEV-R interface to personalise content for individual users. The integration of these fa- cilities would further enhance the learning environment supported by the CLEV-R architecture and further ad- vance the benefits of mobile devices for e-leaning. ACKNOWLEDGMENT We would like to thank all the test users who assisted with the evaluation of CLEV-R. We also thank our fund- ing agencies, the culture 2000 Project TARCHNA EC Grant Agreement n. 2004-1488/001-001, CLT-CA22 and the Irish Research Council for Science, Engineering and Technology (IRCSET). Research presented in this paper was funded by a Strategic Research Cluster grant (07/SRC/I1168) by Science Foundation Ireland under the National Development Plan. The authors gratefully ac- knowledge this support. REFERENCES [1] F. Lehner, H. Nosekabel and H. Lehmann, ”Wireless E-Learning and Communication Environment: WELCOME at the University of Regensburg”, e-Service Journal 2(3), 23-41, 2003. (doi:10.2979/ESJ.2003.2.3.23) [2] D. Corlett, M. Sharples, S. Bull and T. Chan, ”Evaluation of a Mobile Learning Organiser for University Students” Journal of Computer Assisted Learning 21(3), 162-170, 2005. (doi:10.1111/j.1365-2729.2005.00124.x) [3] T. Brown, ”Towards a Model for M-Learning in Africa”, Interna- tional Journal on E-Learning 4(3), 299-315, 2005. [4] E. Mikleia, ”Mobile Devices to be Applied as Supporting Tools in Research Methods Class for Undergraduate Students” Proc. 7th International Conference on Learning Sciences’, Bloomington, In- diana, USA, pp. 918-919, 2006. [5] J. Nix, J. Russell and D. Keegan, ”Mobile Learning/SMS (Short Messaging System) Academic Administration Kit”, European Dis- tance and E-Learning Network 2006. Retrieved June 2008 at http://www.edenonline.org/contents/publications/SMS/Ericsson.M obile.A5.pdf [6] J. Attewell “Mobile Technologies and Learning; A Technology Update and M-Learning Project Summary”, Technical report, Technology Enhanced Learning Research Centre, Learning and Skills Development Agency, Retrieved May 2009 at http://www.m-learning.org/docs/The m-learning project - technol- ogy update and project summary.pdf [7] P. Lonsdale, C. Baber, M. Sharples and T. Arvanitis, ”A Context Awareness Architecture for Facilitating Mobile Learning”, Proc. iJIM – Volume 3, Issue 4, October 2009 31 http://dx.doi.org/10.2979/ESJ.2003.2.3.23� http://dx.doi.org/10.1111/j.1365-2729.2005.00124.x� http://www.edenonline.org/contents/publications/SMS/Ericsson.Mobile.A5.pdf� http://www.edenonline.org/contents/publications/SMS/Ericsson.Mobile.A5.pdf� http://www.m-learning.org/docs/The m-learning project - technology update and project summary.pdf� http://www.m-learning.org/docs/The m-learning project - technology update and project summary.pdf� USABILITY TESTING OF A COLLABORATIVE AND INTERACTIVE UNIVERSITY ON A MOBILE DEVICE Learning with Mobile Devices’, Learning and Skills Development Agency, London, UK, pp. 79-85, 2003. [8] D. Keegan, ”The Future of Learning: From eLearning to mLearn- ing”, Institute for Research into Distance Education, Fern Univer- sity, Hagen, Germany, 2002. Retrieved June 2008 at http://eric.ed.gov/ERICDocs/data/ericdocs2sql/content storage 01/0000019b/80/1a/b6/8e.pdf [9] D. Keegan,”Mobile Learning: The Next Generation of Learning”, Distance Education International, 2005. Retrieved June 2008 at http://learning.ericsson.net/mlearning2/files/workpackage5/book.d oc. [10] J. Attewell, ”Mobile Technologies and Learning; A Technology Update and M-Learning Project Summary” Technical Report 2005, Technology Enhanced Learning Research Centre, Learning and Skills Development Agency, Retrieved June 2008 at http://www.m-learning.org/archive/docs/The m-learning project - technology update and project summary.pdf. [11] MOBIlearn, MOBIlearn Project Final Report, Technical Report 2005, http://www.mobilearn.org/results/results.htm. [12] D. Kitchen and D. McDougall, ”Collaborative Learning on the Internet”, Journal of Educational Technology Systems 27(3), 245- 258, 1999. [13] J. Laister and S. Kober, ”Social Aspects of Collaborative Learning in Virtual Learning Environments”, Proc. Networked Learning Conference, Sheffield, UK. March 26-28, 2002. [14] S. Redfern and N. Naughton, ”Collaborative Virtual Environments to Support Communication and Community in Internet-Based Dis- tance Education”, Journal of Information Technology Education 1(3), 201-211, 2002. [15] G. Salmon, ”E-tivities: The Key to Active Online Learning” Routledge- Falmer Press, London, UK, 2002. [16] I. Hamburg, C. Lindecke and H. ten Thij, ”Social Aspects of ELearning and Blending Learning Methods”, Proc. 4th European Conference on E-Commerce, E-Work, E-Learning, E-Health, EBanking, E-Business, On-line Services, Virtual Institutes, and their Influences on the Economic and Social Environment (E- Comm-Line), Bucharest, Romania, pp. 11-15. September 25-26, 2003. [17] V. Thurmond and K. Wambach, ”Understanding Interactions in Distance Education: A Review of the Literature”, Instructional technology and distance learning 1(1), 9-33, 2004. [18] Y. San Chee and Y. Liu, ”Grounding Concept in Percept: Learn- ing Physics Experientially in Multi-User Virtual Worlds”’, Proc. IEEE International Conference on Advanced Learning Technolo- gies, Joensuu, Finland, pp. 340-344, 2004. [19] C. Bouras and T. Tsiatsos, ”Educational Virtual Environments: Design Rationale and Architecture”, Multimedia Tools and Appli- cations 29(2), 153-173, 2006. (doi:10.1007/s11042-006-0005-7) [20] C. Liu,”Second Life Learning Community: A Peer-Based Ap- proach to Involving More Faculty Members in Second Life”, Proc. Second Life Education Workshop at Second Life Community Convention, San Francisco, California, USA, pp. 6-10, 2006. [21] M. Childress and R. Braswell, ”Using Massively Multiplayer Online Role- Playing Games for Online Learning”, Distance Edu- cation 27(2), 187-196, 2006. (doi:10.1080/01587910600789522) [22] G. McArdle, T. Monahan and M. Bertolotto, “Using Multimedia and Virtual Reality for Web-Based Collaborative Learning on Multiple Platforms”, Ubiquitous and Pervasive Knowledge and Learning Management, pp. 118-157. IDEA Group Publishing. Hershey, Pennsylvania, USA and London, UK, 2007. [23] ISO-9241-11, ”Ergonomic Requirements for Office Work with Visual Display Terminals (VDTs), Part 11: Guidance on Usability Specification and Measures, Technical report”, 1998, Retrieved June 2008 at http://www.idemployee.id.tue.nl/g.w.m.rauterberg/lecturenotes/IS O9241part11.pdf [24] T. Monahan, G. McArdle and M. Bertolotto, ”mCLEV-R: Design and Evaluation of an Interactive and Collaborative M-Learning Application” International Journal of Emerging Technologies in Learning (iJET), 2(2), 47-53. Open Journal Systems Press, 2007. [25] J. Lewis, ”Psychometric Evaluation of an After-Scenario Ques- tionnaire for Computer Usability Studies: The ASQ”, ACM SIG- CHI Bulletin 23(1), 7881, 1991. [26] J. Lewis, ”IBM Computer Usability Satisfaction Questionnaires: Psychometric Evaluation and Instructions for Use”, International Journal of Human-Computer Interaction 7(1), 5778, 1995. (doi:10.1080/10447319509526110) [27] J. Benedek and T. Miner, ”Measuring Desirability: New Methods for Evaluating Desirability in a Usability Lab Setting” Proc. Us- ability Professionals Association Conference, Orlando, Florida, USA, July 8-12, 2002. AUTHORS Dr Teresa Monahan received a BSc (Hons) degree in Computer Science from University College Dublin (UCD) in 2003. She subsequently undertook a PhD degree in UCD where her studies involved research into online learning environments, 3D interfaces and mobile learning. She obtained her doctorate in 2007. Her research interests include collaborative e-learning systems, virtual reality and mobile applications. Dr Michela Bertolotto received a BSc and a PhD de- gree (1998) in Computer Science from the University of Genova. Subsequently she held a postdoctoral position at the National Center for Geographic Information and Analysis at the University of Maine (1998-2000). Dr Ber- tolotto is currently a Senior Lecturer in the School of Computer Science and Informatics at University College Dublin. Her research interests include spatial data han- dling, virtual reality and geometric modeling for GIS. Dr Gavin McArdle received a BSc (Hons) degree (2003) in Computer Science from University College Dublin. He subsequently completed a PhD degree in Uni- versity College Dublin in 2008. Gavin is currently a post- doctoral fellow within the Strategic Research in Advanced Geotechnologies (StratAG) Cluster. Gavin’s research in- terests include HCI, collaborative environments, location based services and personalisation. Submitted 20 April 2009. Published as resubmitted by the authors on 9 October 2009. 32 http://www.i-jim.org http://eric.ed.gov/ERICDocs/data/ericdocs2sql/content� http://dx.doi.org/10.1007/s11042-006-0005-7� http://dx.doi.org/10.1080/01587910600789522� http://dx.doi.org/10.1080/10447319509526110�