PAPER THE EFFECTIVENESS OF MOBILE LEARNING IN THE FORM OF PERFORMANCE SUPPORT SYSTEM IN HIGHER EDUCATION The Effectiveness of Mobile Learning in the Form of Performance Support System in Higher Education http://dx.doi.org/ijim.v5i4.1692 N. Mileva Plovdiv University, Plovdiv, Bulgaria Abstract—Students in engineering education need high- quality educational materials, and what is the most impor- tant, continually up-to-dated course. The students need a possibility to self-assess their skills and knowledge as the learning steps for improvement - the courses, will provide them knowledge on the newest achievements in the field of engineering. The complexity of the knowledge and skills, necessary to perform successfully the tasks, determines the learners’ needs of “practical learning”. The view of learning process, as а performance-centered, is highly effective as a means of providing students in engineering education timely and relevant information. Performance-centered educa- tional systems have a strong potential to help students mas- tering job-related skills. This paper presents m-learning as a form of performance support system for educational and training purposes in higher education. m-Learning solutions integrate mobile devices within the learning process to help students perform a task by providing information, guidance, and learning experiences when and where they are needed. Index Terms—Higher education, mobile learning, per- fomance-centered learning, technology enhanced learning. I. INTRODUCTION A review of the literature is engineering education re- veals the following four major issues in delivering effec- tive instruction: Problems in Content Understanding, Problems in Concentration and Lack of Variety, Need for Multiple Explanations, and Lack of Assistance [1]: Performance-centered approach helps in solving all these four problems in teaching Engineering disciplines. The view of learning process, as а performance-centered is highly effective as a means of providing students in professional education timely and relevant information. As a new technology, performance-centered educational systems will move the traditional teaching systems to training closely related to the job. These systems have a strong potential to help students mastering job-related skills. Why we need to move from performance-centered edu- cational system to mobile performance-centered educa- tional system? The main reason is to contribute to the continued de- velopment of mobile learning and to address the imbal- ance between the availability of mobile devices and the lack of education and training provision on the sophisti- cated communications devices which every student and actively working person carries and uses constantly – ex- cept in education. We present mlearning as a form of performance support system for educational and training purposes. The advantages for trainees are derived from providing learners with a job aid in the context of their work:  puts training and performance support where the actual work takes place  allows new skills or knowledge to be immediately applied  enables training when it is needed  allows use of rich media when appropriate The advantages for students:  students have more flexibility and choice in where and when they learn outside of the wired (or un- wired) classroom.  students use the technology in their study that would enhance their readiness for tomorrow’s workplace where employers want graduates who know how to use technology for learning and working. Given the trend to lifelong learning, many “students” are working adults with full- or part-time jobs. Mobility offers them an opportunity to maximize learning time. II. MODEL FOR PERFORMANCE-CENTERED MOBILE LEARNING (PML) A. The concept of PML Integrating performance-centered learning and m- learning, results in a performance-centered mobile learn- ing (PML) approach, in which students receive perform- ance support via a mobile device when performing job- related tasks [2]. Important features of m-learning are that it makes learning processes more flexible and it offers the opportunity to individualize the learning material and en- vironment. By using a mobile device that meets the aforementioned requirements (i.e., small, easy to carry) the learner has access to information at the right time (i.e., just-in-time), for the specific learning situation at hand (i.e., just-in-case), which fits his or her need (i.e., just enough) and fits his or her learning style (i.e., just-for- me). These features are in line with the aims of perform- ance-centered learning. Performance-centered m-learning (PML) (i.e., mobile performance support systems) are similar to traditional PSS. That is, m-Learning solutions integrate mobile devices within the learning process to help students perform a task by providing information, iJIM ‒ Volume 5, Issue 4, October 2011 17 http://dx.doi.org/ijac.v4i4.1692 PAPER THE EFFECTIVENESS OF MOBILE LEARNING IN THE FORM OF PERFORMANCE SUPPORT SYSTEM IN HIGHER EDUCATION guidance, and learning experiences when and where they are needed. Some of the advantages of PML are: PML enables training when it is needed. PML allows use of rich media when appropriate. PML provides access to experts. PML builds a community of practice. PML knows user’s expertise. B. Instructional design for performance-centered m- learning It is important that the existing course is developed ac- cording to the principles of performance support for learn- ing [3]. First, it is important that the performance in edu- cation that is supported is related to performance of stu- dents’ future working environments. During the course students should be confronted with and trained for situa- tions they will also encounter in their future profession:  Identify the reference situation of a particular course. These are the professional settings where students are going to apply in practice what they have learned during the course.  Define a set of authentic problems and develop tasks related to a specific working environment.  Shift the instructional focus from knowledge and un- derstanding (i.e., the lower levels of the learning tax- onomy), towards solving real-world problems (i.e., the higher levels of the learning taxonomy).  Applying adequate summative performance-oriented assessment methods. Second, to design support for students to perform these authentic problems and especially learn from performing these tasks, the following instructional guidelines can be formulated:  Sequence the learning tasks from easy to complex.  Create opportunities for deliberate practice of the tasks.  Gradually diminish the amount of support provided to perform the task (scaffolding).  Provide a variety of instructional stimuli (resources) such as background information, examples and pro- cedures.  Allow constant access to learning resources.  Provide formative performance feedback.  Adapt the instruction to the level of knowledge and learning style of the students. This can be done by the system or the students themselves. Third, students’ learning will profit from adapting the type and amount of performance support to the students’ needs and learning style. This adaptation can be done by the system or by the students themselves. Mobile performance support systems can be imple- mented in education according to four possible instruc- tional design scenarios: 1. Mobile performance support courseware - mobile learning is interpreted as mobility of the content. A course provided via a wired device is simply repack- aged and transferred to a wireless mobile device 2. Industry-based mobile performance support systems - performance support system is implemented in higher education in a similar way as the support sys- tem is implemented in industry 3. Mobile social support systems - Learning as a life- long activity implies that it may happen anywhere and anytime. The advent of technology and its impact in our daily life provides an opportunity to support learning in different situations, irrespective of time and location 4. Integrated mobile performance support learning - mobile performance support is part of a blended solu- tion of knowledge distribution rather than a primary channel for content delivery. This scenario is based on the assumption that mobile learning is effective if considered as only one of the components of the overall programme of learning interventions. Another assumption that underlies this scenario is that when applying mobile performance support systems in education, instructors should use different materials, devices and resources as building blocks to develop a learning design. For each type of information the most suitable information, delivery device and re- source should be selected. This means, that only when delivering support through a mobile device has an added value for learning mobile performance sup- port should be implemented Practical instructional design of performance-centered m-learning deals with: small screens, graphics and illustra- tions, problems with data input and assessments. III. IMPLEMENTATION OF PML - REALIZATION OF SCENARIOS 4, MDIPSEIL SYSTEM For PML implementation a mobile learning (m- learning) environment was developed, called mDIPSEIL. This learning environment was based on the existing e- learning environment (Fig.1), DIPSEIL – performance- centred e-learning management system http://env.dipseil.net. The instructional design of the DIPSEIL and mDIPSEIL courses was based on the prin- ciples of performance support system (PSS), including a task description, task-specific training, reference informa- tion, and expert advice. There are two user types: Students are the learners of the educational content. They use their mobile devices to login to a simplified web-based system. The mobile de- vice should have an HTML-supporting web browser. Be- fore they login, they should have an account in the DIPSEIL system. The mDIPSEIL provides a simplified interface for studying the educational content. The main informational content, such as tasks list or learning mate- rials, is displayed at the centre of the screen, wrapped ac- cording to the resolution of the mobile device. Navigation menu is found underneath the informational content and helps the student navigating through the educational con- tent; Teachers are users, who develop, upload and organ- ize educational materials in mDIPSEIL system. Figure 1. Integration of mDIPSIEL with DIPSIEL 18 http://www.i-jim.org http://env.dipseil.net/ PAPER THE EFFECTIVENESS OF MOBILE LEARNING IN THE FORM OF PERFORMANCE SUPPORT SYSTEM IN HIGHER EDUCATION They can reorganize courses, modules and tasks devel- oped, upload, delete objects in the system and add or re- move them from their task specific objects. The environment from students point of view, from GSM, iPhone, PDA, Smart phones – http://m.dipseil.net The student has to be registered in DIPSEIL and the mod- ules from the courses have to be chosen, student’s course building. Selecting a task and choosing “Learn Selected” will show the Task Description. The system’s menu is always at the bottom of the screen. Use it to navigate through the different learning objects and to use available instruments. If there are videos or sounds, they will be presented as links, which allow you to download the mul- timedia content and study it. On Fig. 4 one can see examples with screens from the course “Information Technology”. A. Performance-centered m-learning environment evaluation To evaluate the effects of a system it is important to evaluate the effects of the system on the learning proc- esses for which it is designed [4]. We have performed pilot studies to investigate what effects mDIPSEIL had on learners’ performance (i.e., an objective measure) and how learners perceived the effectiveness of the system (i.e., a subjective measure). It was hypothesized that mDIPSEIL would help learners to adapt the learning envi- ronment to their needs and get just- in-time information, which would have a positive effect on their learning out- comes. Regarding the perceptions of the users it was hy- pothesized that the feeling of being in control of when to learn, what to learn and to ask for support when needed, would have a positive effect on students’ performance. We give here the procedure, participants and results from one of our pilots, performed at Plovdiv University 2009/2010 study year. Figure 2. http://menv.dipseil.net Figure 3. mDIPSEIL Editor Main task screen Participants were 37 first-year students (21 male, 16 female; M age = 20,7; sd = 2,7) who were enrolled in the course Information Technology at the Plovdiv University in Bulgaria. Students voluntarily decided whether they wanted to use mDIPSEIL to study the material (i.e. ex- perimental condition; n = 21) or to use DIPSEIL for this purpose (i.e., control condition; n = 16). Learning style questionnaire. To measure students’ learning style, the short 40-items version of Learning Style Questionnaire (LSQ) of Honey and Mumford (1992) was used. The questionnaire helps to indicate whether a Figure 4. Examples with screens from the course “Information Tech- nology” iJIM ‒ Volume 5, Issue 4, October 2011 19 http://m.dipseil.net/ PAPER THE EFFECTIVENESS OF MOBILE LEARNING IN THE FORM OF PERFORMANCE SUPPORT SYSTEM IN HIGHER EDUCATION student is an activist (i.e., prefers the challenges of new experiences and problems), a reflector (i.e., likes to first watch and think about actions and to review them after- wards), a theorists (i.e., think systematically step-by-step problems), or a pragmatist (i.e., likes to apply new learn- ing to actual practice). Learning outcomes. Students’ learning outcomes, were measured per task and by an overall score. Students’ scores (min 1, max 10 per task), for the 10 performance tasks they completed during the course were gathered. In addition, students received a final, overall score (min 2, max 6) on the course, computed by the teachers of the course. The 10 grade scores were transformed into 6 grade scores, because this is the official assessment scoring sys- tem in Bulgaria. Reflective questionnaire. A reflective questionnaire was developed to gain more insight into the perceptions of the students on several topics. These topics were (a) the us- ability of mDIPSEIL (e.g., it was easy to view the graphs), (b) the efficiency of mDIPSEIL (e.g., I could learn when- ever I wanted), (c) the effectiveness of mDIPSEIL on their learning (e.g., it helped me to learn better than with tradi- tional learning), and (d) the effects on their motivation (e.g., it was fun and appealing). Furthermore the question- naire was aimed at getting some information about the mDIPSEIL performance support system, asking how help- ful they found the support. Finally students were asked for their appreciation of mDIPSEIL in terms of action. The questionnaire’s items were formulated as state- ments and students were asked to indicate the extent to which they agreed with the statements on a 4-point Likert scale (1 = I totally agree, 4 = I totally disagree). Think aloud protocol. To identify possible issues or problems with mDIPSEIL, eleven students had to perform the task ‘Routing’, using mDIPSEIL on their mobile de- vice. They had to access the course, find the task, try all options for performance support and send the assignment back to the teacher. While performing the task students got the instruction to describe verbally what they are do- ing and experiencing. Students’ behaviour was recorded and their utterances were transcribed. During the first week of the course, all students filled out the 40-items Learning Style Questionnaire. Next, dur- ing 4 weeks, students studied the course material using either mDIPSEIL (i.e., on their Smartphone or iPhone) or DIPSEIL. Students in both conditions followed the course only at their university, in a classroom in which two teachers observed and assisted the students if necessary. At the end of the course all students in the experimental condition filled out the reflective questionnaire. In addi- tion, eleven students in the experimental condition per- formed a think aloud protocol. The learning styles were measured for the experimental group only. The majority of the students (n=8) appeared to have an activist learning style. Only one student scored a theorist learning style. The other students had a pragmatist (n=5) or reflector (n=7) learning style (Fig.5). Students’ learning outcomes. Students in the experi- mental condition outperformed the students in the control condition in 7 of the 10 tasks, whereas the students in the control condition had a slightly higher overall score (Fig.6). All differences between the conditions were not significant. In addition, no effect of learning style was found. Students’ perceived effectiveness: questionnaire. Re- garding usability, students indicated that it was easy to navigate through mDIPSEIL and to view graphics and presentations. With respect to the effectiveness students agree that mDIPSEIL helped them to learn the material better and more often. They disagree that it hampered their learning. However, also seven students indicated that mDIPSEIL did not fit their way of learning. When being asked specifically about the helpfulness of the perform- ance support system components, students indicated that all components were helpful, except for the expert advice. With respect to efficiency, students agree that mDIPSEIL allowed them to learn whenever and wherever they wanted and when needed. Regarding motivation students even strongly agree that learning with mDIPSEIL was fun and appealing to them. Regarding the appreciation of the course in terms of actions, students would recommend others to take a course in a m-learning environment or to do so themselves. They also disagreed that the course would have no added value. However, most striking was, that students also agree that they would have preferred a traditional course for this course material. No significant differences were found regarding the responds to ques- tionnaire of students with different learning styles. Students’ perceived effectiveness: think aloud protocol. The think aloud protocol also provided insight into the perceived and experienced effectiveness of mDIPSEIL. The analysis of the transcriptions and the interviews with the observers of the think aloud protocol did not show any problems with using mobile devices in this course. All students were able to accomplish the task efficiently and successfully. The data and the whole investigation could be found on http://mpss.dipseil.net. Project results section, Evaluation report from the pilots. Figure 5. Students’ learning styles distribution Figure 6. Score per learning task during the course 20 http://www.i-jim.org http://mpss.dipseil.net/ PAPER THE EFFECTIVENESS OF MOBILE LEARNING IN THE FORM OF PERFORMANCE SUPPORT SYSTEM IN HIGHER EDUCATION IV. CONCLUSIONS A. Conclusions from the performed investigations and in the light of the hypothesis It appears that learning the course on a mobile device does not contribute more to students’ learning than e- learning does. However, these results also indicate that it does not impair or hamper learning. Learners performed very well. This was also confirmed by the students who indicate that the use of a mobile device did not hamper their learning. The absence of a significant contribution to learning might be explained by the fact that the implemen- tation scenario used in this study, is one of the most straightforward scenarios. That is, the only difference be- tween the control and experimental group is the use of a mobile device. The content is the same, only the medium used differs. A more sophisticated implementation ap- proach in which, for example, the mobile device would provide the student with information just-in-time, just- when-needed, while performing a learning task, would have made m-learning more beneficial for learning com- pared to e-learning. Regarding the hypothesis about perceived effectiveness, the hypothesis was mainly confirmed. Students agreed that mDIPSEIL was easy to use, helped them to learn bet- ter, and whenever or wherever they wanted and its use was motivating. Also regarding the use of the components of the performance support system, students indicated that they use them frequently and that they are helpful. Overall the students positively appreciated the mDIP- SEIL and perceived it to be effective to learning. How- ever, some answers of the students in the questionnaire and interviews show that students are not completely con- vinced of the effectiveness and positive effects of mDIP- SEIL on learning. The most important lesson learned from the studies conducted is to assure that the way m-learning is imple- mented and designed in a course, has an added value to the learning of students. In the current studies, m-learning is interpreted as mobility of the content, enabling learners to consult information resources, ask for advice or any other type of support whenever they want and wherever they need it. With respect to instructional design, this sce- nario requires simple repackaging performance support that is already available on a wired device and transferring it to a wireless mobile device. It is important that the exist- ing course is developed according to the principles of per- formance support for learning. B. How we use these conclusions in on-going work Using our experience and investigations in implementa- tion of performance-centered approach in mobile learning, we are going further to the development and mainstream- ing Europe wide skills recognition and qualification ser- vice for m-learning job roles – in management, develop- ment and other individual management level charged with evaluating sourcing or implementing m-learning solutions. We started with m-Learning Manager job role in the con- text of growth of m-learning because of m-technology development and availability, and qualifications and pro- grams paucity at managerial (as premise and stimulus to design and development levels) levels in the field. Here we give answers to some possible questions, which could be raised: Why “m-learning”? There never was a technology as widely available to citizens as mobile telephony. This technology connects people working at different places and different education and learning paths with opportuni- ties for expert and peer feedback and co-learning. Mobile technology offers unprecedented possibilities for combin- ing the strengths of formal and non-formal education and professional internship. For the first time in the history of the use of technology in education and training, is a tech- nology that will cost the learners nothing, because they own the technology to be used. Why “m-learning job-role”? The new mobile learning arena imposes significant new design requirements of training programs - the ways they are structured and main- tained. The effective m-learning imposes specific usability requirement. The assessment of the mobile learning in terms of learning outcomes is similar in all VET systems but techniques in m-learning are specific. The validation of the assessed formal and non-formal m-learning should be done in accordance with the common European princi- ples. The quality assurance should be an integral part of the management of m-learning providing institutions. Why “m-learning manager job-role”? Our study shows that managerial level in the field will acts as premise and stimulus to development of job-roles in the design and development levels. Why “m-learning job-role now”? The m-learning is emerging as a new sector in education and training provi- sion, side by side with face-to-face education, distance education and e-learning. Just as distance education was recognized and accepted as a field in the 1970s when the great European Open Universities were founded and e- learning was accepted as a field in 1995 when the World Wide Web first became available to educators. We can say that we have been in the process of acceptance of m- learning from the beginning of the 21st century, along with 3G/UMTS and Smartphone. REFERENCES [1] Charles Downing, A Simple and Effective Method for Teaching Information Systems Vocabulary and Concepts in a Large Lecture Setting. Journal of Information Systems Education 13(3). [2] D. Keegan, N. Mileva, at al., “Mobile learning performance sup- port system for vocational education and training“, 2010, http://mpss.dipseil.net . [3] S. Stoyanov, P. Kommers, T. Bastiaens, and K. Martínez Me- diano, Performance Support System in Higher Engineering Edu- cation – Introduction and Empirical Validation. International Journal of Continuing Engineering Education and Lifelong Learning, 18 (4), 491-506, 2008. http://dx.doi.org/10.1504/ IJCEELL.2008.019647 [4] W. Kicken, S. Stoyanov, “Evaluation report from the pilots“, mPSS project 142788-LLP-1-2008-1-BG-LEONARDO-LMP. http://mpss.dipseil.net Project results. AUTHORS N. Mileva is a Professor in the ECIT Department, Uni- versity of Plovdiv, Bulgaria (nevena.mileva@gmail.com). This work was supported in part by the European Union Leonardo pro- ject: mLeMan 509960-2010-LLP-BG-LEONARDO-LMP. This article is an extended version from a paper presented at the International Confer- ence IEEE EDUCON2011, held at Princess Sumaya University for Technology in Amman, Jordan, 04-06 April 2011. Received June 9th, 2011. Published as resubmitted by the authors September 27th, 2011. iJIM ‒ Volume 5, Issue 4, October 2011 21 http://mpss.dipseil.net/ http://dx.doi.org/10.1504/%0BIJCEELL.2008.019647 http://dx.doi.org/10.1504/%0BIJCEELL.2008.019647 http://mpss.dipseil.net/ Miguel_iJIM_1648-5783-1-CE_edited_ssr.pdf Mileva_iJIM_1692-5786-1-CE_edited_ssr.pdf I. Introduction II. Model For Performance-Centered Mobile Learning (PML) A. The concept of PML B. Instructional design for performance-centered m-learning III. Implementation of PML - realization of Scenarios 4, mDIPSEIL System A. Performance-centered m-learning environment evaluation IV. Conclusions A. Conclusions from the performed investigations and in the light of the hypothesis B. How we use these conclusions in on-going work References Authors Nguyen_iJIM_1789-6220-1-CE_edited_ssr.pdf Redd_iJIM_1683-5787-1-CE_edited_ssr.pdf Khraisat_iJIM_1781-6094-1-CE_edited_ssr.pdf Lee_iJIM_1823-6253-1-CE_edited_ssr.pdf