Microsoft Word - 17-800-ed.doc Engineering, Technology & Applied Science Research Vol. 6, No. 5, 2016, 1207-1211 1207 www.etasr.com S. Vairamuthu and S. Margret Anouncia: Reconnoitering Students' Satisfaction of an Online Based… Reconnoitering Students' Satisfaction of an Online Based Assessment System to Improve Usability using PSO: An Examination into a Problem Solving and Programming Course S. Vairamuthu School of Computing Science and Engineering VIT University Vellore, India svairamuthu@vit.ac.in S. Margret Anouncia School of Computing Science and Engineering VIT University Vellore, India smargretanouncia@vit.ac.in Abstract—End users’ computing environment has been consistently changing in recent years due to the major advancements of Information Technology. This work aimed to measure the level of users’ satisfaction and provide feedbacks for continuous improvement of a course offered in an academic institution. End users here were the students enrolled for the course and the faculty members who offered the same and also acted as an assessor for the assessments. All assessments were scheduled and conducted online. This study was conducted to focus on two different aspects: Measuring User satisfaction and investigating information systems measures to improve usability using nature inspired computing. For user satisfaction analysis, the study employed the Multicriteria Satisfaction Analysis. The course considered was a problem solving and programming course offered to the fresh students enrolled in the first year of the undergraduate degree program in the academic institution during 2015-2016. To identify the factors for improved usability, PSO was employed in this work. Keywords-user satisfaction; multi-criteria user satisfaction analysis; web portal; information systems I. INTRODUCTION Academic and traditional portals, also termed as web portals, play a nominal role in recent technological innovations, as a gateway for services and information from various sources in a united way with the help of a single user interface [2]. These portals facilitate search capabilities, database access, registration for new users and personalization, etc. When employed in academic institutions their role slightly differs. Many universities are making an attempt to introduce their services through academic portals, third party ones or institution based. Such portals may be used for education, notification and even examination purposes. As these kinds of services are offered by third parties sometimes, it becomes mandatory that the investment and the efforts made for this purpose should satisfy the expectations of all the stake holders. This can be measured based on the level of satisfaction of all end users. It should be noted that an individual’s attitude to use of computers and the related actions to accomplish a task in an organization can be considered as end user satisfaction [3]. Evaluation of online based assessment systems based on usability factors would be more interesting and difficult when compared with normal applications for two major reasons: locality of access and access environment. This kind of usability testing of assessment systems would normally have a greater impact on various kinds of users on a day-to-day basis otherwise the difficulties encountered will cause the users not to choose the same category for these kinds of assessment purposes. II. LITERATURE SURVEY Usability has been evolving since 1980’s with its roots in HCI (Human Computer Interaction). It examines the interaction between users and the computers so as to make it more meaningful and effective. HCI investigates the way developing and using interactive applications affect users, societies and organizations, and in [4] the impact of not proper designed was investigated. In [5], it was stated that HCI deals with user tasks, actions or behaviors and goals with computerized systems and interfaces. In [6], it was claimed that evaluating usability contains methods to measure usability characteristics and to identify specific problems. In [7], it was claimed that there was not unique agreement on a single definition of usability. In [8], usability was defined as an effort to calculate friendliness to yield some measurable attributes of end users like users skill, time to learn, productivity from using the system and evaluation of users approach with the system. The financial aspect is also important. For example, in [9], it was shown that poorly designed websites yielded negative impacts financially. Organizations may move to alternatives if any website fails to meet the end users’ needs and expectations [10]. In [11], it was observed that web services to implement business processes had low usability even though there were significant advances and investments made. In [12], it was stated that easy navigation was one of the critical components Engineering, Technology & Applied Science Research Vol. 6, No. 5, 2016, 1207-1211 1208 www.etasr.com S. Vairamuthu and S. Margret Anouncia: Reconnoitering Students' Satisfaction of an Online Based… of usability, and an attempt to specify the term was made in [7]. In [13], a methodology to evaluate the usability of web portals that encompassed task-based testing, questionnaire based on satisfaction and semi-structured interviews was proposed. In [14], several usability criteria were mentioned, broadly categorized as organization of contents and readability, links and navigation, design of user interface, effectiveness and performance and information on education. In [15], a model for web usability evaluation was designed to evaluate higher educational websites that involved students of the concerned institutions with a similar categorization (five major elements). In [16], Website Analysis & Measurement Inventory (WAMMI) was used for assessing a university’s website. The Multicriteria Satisfaction Analysis (MUSA) method has been proposed as a mean to measure customer’s satisfaction [17]. The underlying principle in [17] was to aggregate the individual’s collective value function claiming that the global satisfaction of any customers depend on a group of criteria that represents various dimensions of their service characteristics. In [18], authors performed a study based on questionnaire to measure the usability of six universities of Lebanon. The questionnaire measured various usability attributes among various users. In [19], it was demonstrated that usability would be a paired act–insufficient functionality that would reduce the application’s use when clutter and complexity made any interface very difficult to be used. In [20, 21], authors claimed that knowledge-intensive workers and organizations relied more on the available services at web portals which can incorporate business processes playing a vital role in organization’s functionality. In [22], six categories to measure usability along with the methods to collect analyze and present the data related to usability were proposed. In [23], it was debated that information sources together with users’ activity in web portal accounted towards appropriate observation and modeling. Further, techniques to determining model parameters and presented models based on mathematics towards portal operation were also proposed. In [24], it was stated that usability improvement in organization portals require effective management work in several stages and suggested to approach any information systems through several levels. In [25], the use of analytics was recommended as it provide timely intelligence for business that would best be suitable for maintaining and gaining competitive advantage. In [26], it was claimed that analytics enabled the evidence-based management. In [27], it was argued that the influences of IT were often influenced by human and indirect, environmental and organizational factors and so measuring information systems success was complex and illusive. In [28-30], it was proclaimed that system quality was strongly related to satisfaction of the user in any knowledge management system. III. OBJECTIVES The objective of this study was to evaluate the usability of a web based assessment system for one of the courses offered inan academic institution. The study focused insight into the following: • Analyzing the various usability attributes. • Getting students’ subjective opinion. • Checking individual’s ability for completion of tasks. • Checking accessibility compliance. • Identifying the factors to improve usability of the system using PSO IV. OVERVIEW OF COURSE STRUCTURE AND ORGANIZATION The course taken for this work is problem solving and programming which was offered in an academic institution in the first year of students enrolled in 2015-16. The institution makes use of the skillrack portal for this course only for assessment process. The course is offered as a lab based course for a period of six months. The major stake holders of this course are students and faculty members offering the course. The maximum enrollment is 65 per class. Three lab sessions per week for a period of 15 weeks is planned for this course. The maximum duration for the course is 90 hours which is scheduled as 45 sessions of 2 hours each. Based on the regulations of the institution, the students are eligible to earn 3 credits upon successful completion. This course aims to make students familiarize themselves in the core idea of fundamentals in problem solving. Python is chosen as our platform programming language to exercise the learned concepts in this course. As part of this course, students are expected to complete a small set of problems termed as practice problem set. This series of problem solving by students contribute 40% towards their final grading. Also, a series of assessments are planned for evaluating the students’ competency level obtained via this course. Usually, multiple choice questions, debugging questions and problems with coding will be the part of individual assessment. Every assessment will be evaluated for 10 marks for MCQs, 10 marks for Debugging and 30 marks for coding thus 50 marks in total. Apart from this practice problems and assessments, to test the students’ higher order thinking level, two challenging tasks will be given for the students each contributing to 10% in their final grading. Put together, the assessments contribute 40% and the practice problem set contribute 40% and the challenging tasks contribute 20% for the final grading. A. Stakeholders of this Course The course as stated uses the portal for several purposes. Various stake holders of this course are: • Core Management Team • Dean • Facilitator • Faculty members • Technical personnel for Portal • Students The Core Management Team includes the management authorities who initiated this process to introduce the change in delivery mode of course viz. online. Dean is the person who is responsible for the entire process as he monitors the entire Engineering, Technology & Applied Science Research Vol. 6, No. 5, 2016, 1207-1211 1209 www.etasr.com S. Vairamuthu and S. Margret Anouncia: Reconnoitering Students' Satisfaction of an Online Based… process and keeps things updated with the top management. One of the faculty members is designated as the facilitator who holds all the rights similar to an administrator. The facilitator may be assisted by another team of faculty and technical staff. Facilitator responsibilities include: Preparing and uploading questions for practice sessions and assessments, Scheduling/re- scheduling of assessments. Faculty members play the role of assessor of this course apart from the traditional teaching. The technical staff from the company assists the moderator in managing the users, preparing the system for practice problems and assessments like question uploading, time management, etc. Students use this portal for solving their practice problem sets and taking up the assessments online. They can idealize their own progress after individual assessments. Figure 1 depicts the interaction. Fig. 1. Use case diagram of the course Evaluation for the entire course was based on several rubrics which were decided by a team of experts. There are 6 rubrics finalized for grading the students. Figure 1 captures the various users of this portal related to the course apart from the traditional stake holders i.e. the management team. The course was offered in two different sessions with combination of many lab slots in morning and evening. In all sessions, there were a minimum of two persons responsible for the course delivery and monitoring. One of them must be the faculty member, who was in charge of the entire course assisted by a person who helped in monitoring. In some cases, the lab may contain two faculty members. V. RESEARCH MATERIALS AND METHOD As part of this study, many tasks were identified from the entire stake holder’s perspective. Students started using the portal for their daily activities. After successful implementation of this system, we decided to study the level of fulfillment of users and their expectations satisfied out of this course. A traditional questionnaire was formulated focusing on aspects of usability and users’ responses were gathered through a Google form. MUSA (Multi-criteria User Satisfaction Analysis) was employed for result interpretation. It was focused on two different perspectives: global satisfaction and partial satisfaction analysis, based on a 5-point scale. Major classification included: • System Quality • Information Quality • Technical Quality • Service Quality The questionnaire focused on the acquired competency level of individual students along with the facility to mark whether or not the course met their objective. Also, students were given provisions to express their suggestions to make improvements in the portal. As we concentrated in two different satisfactions measurement levels (global satisfaction and partial satisfaction), we finally gave the students a choice to rank the overall satisfaction level they experienced from this portal. Around 2000 students recorded their feedback on the usability of the portal. Many students didn’t want to reveal their identity. VI. RESULTS To interpret the results using the MUSA method, the segregation of data collected from the students were carried out as the first step. To facilitate this process, the collected 8 parameters were organized into 4 factors viz. system quality, information quality, technical quality and service quality as shown in Figure 2. Results are shown in Table I. Fig. 2. Usability Factors In order to find out the factors to improve by MUSA method, it was decided to follow the procedure given below: • Identify the desired level of satisfaction measure. • Determine how usability factors plays in deciding the overall satisfaction score. • Apply regression with four factors. • Interpret the values arrived in regression. Engineering, Technology & Applied Science Research Vol. 6, No. 5, 2016, 1207-1211 1210 www.etasr.com S. Vairamuthu and S. Margret Anouncia: Reconnoitering Students' Satisfaction of an Online Based… • Following were the desired procedure: • Desired level of satisfaction: 60% Usability factors and their individual contributions are shown in Table II. TABLE I. SUMMARY OF DATA COLLECTED Factors(Fi) and Attributes (Ai) Mean Standard Deviation F1. System Quality A1. Ease-of-use 3.40 1.11 A2. Efficiency 3.31 1.25 F2. Information Quality A1. Interaction 2.42 1.08 A2. Navigation 3.38 1.08 F3. Technical Quality A1. Memorability 2.94 0.96 A2. Learnability 1.63 0.73 F4. Service Quality A1. Response Time 3.86 1.03 A2. Users Satisfaction 3.24 1.02 TABLE II. USABILITY ATTRIBUTES Factors Attributes % measure A1. Ease-of-use 15 System Quality (30%) A2. Efficiency 15 A1. Interaction 14 Information Quality (20%) A2. Navigation 6 A1. Memorability 7.5 Technical Quality (30%) A2. Learnability 22.5 A1. Response Time 8 Service Quality (20%) A2. Users Satisfaction 12 Considering the above four factors, regression analysis was carried out as follows: Let Y be the Overall Quality (dependent variable) that would be determined by four independent variables X1, X2, X3 and X4. Modeling the relationship between the above mentioned dependent and independent variables, the results obtained were: One-on-one Regression [Y=MX+C], where Y refers Overall Quality and X refers Factors is given below: (0.508) (0.002)Total SQ= + (1) (0.544) ( 0.001)Total IQ= + - (2) (0.515) ( 0.001)Total TQ= + - (3) (0.503) (0.009)Total SeQ= + (4) where Total denotes Overall quality while SQ, IQ, TQ and SeQ represents individual quality factors viz., System Quality, Information Quality, Technical Quality and Service Quality respectively. Combining all the four independent variables SQ, IQ, TQ and SeQ and results of multiple regressions carried out and the result is given below in equation 6 which take the general form: 1 2 3 4Y C X SQ X IQ X TQ X SeQ= + + + + (5) where, Y is Overall Quality and X1 to X4 represents 4 factors viz System Quality, Information Quality, Technical Quality and Service Quality and C is constant. SeQTQIQSQY )002.0()009.0()008.0()002.0()52.0(  (6) To achieve the desired results of regression, the nature inspired computation PSO was used. PSO optimizes a problem using iterations to improve the measure of desired quality. It uses simple formulae to move particles around search space using the given formulae. Individual iteration influences the move from local best toward the best known position in search space thus by optimizing the measure. To obtain the desired level of satisfaction which we had fixed at 60%, by applying PSO, the System Quality at 0.07 with Information Quality at 0.24 whereasTechnical Quality was at 0.07 and the Service Quality took the value 0.24. Only for the above mentioned values for the chosen factors, it was found to be optimized using PSO. But observing the results from the data gathered, it was evident that both System quality and Technical quality needs improvement. The attributes Ease-of-use and Efficiency of System Quality along with Memorability and Learnability of Technical Quality were influencing the overall quality. VII. DISCUSSION Analyzing the individual components in partial satisfaction measure, student’s previous knowledge about computers contributed less in deciding the overall satisfaction level. If we take a closer look into this interpretation, the global satisfaction level of the portal is entirely different from the partial satisfaction level criteria. The factors contributed for the partial satisfaction level when measured individually differs drastically from the overall satisfaction measure. Several suggestions were recorded as their feedback to improve the usability of the portal that included improvements in user interface, competency levels of questions etc. Some of the major findings were as follows: • Almost 50% of the students that participated insisted that the interface needs more changes in UI. • The questions for assessments should consider the competency level of all the students that depends on memorability and learnability factors. • The infrastructure in terms of connectivity needs improvement for efficient utilization. • The time bound assessments can be changed so as to make portal easy to use at any time. The achieved usability was yet to be improved. The usability of the portal can be improved by incorporating Multimodal HCI features with the existing features of the portal. MMHCI features include speech synthesis, gesture recognition, eye tracking, etc. The fundamental aim of incorporating multi modal aspect into the existing system was to enhance the users’ experience. The satisfaction may be improved when the portal fulfill the users requirements. So, incorporating any of the multimodal concepts would lead towards improved usability. Possible modality may be integrating speech synthesizer would lead to reduce the number of key strokes for all the users. Minimal number of key strokes Engineering, Technology & Applied Science Research Vol. 6, No. 5, 2016, 1207-1211 1211 www.etasr.com S. Vairamuthu and S. Margret Anouncia: Reconnoitering Students' Satisfaction of an Online Based… may help users achieving more interactivity and improve speed during their interaction. VIII. CONCLUSION User interface design plays a vital role in usability of any online portal. The usability evaluation of an online assessment system was studied here in depth considering all factors that contribute in usability. The object of the study was a course offered in an academic institution. Faculty members and students were considered as end users. A questionnaire was filled and Multi-criteria User Satisfaction Analysis was employed for result interpretation. PSO was employed to identify the factors that may improve usability. References [1] W. H. Delone, E. R. McLean, “The DeLone and McLean model of information systems success: a ten-year update,” J. Manag. Inf. Syst., Vol. 19, No. 4, pp. 9–30, 2003 [2] A. Tatnall, Web portals: the new gateways to Internet information and services, IGI Global, 2005 [3] M. B. O. Ajoye, W. E. Nwagwu, “Information Systems User Satisfaction: A Survey of The Postgraduate School Portal, University of Ibadan, Nigeria,” Libr. Philos. Pract., p. 0_1, 2014 [4] B. Myers, J. Hollan, I. Cruz, S. Bryson, D. Bulterman, T. Catarci, W. Citrin, E. Glinert, J. Grudin, Y. Ioannidis, “Strategic directions in human-computer interaction,” ACM Comput. Surv., Vol. 28, No. 4, pp. 794–809, 1996 [5] C. Ardito, M. F. Costabile, M. De Marsico, R. Lanzilotti, S. Levialdi, T. Roselli, V. Rossano, “An approach to usability evaluation of e-learning applications”, Univers. Access Inf. Soc., Vol. 4, No. 3, pp. 270–283, 2006 [6] C. E. Downing, C. Liu, “Does web site usability correlate with web site usage?,” J. Int. Technol. Inf. Manag., Vol. 18, No. 3–4, pp. 443–455, 2009 [7] P. N. Sindhuja, S. G. Dastidar, “Impact of the factors influencing website usability on user satisfaction”, IUP J. Manag. Res., Vol. 8, No. 12, p. 54, 2009 [8] R. S. Pressman, Software engineering: a practitioner’s approach. Palgrave Macmillan, 2005 [9] M. Tarafdar, J. Zhang, “Analyzing the influence of web site design parameters on web site usability”, Inf. Resour. Manag. J., Vol. 18, No. 4, p. 62, 2005 [10] J. J. Cappel, Z. Huang, “A usability analysis of company websites,” J. Comput. Inf. Syst., Vol. 48, No. 1, pp. 117–123, 2007 [11] P. Géczy, N. Izumi, K. Hasida, “Foundations for effective portal service management,” Glob. J. Bus. Res., Vol. 5, No. 2, pp. 131–141, 2010 [12] J. M. Pearson, A. M. Pearson, “An exploratory study into determining the relative importance of key criteria in web usability: a multi-criteria approach”, J. Comput. Inf. Syst., Vol. 48, No. 4, pp. 115–127, 2008 [13] A. Granić, “Experience with usability evaluation of e-learning systems”, Univers. Access Inf. Soc., Vol. 7, No. 4, pp. 209–221, 2008 [14] S. H. Mustafa, L. F. Al-Zoua'bi, “Usability of the academic websites of Jordan’s universities: an evaluation study,” 9th International Arab Conference for Information Technology, pp. 31–40, 2008 [15] M. Manzoor, W. Hussain, “A web usability evaluation model for higher education providing Universities of Asia,” Sci., Tech. Dev., Vol. 31, No. 2, pp. 183–192, 2012 [16] S. A. Mentes, A. H. Turan, “Assessing the usability of university websites: an empirical study on Namik Kemal University,” TOJET Turkish Online J. Educ. Technol., Vol. 11, No. 3, 2012 [17] E. Grigoroudis, Y. Siskos, “Preference disaggregation for measuring and analysing customer satisfaction: The MUSA method,” Eur. J. Oper. Res., Vol. 143, No. 1, pp. 148–170, 2002 [18] L. A. Daher, I. Elkabani, “Usability evaluation of some Lebanese Universities Web Portals,” 13th international Arab conference on information technology ACIT, pp. 10–13, 2012 [19] B. Shneiderman, C. Plaisant, M. Cohen, S. Jacobs, N. Elmqvist, N. Diakopoulos, Designing the user interface, Pearson, USA, 2017 [20] M. Alvesson, Knowledge work and knowledge-intensive firms, Oxford University Press, 2004 [21] T. H. Davenport, Thinking for a living: how to get better performances and results from knowledge workers, Harvard Business Press, 2013 [22] W. Albert, T. Tullis, Measuring the user experience: collecting, analyzing, and presenting usability metrics, Newnes, 2013 [23] A. V Bosov, “Modeling and optimization of functioning of the information web portal,” Program. Comput. Softw., Vol. 35, No. 6, pp. 340–350, 2009 [24] A. Burton-Jones, M. J. Gallivan, “Toward a deeper understanding of system usage in organizations: a multilevel perspective”, MIS Quarterly, pp. 657–679, 2007 [25] T. H. Davenport, J. G. Harris, Competing on analytics: The new science of winning, Harvard Business Press, 2007 [26] G. Hamel, B. Breen, The future of management, Harvard Business Press, 2013 [27] S. Petter, W. DeLone, E. McLean, “Measuring information systems success: models, dimensions, measures, and interrelationships,” Eur. J. Inf. Syst., Vol. 17, No. 3, pp. 236–263, 2008 [28] U. R. Kulkarni, S. Ravindran, R. Freeze, “A knowledge management success model: Theoretical development and empirical validation”, J. Manag. Inf. Syst., Vol. 23, No. 3, pp. 309–347, 2006 [29] J. Wu, Y. Wang, “Measuring KMS success: A respecification of the DeLone and McLean’s model”, Inf. Manag., Vol. 43, no. 6, pp. 728– 739, 2006 [30] L. A. Halawi, R. V McCarthy, J. E. Aronson, “An empirical investigation of knowledge management systems’ success”, J. Comput. Inf. Syst., Vol. 48, No. 2, pp. 121–135, 2008