Research in Social Sciences and Technology (RESSAT) E-ISSN: 2468-6891 122 Evaluation on the Effectiveness and Usability of the Developed Collaborative Interaction Management System Salvacion Mata-Domingo1 Abstract A Collaborative Interaction Management System (CIMS) is a system that potentially provide some assistance to the teachers in managing the student’s collaborative interaction in forum discussions. The projected main function of the CIMS is to provide tools that automatically gauge the quality of interaction as reflected in student’s individual contributions so as to make it easier for the instructor to make judgments on the effectiveness of the collaborative process. This study aims to assess how effective and usable is the developed CIMS based on the evaluation of faculty and students. The descriptive research design is utilized in this study. The five-point Likert scale is used and the gathered data are statistically interpreted through the utilization of a weighted mean. The sample population of this study is the one hundred twenty participants coming from the different universities in the area where the survey was conducted. The researcher adopted a survey instrument made by some software specialists. The bases of the instrument are some of the characteristics listed on the software quality model of ISO 9126. These are the functionality, usability and the reliability. Each of these software characteristics is described by some indicators to obtain the responses of the participants. The overall evaluation of the participants displayed the weighted mean of 4.0, thus it can be concluded that the functionality, usability and reliability of the software is very good. Key words: effectiveness, usability, software quality, Collaborative Interaction Management System Introduction In the traditional lecture-based method of teaching and direct instruction, learning is primarily perceived as involving the simple transfer of fixed knowledge from the teacher to the students. In this paradigm, the teacher's main responsibility is to cover as much course material and deliver knowledge to their students. In such a setup, the minds of students are viewed as empty vessels in which knowledge can easily be poured into; as such, learning is simplistically presumed to have occurred when students have individually integrated and organized the knowledge they received 1 Lecturer - Gulf College, Mabela, Muscat, Sultanate of Oman, salvacion.sally@gulfcollege.edu.om mailto:salvacion.sally@gulfcollege.edu.om Mata-Domingo in their heads. While such methodology and perspective has been tried-and-tested, it nevertheless has numerous disadvantages. Over the years, perspectives on learning have evolved considerably. Educators nowadays, have increasingly realized the inadequacy of the lecture-based approach and have started to view learning as a process that is based more on student’s personal experience. In this new paradigm, learning is defined as being context-sensitive and focused more on self- discovery and exploration. Knowledge therefore is viewed as being created within a group process where interaction between individuals and the sharing of experiences in learning activities are the primary motivators. The teacher’s responsibility in this scenario is shifted towards getting students to interact with each other in a productive and supportive manner. Learning environments that emphasize the social nature of learning are considered as valuable tools in this new paradigm (Baytak, Tarman & Ayas, 2011; Tarman & Baytak, 2011; Tarman, 2012). Tools like forum discussion that pays particular attention to promoting and encouraging communicative cooperation and collaboration is a good example of this kind of environment. In fact, in the e-learning panacea, both in the academe and the industry, discussion forums are gaining tremendous popularity as a mechanism for increasing learner interaction. The utilization of discussion forum in academic setting is often problematic. The main reason is the enormous amount of time and effort that it imposes to implementing teacher-mediators in monitoring and guiding student interaction. Another factor is the high level of difficulty faced by the implementing teachers in evaluating the contributions of students in such online interactions. In the field of e-learning, similar problems are being addressed through the development of specialized environments such as the Learning Management System (LMS) and Content Management System (CMS). After thorough study the researcher was able designed a framework that was used to develop a management system that potentially provide the teachers some assistance in managing student’s collaborative interaction in forum discussions. This is the system that is referred to as a Collaborative Interaction Management System (CIMS). The task of the researcher does not end after the development of CIMS. There is a need to evaluate the effectiveness and usability of the developed CIMS; thus, this study was conducted. Research in Social Sciences and Technology (RESSAT) 2018: 3 (1), 122-133 Objectives and Rationale of the Study There are huge number of researches that explore about some areas of information and communication technology; however, it is believed that there are limited studies that explained its effectiveness and usefulness which would guide the developers on how to enhance their newly created system. The developed CIMS could reduce the time and tasks of the teachers when making decisions on the assessment’s performance of every student. Through the results of this study the teachers would be equipped with more knowledge and they will be able to extend more support to the students in managing their collaborative interactions. Moreover, it would also address the end users’ perspective (teachers and students) as regards to the issues of measurement on the quality characteristics of the developed software. In addition, the outcomes of this research would serve as a tool, to be utilized by the end users namely teachers and students as well as designers of instructional technology, which could expedite progressions in their communication and instructions activities. Review of Related Literature Nowadays, in the area of education technology is widely used. This includes the use of Powerpoint presentation, study reading materials, online books, and the like. Technology has become very important for any business whether it is a small medium enterprises or huge corporations. The meetings with employees around the world are now scheduled with the use of the Internet. Technology changes most of people’s lives (Tarman, 2009; Yucel et.al, 2010). The connection between families even in the different parts of the world are maintain through the use of cellphones and electronic mail. In other words, technology is used in various activities and in the different areas; thus, it is expected that improvements will continue over time and it will aid in making the works of average people a lot easier (“What role does technology”, 2017). Yet, in the paper of Bocar and Biong ( 2015 ) they recommended that students must be more careful in the utilization of on information and communication technology either it be within or outside the school so that Mata-Domingo the time and money could be spent wisely so that it will bear good results and goals for success will be achieved. ISO 9126 is an international standard for the evaluation of software. The ISO 9126 or other quality model transports clearness on the meaning of purpose and operating capability; however, the requirements, compliance and operating environment of software continuously changing; thus, the search to find valuable characteristics that facilitate measurement and control of the software production process continue (McCall, 1977). The important purpose of a product or service connotes functionality. The more functions the products or services have the more complicated it becomes. In the case of software there is specified list of functions. Usability refers to the easiness of use for a given function. It may also refer to the ability to learn on how to use a system. It is considered as a major sub-characteristic of usability. Specifically, reliability is define as the capability of the system to uphold the provision of its service under defined conditions for definite periods of time. It is a software characteristic which include fault tolerance. This means that the system has the ability to recover and continue its functioning after it experienced failure (McCall, 1977). According to the study of Bala and Chana (2012) “fault tolerance is a major concern to guarantee availability and reliability of critical services as well as application execution”. It was explained in their study that the impact of failure can be minimized through pre-emptive measures. Technical people must forecast failures and take proper actions before failures actually happen. This is the meaning of fault tolerance techniques. The researchers above mentioned proposed cloud virtualized system architecture. In the proposed system autonomic fault tolerance was implemented and it was found out that the proposed system can deal with various software faults for server applications in a cloud virtualized environment. According to Britain and Liber, (2003) since early 1980’s the academe have used email to communicate and the influence of technology continues as the growth and explosion of internet came very popular that leads academician to undertake in making their lecture notes and other supplemental reading materials on the web for the students to look through. It is common nowadays to some universities to have web-interfaces. Some institutions used conferencing software to create online group discussions. These are the innovations brought about by the modern technology. It is not surprising when IT researchers would be able to develop software to further Research in Social Sciences and Technology (RESSAT) 2018: 3 (1), 122-133 make the technology more useful in the educational setting. The developed software could be used as a teaching and learning tools. There are numbers of software quality characteristics; however, Chua and Dyson (2004) explained that the three software quality characteristics amongst others namely: functionality, reliability and usability are assessable, thus, this encourage the researcher to evaluate the effectiveness and usability of the developed CIMS. An instrument with specific indicators designed from the quality characteristics of the ISO 9126 software quality model by experts (Davis, 1989; Tullis and Stetson, 2004) was adopted to assess the developed CIMS. The Problem The implementation of the developed CIMS was not assumed as perfect; thus, assessment needs to be done. This study specifically focus on what is the level of the effectiveness and usability of the developed CIMS based on the evaluation of the (a) teachers, and (b) students? Scope and Limitations of the Study This study focuses on the evaluation of the software’s quality that is assessed by the Information Technology (IT) teachers and students from various Universities in Metro Manila using evaluation criteria provided in the ISO 9126 software quality assessment tool. The general objective of this study is to determine the observation of the user-participants on the developed CIMS. Method This section discussed the process on how this study was conducted. The descriptive research design is utilized in this study. The projected main function of the CIMS is to provide tools that automatically gauge the quality of interaction as reflected in student’s individual contributions, thus, it makes it easier for the instructor to make judgments on the effectiveness of the collaborative process. In order to fully investigate the CIMS as a system, which is appropriate for use by faculty and students in the subject area, the characteristics of the CIMS have to be evaluated. The quality characteristics items mentioned in ISO 9126 software quality model was the bases of the adopted evaluation instrument on the effectiveness and usability of the developed CIMS. These are the Mata-Domingo functionality, usability and reliability. Each of these software characteristics is described by some indicators, from the work of Davis, (1989); Tullis and Stetson, (2006) to obtain the responses of the participants. The sample population is randomly chosen which composed with the thirty teachers and thirty students from the selected universities in the National Capital Region area. One hundred twenty are the total number of participants of this study. Statistical Tools The five-point Likert scale was used and the gathered data are statistically interpreted through the utilisation of the weighted mean. For purposes of interpretation, analysis, and discussion the researcher constructed the scale of measurement as follows: Numeric Value Hypothetical Mean Range Qualitative Description Verbal Interpretation 1 1.00 - 1.80 Poor means that the functionality/ usability/ reliability is not at all effective / usable 2 1.81 – 2.60 Fair means that the functionality/ usability/ reliability is slightly effective / usable 3 2.61 – 3.40 Good means that the functionality/ usability/ reliability is effective / usable at majority of the time 4 3.41 – 4.20 Very Good means that the functionality/ usability/ reliability is effective / usable at most of the time 5 4.21- 5.00 Excellent means that the functionality/ usability/ reliability is always effective / usable Data Gathering Procedure Prior to the administration of the instrument the researcher ask permission from the heads of office of the four universities. For confidentiality and ethical reasons they are named as, A University, B University, C University and D University. After the approval of the request the questionnaire was distributed to the participants. The answered questionnaires were collected and the gathered data are tallied, interpreted, and analised. Results and Discussion Research in Social Sciences and Technology (RESSAT) 2018: 3 (1), 122-133 The table below reflects the results of the study. The participants exposed the level of the effectiveness and usability of the developed system. Functionality Based on the gathered data the participants revealed the functionality of the developed CIMS. It can be observed in the results as manifested by the participants that out of the six (6) description or indicators to measure the functionality of the developed CIMS four (4) of them were rated very good. Nevertheless, the two (2) of them were rated excellent. The results show that to some extent the functionality of the developed CIMS is effective at most of the time. Moreover, its usefulness would improve job performance and enhance effectiveness always. Table 1 Effectiveness and Usability of the Developed CIMS Legend: Hypothetical Mean Range Qualitative Description Verbal Interpretation Functionality Item Average (µ) Qualitative Description 1. Using the system in my job would enable me to accomplish tasks more quickly 4.20 Very Good 2. Using the system would improve my job performance 3. 90 Excellent 3. Using the system in my job would increase my productivity 4.60 Very Good 4. Using the system would enhance my effectiveness on the job 4.51 Excellent 5. Using the system would make it easier to do my job 4.41 Very Good 6. I would find the system useful in my job 3.50 Very Good Factor Average 4.19 Very Good Usability 1. I think that I would like to use this system frequently 4.56 Excellent 2. I found the system unnecessarily complex 3.72 Very Good 3. I thought the system was easy to use 4.21 Excellent 4. I think that I would need the support of a technical person to be able to use this system 3.42 Very Good 5. I found the system very cumbersome to use 3.85 Very Good 6. I felt very confident using the system 4.62 Excellent Factor Average 4.06 Very Good Reliability 1. Overall, I am satisfied with the ease of completing the tasks in this scenario 4.13 Very Good 2. There are not enough errors preventive messages in the system 2.70 Good 3. With the amount of time it took to complete the tasks in this scenario 4.37 Excellent 4. Overall, I am satisfied with the support information (online-line help, messages, documentation) when completing the tasks 3.78 Very Good Factor Average 3.75 Very Good Overall Average 4.00 Very Good Mata-Domingo 1.00 - 1.80 Poor means that the functionality/ usability/ reliability is not at all effective / usable 1.81 – 2.60 Fair means that the functionality/ usability/ reliability is slightly effective / usable 2.61 – 3.40 Good means that the functionality/ usability/ reliability is effective / usable at majority of the time 3.41 – 4.20 Very Good means that the functionality/ usability/ reliability is effective / usable at most of the time 4.21 - 5.00 Excellent means that the functionality/ usability/ reliability is always effective / usable Usability Moreover, with respect to the usability of the developed CIMS, the table above showed split results. As evaluated by the participants, they rated excellent on the three of the six (6) descriptions or indicators. They revealed that they would like to use this system frequently since it was easy to use, and they felt very confident using the system always. However, in the other three (3) descriptions or indicators the participants rated the effectiveness and usability of the developed CIMS at most of the time. This signifies that the developed CIMS is effective and usable but nonetheless the assistance of the technical person is necessary at some extent as it shows the lowest item average (µ=3.42) among other descriptions in this characteristic. This situation is reasonable since every individual has his or her own expertise. As per the definition of American Heritage Dictionary of the English Language (2016) a technical person in the scientific field means “one who is having or demonstrating special skill or practical knowledge relating to information technology”. In the present study some of the participants are students which is it presumed that their skills and practical knowledge are yet inadequate. Reliability In addition, with regard to the probability that the developed system does not experience many failures in the given time interval the participants demonstrated that they are satisfied on the effectiveness and usability of the developed CIMS as shown in the results on the table above. However, the participants established that errors preventive messages must be increased in the system to decrease the failures on the part of the users. The overall results displayed the rating of very good which means that the effectiveness and usability of the CIMS is at most of the time. The results of this present study strengthen the words of Bala and Chana (2012) when they stated in their study that to minimize failure, forecast on the future failures needs to be done and take the appropriate actions before failures actually happen. This signifies the application of fault tolerance techniques. Table 2. Summary of the Participants Evaluation on the Effectiveness and Usability of the Developed CIMS Legend: Software Characteristics Factor Average (µ) Qualitative Description Research in Social Sciences and Technology (RESSAT) 2018: 3 (1), 122-133 Hypothetical Mean Range Qualitative Description Verbal Interpretation 1.00 - 1.80 Poor means that the functionality/ usability/ reliability is not at all effective / usable 1.81 – 2.60 Fair means that the functionality/ usability/ reliability is slightly effective / usable 2.61 – 3.40 Good means that the functionality/ usability/ reliability is effective / usable at majority of the time 3.41 – 4.20 Very Good means that the functionality/ usability/ reliability is effective / usable at most of the time 4.21 - 5.00 Excellent means that the functionality/ usability/ reliability is always effective / usable In total, the Table 2 above encapsulated the participants’ evaluation on the developed CIMS. It is awe-inspiring that after rigorous effort of the researcher the users of the developed CIMS are pleased and rated the effectiveness and usability of the system at very good level even though there are areas that needs more valuable improvements. The rating that the participants established implies that they found the developed CIMS effective and/or usable at most of the time. Findings After the thought-provoking interpretation and analysis of the gathered data the researcher found that: (1) the effectiveness of the functionality characteristic of the developed CIMS is at most of the time; (2) there is a split result as to the usability characteristic of the developed CIMS; however, fifty percent of the indicators or descriptions are rated at excellent level; (3) participants demonstrated that they are satisfied on the effectiveness and usability of the developed CIMS as regards to its reliability though the system needs more additional preventive messages to reduce the failures of the users; (4) the overall results showed that the developed CIMS is effective and usable at most of the time. Discussion, Conclusion and Implications Based on the results, analyses and findings of this study it can be concluded that though the functionality, usability and reliability of the developed CIMS was evaluated by the participant- users at very good level which implies that at most of the time it is effective and usable. The evaluation is a good motivation to address the area which shows a little weaker point. These weak 1. Functionality: The ability to do something well or achieved a desired result from the forum. 4.19 Very Good 2. Usability: Measured in terms of the increase in productivity using the new system. 4.06 Very Good 3. Reliability: It refers to the probability that the system under consideration does not experience many failures in the given time interval. 3.75 Very Good Overall Average 4.00 Very Good Mata-Domingo points need further study and enhancement in order to make it more effective and usable. Specifically, to wit: in the usability characteristic of the developed CIMS the description which says that, the participants need the support of a technical person to be able to use this system must be given greater consideration since this signifies that though they rated the overall usability at very good level they still found some difficulty to some extent. Moreover, it can be noted that the reliability characteristic gathered the lowest factor average among the other two characteristics. Thus, more improvements or enhancement in this area must be done. Research in Social Sciences and Technology (RESSAT) 2018: 3 (1), 122-133 References American Heritage Dictionary of the English Language. (2016). Fifth Edition. Houghton Mifflin Harcourt Publishing Company. Retrieved September 2016 from http://www.thefreedictionary.com/technical Bala, A. and Chana, I. (2012). Fault Tolerance- Challenges, Techniques and Implementation in Cloud Computing. IJCSI International Journal of Computer Science, 9(1). ISSN (Online): 1694-0814. www.IJCSI.org. Baytak, A., Tarman, B., & Ayas, C. (2011). Experiencing technology integration in education: Children's perceptions. International Electronic Journal of Elementary Education, 3(2), 139-151. Bocar, A C and Biong, C.T. (2015). Role of information and communication technology: Its impact on S\students’ learning and the extent of effects to social, recreational and sports activities. American Journal of Information Science and Computer Engineering, 1(2), pp. 59-67. Retrieved July 2016 from http://www.aiscience.org/journal/ajisce Britain, S. & Liber, O. (2003). A framework for pedagogical evaluation of virtual learning environments. Retrieved June 2015 from http://www.leeds.ac.uk/educol/documents/ 00001237.htm Chua, B. B. & Dyson, L. E. (2004). Applying the ISO 9126 model to the evaluation of an e-learning system. ASCLITE Conference Proceedings. Retrieved May 2015 from https://www. ascilite.org/conferences/perth04/procs/chua.html Davis, F.D. (1989). Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology. Management Information Systems Research Center, 13(3), pp. 319-340. DOI: 10.2307/249008. Retrieved August 2015 from http://www. jstor.org/stable/249008 Mabrito, M. (2006). “A study of synchronous versus asynchronous collaboration in an online business writing class”. The American Journal of Distance Education, 20(2), 93-107 McCall, J. (1977) ISO 9126 Software Quality Characteristics. Retrieved March 2015 from http://www.sqa.net/iso9126.html Murphy, E. & Jerome, T. (2005) Assessing students' contributions to online asynchronous discussions in university-level courses. Electronic Journal of Instructional Science and http://www.thefreedictionary.com/technical http://www.aiscience.org/journal/ajisce Mata-Domingo Technology, 8. Retrieved September 1, 2014, from http://www.usq.edu.au/electpub/e- jist/docs/vol8_no1/commentary/ stu_contrib_ansynch.htm Tarman, B. (2009). The Digital Divide in Education, ERIC (ED508213). Tarman, B., Baytak, A. (2011). The New Role of Technology in Education: Social Studies Teacher Candidates’ Perceptions, Gaziantep University Journal of Social Sciences, 10 (2),891-908. Tarman, B. (2012). Effective leadership in culturally diverse schools. Energy Education Science and Technology Part B: Social and Educational Studies, 4(2), 1103-1114. Tullis, T. and Stetson, J.N. (2006). A Comparison of Questionnaires for Assessing Website Usability. Retrieved September 2015 from https://www.researchgate.net/publication/ 228609327_A_Comparison_of_Questionnaires_for_Assessing_Website_Usability Valenti, S., Cucchiarelli, A. & Panti, M. (2002). Computer Based Assessment Systems Evaluation via the ISO9126 Quality Model. Journal of Information Technology Education, 1(3), 157- 175. What role does technology play in our lives? (2017). IAC Publishing, Retrieved July 2017. LLC. From https://www.reference.com/world-view/role-technology-play-livesa6e4ca6f4c2593 ca?qo=contentSimilarQuestions# Yucel, C., Acun, I., Tarman, B., & Mete, T. (2010). A model to explore Turkish teachers' ICT integration stages. Turkish Online Journal of Educational Technology, 9(4), 1-9. https://www.reference.com/world-view/role-technology-play-livesa6e4ca6f4c2593%20ca?qo=contentSimilarQuestions https://www.reference.com/world-view/role-technology-play-livesa6e4ca6f4c2593%20ca?qo=contentSimilarQuestions