Hamutoğlu, N. B., Güngören, Ö. C., Duman, İ., Horzum, M. B., Kıyıcı, M., & Akgün, Ö. E. (2019). The scales for acceptance, usability, and satisfaction for web-based monitoring and support systems. International Online Journal of Education and Teaching (IOJET), 6(3). 563-580 http://iojet.org/index.php/IOJET/article/view/4 04 Received: 19.03.2019 Received in revised form: 15.06.2019 Accepted: 29.06.2019 THE SCALES FOR ACCEPTANCE, USABILITY, AND SATISFACTION FOR WEB- BASED MONITORING AND SUPPORT SYSTEMS Research Article Nazire Burçin Hamutoğlu Kırşehir Ahi Evran University nazire.hamutoglu@ahievran.edu.tr Özlem Canan Güngören Sakarya University ocanan@sakarya.edu.tr İbrahim Duman Hakkari University ibrahimduman@hakkari.edu.tr Mehmet Barış Horzum Sakarya University mhorzum@sakarya.edu.tr Mubin Kıyıcı Sakarya University mkiyici@sakarya.edu.tr http://iojet.org/index.php/IOJET/article/view/404 http://iojet.org/index.php/IOJET/article/view/404 mailto:nazire.hamutoglu@ahievran.edu.tr mailto:ocanan@sakarya.edu.tr mailto:ibrahimduman@hakkari.edu.tr mailto:mhorzum@sakarya.edu.tr mailto:mkiyici@sakarya.edu.tr https://orcid.org/0000-0003-0941-9070 https://orcid.org/0000-0002-9184-6110 https://orcid.org/0000-0002-4662-5503 https://orcid.org/0000-0003-3567-0779 https://orcid.org/0000-0001-9458-7831 Özcan Erkan Akgün Medeniyet University ozcan.akgun@medeniyet.edu.tr Nazire Burçin Hamutoğlu is a PhD at Kırşehir Ahi Evran University, Faculty of Education, The Department of Computer and Instructional Technologies. Ozlem Canan GUNGOREN is an Assistant Professor at Sakarya University, Faculty of Education, the Department of Computer and Instructional Technologies. Ibrahim DUMAN is a Research Assistant at Hakkari University and a PhD student at Sakarya University, Faculty of Education, the Department of Computer and Instructional Technologies. Mehmet Baris HORZUM is an Associate Professor and Vice-Dean at Sakarya University, Faculty of Education, the Department of Computer and Instructional Technologies. Mubin KIYICI is an Associate Professor and Head of the department of Computer and Instructional Technologies at Sakarya University, Faculty of Education. Ozcan Erkan AKGUN is an Associate Professor and Vice-Dean at Medeniyet University, Faculty of Education, the Department of Computer and Instructional Technologies. Copyright by Informascope. Material published and so copyrighted may not be published elsewhere without the written permission of IOJET. mailto:ozcan.akgun@medeniyet.edu.tr https://orcid.org/0000-0002-6486-0486 International Online Journal of Education and Teaching (IOJET) 2019, 6(3), 563-580 563 THE SCALES FOR ACCEPTANCE, USABILITY, AND SATISFACTION FOR WEB-BASED MONITORING AND SUPPORT SYSTEMS Nazire Burçin Hamutoğlu nazire.hamutoglu@ahievran.edu.tr Özlem Canan Güngören ocanan@sakarya.edu.tr İbrahim Duman ibrahimduman@hakkari.edu.tr Mehmet Barış Horzum mhorzum@sakarya.edu.tr Mubin Kıyıcı mkiyici@sakarya.edu.tr Özcan Erkan Akgün ozcan.akgun@medeniyet.edu.tr Abstract The purpose of this study is to develop three scales to assess some features of a web-based monitoring and support system for internship processes of trainees which was named Monitoring and Support System (SIDES). These scales are: “SIDES Satisfaction Scale”, “SIDES Acceptance Scale”, and “SIDES Usability Scale”. We developed these scales as an assessment model within the scope of a project. In development phases, the scopes and the names of the scales were formed by results of the literature review. Satisfaction, acceptance and usability scales have item pools with 6, 17 and 27 questions respectively. The scales were submitted to 10 field experts for taking opinions for validity. After taking expert opinions, the scales were applied to 56 senior students who used the SIDES in an experimental condition in an academic term within the scope of the teaching practice classes in Sakarya University Computer and Instructional Technologies Teaching Bachelor Program. To examine the construct validity of the scales, the Exploratory Factor Analyses was used. Cronbach's Alpha internal consistency coefficients were calculated for internal consistency reliabilities. As a result of the analyses, one-factor, 6-item “SIDES Satisfaction Scale”; 3-factor, 14-item “SIDES Acceptance Scale” and 3-factor, 19-item “SIDES Usability Scale” were developed. These scales may be used or considered as examples for examining the effectiveness of web- based monitoring and support systems. Keywords: web-based monitoring and support system, TAM, scale; satisfaction; acceptance, usability, trainees, teacher training. mailto:nazire.hamutoglu@ahievran.edu.tr mailto:ocanan@sakarya.edu.tr mailto:ibrahimduman@hakkari.edu.tr mailto:mhorzum@sakarya.edu.tr mailto:mkiyici@sakarya.edu.tr mailto:ozcan.akgun@medeniyet.edu.tr Hamutoğlu, Güngören, Duman, Horzum, Kıyıcı & Akgün 564 1. Introduction Teacher training policies are constantly improving across the globe. According to reports published by unions such as American Federation of Teachers (2008) and European Commission (2013) underlined the importance of teaching profession and teacher education. According to Organization for Economic Cooperation and Development (OECD), many countries agree that teachers have a crucial role on students’ success. Therefore, teacher education policy is rising rapidly towards the top of the educational agenda (OECD, 2005). Şişman (2003) states that a qualified internship process is needed for training qualified teachers. Although teacher training policies vary, there are similarities regarding internship application processes throughout the world. In Germany, for example, students upon their graduation become teacher trainees for two-years (Terhart, 2003). In Japan, similarly, having teaching certificate is not sufficient to become a teacher. Upon their graduation, students go through an internship process. In order to become a teacher in Finland, it is necessary to be successful in a test known as "matriculation" similar to the one used in Turkey and succeed in the three-stage acceptance test (book examination, interview, and sample lecture) (Malaty, 2006). In the USA, although differences exist in teacher training policies among states, it is essential to complete a four-year higher education program in order to be employed as a teacher at any level (Abazoğlu, 2014). In order to be a teacher in Turkey, it is necessary to sit and succeed in a high stakes national test, upon completion of a four-year undergraduate program in education. If they succeed, only then prospective teachers start working as teachers in public schools. Teaching practice process is considered as an important action taken for professional development above and beyond supporting new knowledge and skills for teachers (Vescio, Ross, & Adams, 2008), and it is one of the important pre-service experiences that teacher candidates have. Teacher candidates can exhibit the ability to use theoretical information they acquire in this module (Özkılıç, Bilgin, & Kartal, 2008). However, higher education institutions have different applications concerning the Teaching Practice module. Moreover, practice teachers, teacher candidates and instructors experience managerial, pedagogical and professional drawbacks during the practice process (Aydın & Akgun, 2014; Aytaçlı, 2012; Cansaran, Idil, & Kalkan, 2006; Gömleksiz, Mercin, Bulut, & Atan, 2006; Koç & Yıldız; 2012; Rıza & Hamurcu, 2000; Sarıçoban, 2008; Seçer, Çeliköz, & Kayılı, 2010). According to Paker (2008), sufficient feedback cannot be provided for students, appropriate guidance is not in place and the course is not well-planned in the teaching practice module. Besides, teacher candidates cannot receive adequate guidance about material choice, class management, course planning, and instruction model during the teaching practice process (Kiraz, 2002). Gökçe and Demirhan (2005) highlight the lack of communication within the triangle of teacher candidate, practice teacher and instructor. Kırksekiz, Uysal, Isbulan, Akgun, Kıyıcı, and Horzum (2015) categorize the problems teaching practice with respect to teacher candidate, practice teacher, instructor and the general process. In addition to emphasizing serious problems in communication experienced by all stakeholders, teacher candidates are not sufficiently assisted by instructors, there is inadequate guidance by instructors and practice teacher, and they cannot spare enough time for each student in the activities. Different applications within the context of universities in Turkey also show that there are no standards for the teaching practice module (Akgun, Kıyıcı, Horzum, Hamutoğlu, Güngören, & Duman, 2015a). McGee (2019) underlines the importance of mentorship in the student teaching internship experience that helps them to improve their practice, and not only teaching experience, but also supporting, guiding, and giving quality feedback to them. Accordingly, considering qualified teachers in teaching practice course, it is important for International Online Journal of Education and Teaching (IOJET) 2019, 6(3), 563-580 565 institutions to surround their internship applications via technology. With technology support, the process of the course might have been more effective and productive. There are many studies on the use of technology in teacher education (Fisher, 1996; Jung, 2005; Hixon & So, 2009; Sourdot, Smith, Anderson, & Whitworth, 2017). However, the fact is that technology has a supportive and/or spectator role in teacher education, and fills or diminish supervisory, psychological, pedagogical, and communicative gap in the literature. It is also thought that technology plays an important role in eliminating the drawbacks that occur during the teaching practices lesson. Considering the gap in the literature, measuring the views towards the “Internship Monitoring and Support System (SIDES)” within the scope of developed instruments are important. SIDES web application developed under cooperation of faculty-institution is thought to be providing teacher candidates with interactive assistance during their internship, and creating a standard may diminish the communicative gap between faculty-institution by supporting supervisory and participatory pedagogical support for the attainments of the teaching practice module. 1.1. SIDES SIDES is a project being conducted under TUBITAK-1001 program. The starting point of this project is that the lack of standard attainments on the national basis within the scope of teaching practice lesson causes several drawbacks experienced by teacher candidates at their practice schools (Aydin, & Akgun, 2014). The attainments of the teaching practice lesson were created with a comprehensive literature review (Akgun, Gökmen, Özer, Kaymak, Horzum, & Kıyıcı, 2015b) and the SIDES web application was developed so that teacher candidates can put these attainments into practice and be assisted. It is aimed by this application that the teaching practice process is completed effectively and in a planned way within the cooperation of teacher candidates, practice teachers and instructors. Dallmer (2004) stated that cooperation is an important factor that affects achievement during teacher training. Teacher candidates will be able to be informed of where their internship schools are; their practice teachers' details and make contact with all people related to the internship over the system. It is therefore thought that the communication-oriented problems experienced by teacher candidates in the beginning of the term could be solved to some extent by the cooperation of teacher candidate, practice teacher and instructor. SIDES enables teacher candidates to upload the papers which they are responsible for preparing for the teaching practice lesson onto the web environment and receive related feedbacks. By this means, instructors can assess the papers without the limitation of space and time and provide teacher candidates with the necessary feedback. Feedback is a vital element that will enhance teacher candidate's professional skills during the process (YOK, 1998). With SIDES developed within the framework of faculty-institution cooperation, it is planned to prevent problems such as the fact that instructors cannot spare enough time for assessing teacher candidates' activities, the communication between students, instructors and practice teachers, insufficient assistance, late feedback and guidance problems. It is thought that SIDES facilitates the internship process under the teaching practice lesson by the support services it provides the teacher candidates, the practice teacher at the school and the instructor at the university. The usability of support services websites affects the confidence, satisfaction and loyalty of users (Flavián, Guinalíu, & Gurrea, 2006). Users acceptance on the use of system affects their beliefs and attitudes to the system. This may Hamutoğlu, Güngören, Duman, Horzum, Kıyıcı & Akgün 566 effect the quality of system (Wixom & Todd, 2005). Ease of use is an important factor affecting the behavior of the system for users and system acceptance (Venkatesh & Davis, 2000). To reveal the effectiveness of SIDES within the scope of this study, “SIDES General Satisfaction Scale (SGSS)”, “SIDES Acceptance Scale (SACS)”, and “SIDES Usability Scale (SUSS)” were developed. The SIDES General Satisfaction Scale developed is deemed important for determining the service quality of SIDES. On the other hand, it is important whether the newly-developed SIDES was accepted by users as an effective, efficient and useful system. Perceived ease of use, perceived usefulness, attitude towards using and behavioral intention to use structures in the Technology Acceptance Model are important factors in determining teacher candidates' levels of accepting SIDES. In addition, it is also aimed by the research that SIDES is used relevantly and effectively. Hence, usability is an important concept that should be taken into consideration for the efficient use of websites and users' satisfaction (Çakmak, Güneş, & Tahsin, 2011; Nielsen, 2000; Pearrow, 2000; Shneiderman, 2004). It is thought that the SIDES Usability Scale will provide significant findings on the usability in terms of the access and the interface of the system. In this respect, it is believed that measuring students’ acceptance on use of such systems will help to identify the factors that students influenced internally e.g. attitude, behavioral intention, etc. In addition to this, usability of the systems also may impact the students’ future usage behaviors, externally. These provisions are claimed by Ertmer, Ottenbreit-Leftwich, Sadik, Sendurur, & Sendurur, (2012) and Sánchez-Prieto, Hernández-García, García-Peñalvo, Chaparro-Peláez, & Olmos-Migueláñez (2019) that internal and external factors have effects on attitudes towards using systems and behavioral intention to use. Moreover, satisfaction is also related to internal constructs, and measuring this construct would emerge some hints for system developers, trainers, administrators, teachers, and teacher educators. Scales developed for measuring those variables all have different scope and structures, but they can be used together as parts of a measurement model for considering obtained results with a holistic perspective about students, teachers, system developers, and administrators. Based on this, measuring acceptance of students toward systems could provide a contribution to teachers and system developers, and identifying usability of systems may give a chance to improve interface of the systems as well. Finally, examining satisfaction of students will be a lens for administrators and teachers, in order to see the weakness and strengths of the teacher training system. Therefore, it is important since these developed scales will shed light on the effectiveness of a product to be created as a result of similar studies. It is also aimed by this research to examine the validity and reliability of the results obtained from the pilot schemes applied to the teacher candidates. 2. Methods 2.1. Research design This research is composed of three separate scale development studies in which necessary data collection tools were prepared to examine the effectiveness of SIDES. The purpose and the method of the study is scale development. In order to serve this aim, a comprehensive literature search was made, and the theoretical framework of the scale was determined. After that, an item pool was created, and revised by an expert group. An initial form was created and applied to participants. The results of this study reached by analyzing the data collected from these participants. International Online Journal of Education and Teaching (IOJET) 2019, 6(3), 563-580 567 2.2. Participants The participants of this research were 56 senior students who study in Sakarya University, Computer and Instructional Technologies Teacher Education Program and also, they took a necessary place in experimental phases in SIDES. The number of participants was naturally limited to 56 because these participants were subjects of the experiment and their number was 56. 2.3. Measurement Tools To assess the effectiveness of SIDES, "SIDES General Satisfaction Scale (SGSS)", "SIDES Acceptance Scale (SACS)", and "SIDES Usability Scale (SUSS)" were developed. SGSS is a 6-item scale composed of one factor. SACS is composed of 14 items and three factors (usage attitude-intention, perceived ease of use, perceived usefulness. SUSS has a structure of 19 items and three factors (Usability of SIDES, [Unproblematic] Access to SIDES, Face of SIDES). While developing the scales, an item pool was created after a literature review with appropriate items that could be included in the scales. The participation level for the items in the item pool for each scale was chosen to be 5-Likert type scoring and the scoring was determined to be (5) Strongly Agree, (4) Agree, (3) Neutral, (2) Disagree, (1) Strongly Disagree; and next, the validity-reliability studies were conducted. In addition, it was aimed to measure the service quality of SIDES with the item “SIDES provides the assistance I need for the internship practices.” The participants were asked to answer this item by choosing one of the scores from (1) Strongly Disagree to (10) Strongly Agree. 2.3.1. Procedures In the purpose of developing scale, firstly literature was comprehensively searched, and studies with similar purpose and background were determined. Through this, an item pool (53 items for SUSS, 6 items for SGSS, 17 items for SACS) was created. We took expert views for content validity of these scales. 10 experts were consulted for the content validity and necessary adjustments were made to finalize the scales consequently. Experts are involving 5 field experts, 2 assessment and evaluation expert, 2 language expert and 1 Psychological Counselling and Guidance expert, and in light of their opinions some revisions (26 of items omitted from the SUSS, 3 of items omitted from the SACS) were made, and content validity was provided. Hence, the scale was administered for the pilot study. A similar group to target consisting of 8 students were invited for a focus group interview in terms of identifying whether items are clear and comprehensible for the target group. These procedures were followed by studies of construct and structure validity. Exploratory factor analysis (EFA) was performed to reveal the structure validity of the scale. It is aimed with the EFA to come up with the significant structure explained by multiple interrelated items which are defined by the scale (Buyukozturk, 2011). The reliability of the scale was examined by internal consistency coefficients, Cronbach Alpha. 2.4. Data analysis For analyzing the data, Statistical Package for the Social Sciences-SPSS was used for EFA and correlations. Confirmatory factor analysis (CFA), and divergent and convergent validity analysis were also performed to test whether the scales have construct validity. The results of this analysis have an accordance with EFA results. But the CFA and related results were not given in this study because of the data collected from the same participants whom their data also used for EFA analysis. According to the results, it can be said that the scales factor structures were approved by CFA and have a valid structure. Hamutoğlu, Güngören, Duman, Horzum, Kıyıcı & Akgün 568 3. Findings The findings of scales were presented separately as general satisfaction, acceptance and usability in the paper. The findings of the scales were listed as preparing the items, content validity, construct validity and reliability successively. The prerequisites for the analysis such as normality, extreme values, multicollinearity and singularity assumptions were controlled to meet the assumptions. VIF and tolerance values and Cook’s distance and Leverage values were computed as well. Accordingly, the data showed normal distribution (p>.05), and skewness - kurtosis values are ranging between -2.5 and +2.5 (Mertler & Vannatta, 2005; Tabachnick & Fidell, 2007). Extreme values were investigated via Mahalanobis Distance in the dataset (p<0.01) (Büyüköztürk, 2011; Hair, Black, Babin, Anderson & Tahtam, 2006), VIF and tolerance values were found smaller than 10 and higher than zero, respectively. Addition to this, in the results while Cook’s distance value was found smaller than 1, Leverage value was smaller than 0.02. Finally, multicollinearity and singularity values were tested and the result showed that they were at a moderate level (see table 6) (Akbulut, 210, p.158). According to findings, the dataset was meet the assumptions and therefore it is possible to perform construct validity analysis. 3.1. SIDES General Satisfaction Scale (SGSS) 3.1.1. Preparing the Items The SIDES General Satisfaction Scale (SGSS) was developed to reveal the general satisfaction levels of teacher candidates concerning the “Internship Monitoring and Support System (SIDES)”. After the literature review, the general satisfaction scale was determined to be 6 items by benefiting from the literature (Aladwani & Palvia, 2002; Parasuraman, Zeithaml & Malhotra, 2005; Şimşek, 1998; Yang, Chai, Zhou & Zhou, 2005). 3.1.1.1. Content Validity 10 field experts were consulted for the item pool in terms of scale's content validity. Necessary adjustments were made in accordance with the expert opinions and the trial form of SGSS which is composed of 6 items was obtained for the pilot scheme. 3.1.1.2. Construct Validity In the validity studies of the SIDES General Satisfaction Scale, the exploratory factor analysis (EFA) was applied to look at the construct validity. It was aimed that the SGSS created with the exploratory factor analysis (EFA) would reveal the factor structure over the teacher candidates that formed the study group. With Kaiser Meyer Olkin (KMO) and Barlett's Sphericity test, the sampling adequacy was tested to determine sample's conformity to the factory analysis. The KMO sample adequacy coefficient which was found to be .86 and Barlett's Sphericity test's value which was found to be (χ2=216.837, N=56, p=0.00) that the scale is suitable for factor analysis (Buyukozturk, 2011). 3.1.1.2.1. Exploratory Factor Analysis It was shown with the exploratory factor analysis that SGSS has a one-factor structure composed of 6 items. SGSS having a one-factor structure with an eigenvalue higher than 1 explains 68.007% of total variance. While the items were included in the scale it should be noted that the eigenvalues of the factors constituting were 1 and above, and the factor loadings were 0.30 and above, and between the factor loadings of the items there is at least a 0.10 difference (Buyukozturk, 2011). International Online Journal of Education and Teaching (IOJET) 2019, 6(3), 563-580 569 The load values of items in the scale vary between .72 and .90. The shared factor variance and the factor load values obtained from the exploratory factor analysis are reported in Table 1. Table 1. Exploratory Factor Analysis Results of SIDES General Satisfaction Scale Item Common Factor Variances (h2) Extracted Factor Loadings s3 .82 .90 s4 .76 .87 s2 .74 .86 s5 .64 .80 s6 .58 .76 s1 .52 .72 Eigenvalue % (Total = 4.08) Explained Variance % (Total = 68.01) 3.1.1.3. Reliability The data obtained from 56 teacher candidates with the SGSS trial form in the pilot scheme was applied with validity and reliability studies. Cronbach's Alpha internal consistency coefficients were used to determine the reliability of the scale. The internal consistency coefficient value was found to be .90 for the whole scale. All internal consistency values being above .70 (Field, 2013) shows that scale's reliability values are high; in other words, it produces consistent data. 3.2. SIDES Acceptance Scale (SACS) 3.2.1. Preparing the Items The theoretical basis of the SIDES Acceptance Scale (SACS) was formed with a comprehensive literature review. Examining the studies by Davis, Bagozzi, and Warshaw (1989) based on TAM (Davis et al., 1989; Venkatesh & Davis, 2000; Venkatesh, Davis, & Davis, 2003; Lee, Kozar & Larsen, 2003; Canan-Güngören, Bektaş, Öztürk, & Horzum, 2014), an item pool of 17 items was created. 3.2.1.1. Content Validity 10 field experts were consulted for the item pool in terms of scale's scope validity. Necessary adjustments were made in accordance with the expert opinions and the trial form of SACS which is composed of 14 items was obtained for the pilot scheme. 3.2.1.2. Construct Validity Exploratory factor analysis (EFA) was performed to reveal the construct validity in the validity studies of SACS. It was aimed that the SACS created with the exploratory factor analysis (EFA) would reveal the factor structure over the teacher candidates that formed the study group. It is necessary to test the sample adequacy to determine scale's conformity to the factor analysis. Kaiser Meyer Olkin (KMO) and Barlett's Sphericity test analyses were carried out to this end. The KMO sample adequacy coefficient which was found to be .89 and Barlett's Sphericity test's χ2 value which was found to be (χ2=1000.287, N=56, p=0.00) that the scale is suitable for factor analysis (Buyukozturk, 2011). 3.2.1.2.1. Exploratory Factor Analysis The EFA results show that SACS has a three-factor structure (Buyukozturk, 2011). Accordingly, the scale has the factors of ease of use, perceived usefulness and usage attitude- intention and is composed of 14 items. Items 1, 2, 3, and 4 in the scale serve to the perceived ease of use. One of the items that serves to this factor is “It is quite easy for me to use Hamutoğlu, Güngören, Duman, Horzum, Kıyıcı & Akgün 570 SIDES.” Another factor in the scale is the perceived usefulness. Items 5, 6, and 7 are studied under the perceived usefulness factor in the scale. One of the scale items that serves to this factor is “Using SIDES has increased my effectiveness in the lesson.” The items that serve to the usage attitude and intention are items 8, 9, 10, 11, 12, 13, and 14; the item “I would like SIDES to be used in the coming terms too” is an example of the items under the before mentioned factor. SGSS having a 3-factor structure with an eigenvalue higher than 1 explains 88.36% of total variance. The load values of items in the scale vary between .70 and .92. The shared factor variance and the factor load values obtained from the exploratory factor analysis are reported in Table 2. Table 2. SACS Exploratory Factor Analysis Results Item Common Factor Variances (h2) Extracted Factor Loadings *UAI **PEU ***PB M2 .93 .93 M1 .90 .92 M3 .89 .89 M4 .89 .84 M6 .92 .91 M5 .86 .89 M7 .84 .78 M13 .87 .88 M12 .92 .85 M14 .86 .83 M9 .79 .80 M11 .89 .77 M8 .84 .74 M10 .90 .70 Eigenvalue % (Total=12.35 ) 9.17 213 1.05 Explained Variance % (Total=88.36 ) 65.56 15.26 7.53 *Usage Attitude and Intention (UAI), **Perceived Ease of Use (PEU), *** Perceived Usefulness (PU) Value lower than .30 were not shown 3.2.1.3. Reliability The data obtained from 56 teacher candidates with the SACS trial form in the pilot scheme was applied with validity and reliability studies. Cronbach's Alpha internal consistency coefficients were used to determine the reliability of the scale. The internal consistency coefficient for the three-factor structure of the scale was found for ease of use, perceived usefulness, attitude towards using and behavioral intention to use respectively: .96, .92, and .97. All internal consistency values being above .70 shows that scale's reliability values are high; in other words, it produces consistent data. International Online Journal of Education and Teaching (IOJET) 2019, 6(3), 563-580 571 3.3. SIDES Usability Scale (SUSS) 3.3.1. Preparing the Items The literature was reviewed for the SIDES Usability Scale (SUSS) and related sources (Aladwani & Palvia, 2002; Green & Pearson, 2006; Parasuraman, Zeithaml & Malhotra, 2005; Wang, Wang & Shee, 2007; Yang, Cai, Zhou & Zhou, 2005) were utilized and the scale items were created. A scale form with 27 items was obtained. 3.3.1.1. Content Validity 10 field experts were consulted for the item pool in terms of scale's scope validity. Necessary adjustments were made in accordance with the expert opinions and the trial form of IUSS which is composed of 27 items was obtained for the pilot scheme. 3.3.1.2. Construct Validity Explanatory factor analysis (EFA) was performed to examine the construct validity in the validity studies of IUSS. It was aimed that the IUSS created with the exploratory factor analysis (EFA) would reveal the factor structure over the teacher candidates that formed the study group. Kaiser Meyer Olkin (KMO) and Barlett's Sphericity test analyses were used to determine scale's conformity to the factor analysis and test the sample adequacy. The KMO sample adequacy coefficient which was found to be .89 and Barlett's Sphericity test's χ2 value which was found to be (χ2=1092.990, N=56, p=0.00) that the scale is suitable for factor analysis (Buyukozturk, 2011). 3.3.1.2.1. Exploratory Factor Analysis 8 items were excluded from IUSS, which was composed of 27 items in the beginning, and the scale was decreased to 19 items. The EFA results show that IUSS has a three-factor structure (Buyukozturk, 2011). Accordingly, scale's structure includes three factors— usefulness (U), unproblematic access (UA) and face (F)—and is composed of 19 items. Items 1, 6, 7, 8, 10, 11, 12, 13, 14, 15, 16, and 17 serve to the usefulness factor. Another factor in the scale is unproblematic access. Items 2, 3, 4, 9, and 20 are studied under the unproblematic access factor in the scale. Items 26 and 27 serve to the face factor of the scale (Kline, 1998). IUSS having a 3-factor structure with an eigenvalue higher than 1 explains 74.78% of total variance. The load values of items in the scale vary between .60 and .85. The shared factor variance and the factor load values obtained from the exploratory factor analysis are given in Table 3. Table 3. IUSS Exploratory Factor Analysis Results Item Common Factor Variances (h2) Extracted Factor Loadings *U **UA ***F U1 .82 .85 U15 .86 .79 U14 .81 .78 U11 .79 .78 U16 .78 .75 U17 .52 .75 U8 .81 .75 Hamutoğlu, Güngören, Duman, Horzum, Kıyıcı & Akgün 572 U10 .67 .73 U13 .71 .70 U12 .73 .68 U7 .66 .68 U6 .72 .64 U3 .67 .78 U4 .71 .77 U2 .61 .73 U20 .89 .71 U9 .65 .63 U27 .86 .85 U26 .88 .83 Eigenvalue % (Total=14.19 ) 11.25 1.92 1.02 Explained Variance % (Total=88.36 ) 59.25 10.12 5.40 *Usefulness (U), ** [Unproblematic] Access (UA), *** Face (F) 3.3.1.3. Reliability The data obtained from 56 teacher candidates with the IUSS trial form in the pilot scheme was applied with validity and reliability studies. Cronbach's Alpha internal consistency coefficients were used to determine the reliability of the scale. The internal consistency coefficient for the three-factor structure of the scale was found for U, UA, and F respectively: .96, .85, and .94. All internal consistency values being above .70 show that scale's reliability values are high; in other words, it produces consistent data. 3.4. Correlations The relationship between the general satisfaction of the teacher candidates concerning the SIDES, their usage attitudes and intentions, the perceived ease of use, perceived usefulness, usefulness and face of SIDES, [unproblematic] access to SIDES and SIDES's service quality was investigated with the correlation analysis. The relationship between the variables is given in Table 4. Table 4. Correlation Results *GS UAI PEU PB U UA F **S Q *GS 1 UAI .496 1 PEU .385 .336 1 PU .581 .664 .637 1 U .640 .849 .487 .859 1 UA .530 .420 .328 .539 .643 1 F .412 .579 .462 .708 .672 .437 1 **SQ .641 .604 .586 .703 .758 .692 .664 1 *General satisfaction (GS), Usage Attitude and Intention (UAI), Perceived Ease of Use (PEU), Perceived Usefulness (PU), Usefulness (U), [Unproblematic] Access (UA), Face (F), **Service Quality (SQ). Correlation coefficients being between 0-.30, .30 and .70, and .70 and 1 indicate low-level, mid-level and high-level relationships respectively (Buyukozturk, 2011). As for the correlation results, it can be seen that there are is a high-level relationship between International Online Journal of Education and Teaching (IOJET) 2019, 6(3), 563-580 573 Usefulness and Perceived usefulness, Usefulness and Usage attitude and intention, Face and Perceived usefulness, Service quality and Perceived usefulness, and Service quality and Usefulness while the correlation between other variables is on a medium level. 4. Discussion Considering the importance of teaching practice course in teacher education, supporting candidate teachers in during their internship process is emerging as a necessity in higher education institutions. Accordingly, this core course of education faculties needs to be more attention than ever. This issue was also be felt by the Ministry of National Education (MEB) of Turkey, which is called “Evaluating Candidate Teacher” web application has developed and the necessity of evaluating the candidate teachers' performance during the internship training process has been put forward (MEB, 2018a). According to this practice, both the practice teacher and instructors of the course are asked to evaluate the performance of the trainee during the internship on developed system. It is clearly understood that policymakers and administrators trying to surround their internship process via technology. In addition to this, within the scope of the coordination and cooperation between the Ministry of National Education and the Higher Education Council for the training of teachers, the feedback on the implementation of the process in order to carry out the successful implementation of the Teaching Practice and Guidance Practices in the education and training institutions affiliated to the Ministry of National Education is needs to be attention in the design of the process (MEB, 2018b). This policy showed the importance of coordination and cooperation in the process of internship. However, considering the recent policies, it is possible to say technology is still not going beyond to use it, and internship process is not evaluating within the framework of integration of technology. During internship process lack of technology integration may damage candidates’ internal beliefs considering lack of interaction between instructor of the course and practice teacher, and technology would only stay as a tool which is not used effective and productive to support candidates’ improvement. Well, to overcome this situation the system developed and called SIDES, conducted underTUBITAK-1001 program, to support candidates’ attainments into practice, be monitored, be informed about details of schools and tasks, be interacted, be communicated and be assisted as claimed in Dallmer’s (2004) study the importance of cooperation, and underlined by YOK (1998c) that feedback is a vital element of the process. Hence, it is thought that developed facilitates by SIDES and under the teaching practice lesson will support teacher candidates, the practice teacher at the school and the instructor at the university, and developed measurement tools within the framework of the usability of support services which affects candidates’ internal beliefs (Flavián, Guinalíu & Gurrea, 2006), acceptance on the use of system which affects beliefs and attitudes to the use of system by candidates (Ertmer et al., 2012; Venkatesh, & Davis, 2000), quality of system (Wixom & Todd, 2005) could shed a light to be enhanced the learning and teaching activities in internship process. Accordingly, it is thought that the developed system and valid and reliable scales may be useful for teachers, instructors, policy-makers, administrators, teacher educators, candidate teachers, and benefit from developed tools to evaluate their any teaching and learning processes which is used technology. Based on the results, users may improve their weakness and save their strength facilities by using these developed scales. 5. Conclusion The scales developed with validity and reliability studies conducted in the pilot scheme of the SIDES project which is carried out under TUBITAK-1001 program are designed to be Hamutoğlu, Güngören, Duman, Horzum, Kıyıcı & Akgün 574 used in the final application. By this means, it should be revealed whether teacher candidates are satisfied with SIDES, accept SIDES, find SIDES as usable while using it; in other words, SIDES's service quality, or SIDES's effectiveness. In addition, these scales ser ve as an example for researchers who develop similar support systems. To reveal the effectiveness of the “Internship Monitoring and Support System (SIDES)” which was developed in a project being conducted under TUBITAK-1001 program, “SIDES General Satisfaction Scale (SGSS)”, “SIDES Acceptance Scale (SACS)”, and “SIDES Usability Scale (SUSS)” were developed in this research. While performing the validity and reliability studies of the scales, the results of construct validity and reliability analyses were also provided. The content validity studies were conducted with expert opinions to develop “SIDES General Satisfaction Scale (SGSS)”, “SIDES Acceptance Scale (SACS)”, and “SIDES Usability Scale (SUSS)” for the “Internship Monitoring and Support System (SIDES)”. According to the results of the EFA which explored scales' structure, and it can be said that the scales have the construct validity. Cronbach's Alpha internal consistency coefficients obtained from the reliability studies show that the scales are reliable. High scores obtained from the scales show that the situation related to the measured feature is positive. The correlation results were utilized to reveal to what extent the scales predicted general satisfaction and service quality and the relationship between the variables. The correlation results indicate that there is a high-level relationship between the variables U and PU, U and UAI, F and PU, SQ and PU and SQ and U. 6. Limitations and Recommendations The scales can be used for coming up with some findings for newly developed web-based monitoring and support systems with considering to determine users' opinions on the usability of the system, and individual acceptance levels, users' general satisfaction. Therefore, the system can be improved to be more effective. These scales can form a basis for the development of more generic scales in future. Although construct validity, divergent and convergent validities and composed reliability were analyzed via using CFA, the results were not reported because of the same data used for EFA analyzes, which collected from the same participants. Due to this limitation, future studies could focus on testing the results of CFA with new and widen samples. Furthermore, divergent and convergent validities and composed reliability can also be tested. Moreover, future studies may also investigate the effects of developed scales on general satisfaction level of system users based on multiple linear regression. The developed scales may shed a light to administrators, educators, teachers, and candidate teachers in terms of evaluating experiences on system users, and help to rehabilitate and monitor their systems, and support system users’ internal beliefs. International Online Journal of Education and Teaching (IOJET) 2019, 6(3), 563-580 575 References Abazoğlu, İ. (2014). Dünyada öğretmen yetiştirme programları ve öğretmenlere yönelik mesleki gelişim uygulamaları. International Periodical for the Languages, Literature and History of Turkish or Turkic, 9(5), 1-46. AFT (American Federation of Teachers) (2008). Principles for Professional Development. Reached from https://www.aft.org/sites/default/files/pd_principles_2008.pdf on 26.07.2017. Akbulut, Y. (2010). Sosyal Bilimlerde SPSS Uygulamaları (1. Baskı). İdeal Kültür Yayıncılık: İstanbul. Akgun, O. E., Kıyıcı, M., Horzum, M. B., Hamutoğlu, N. B., Güngören, Ö. C. & Duman, İ. (2015a). Determination of learning outcomes, learning tasks and measurement- assesment tools on-site teaching practice for student teachers. ERPA International Congresses on Education. Akgun, Ö. E., Gökmen, Ö. F., Özer, E. A., Kaymak, Z. D., Horzum, M. B., & Kıyıcı, M. (2015b). Öğretmenlik uygulamasına destek vermek için geliştirilecek bilişim sistemine yönelik ihtiyaç analizi sonuçları. Eğitim Teknolojisi Kuram ve Uygulama, 5(2), 54-72. Aladwani, A. M. & Palvia, P. C. (2002). Developing and validating an instrument for measuring user-perceived web quality. Information & management, 39(6), 467-476. Aydin, F., & Akgun, Ö. E. (2014). Eğitim fakültesi BÖTE son sınıf öğrencilerinin okul deneyimi ve öğretmenlik uygulaması derslerinde karşılaştıkları sorunlar. Sakarya Üniversitesi Eğitim Fakültesi Dergisi, 28, 1-14. Aytaçlı, B. (2012). İlköğretim matematik öğretmenliği lisans programında yer alan okul deneyimi ve öğretmenlik uygulaması derslerinin değerlendirilmesi (Master’s thesis). Ege Üniversitesi, Sosyal Bilimler Enstitüsü, İzmir. Buyukozturk, Ş. (2011). Veri analizi el kitabı. Ankara: Pegem A Yayıncılık. Çakmak, E. K., Güneş, E., & Tahsin, M. (2011). Web sitesi kullanılabilirlik ölçeğinin geliştirilmesi: geçerlik, güvenirlik analizi ve uygulama sonuçları. Pegem Eğitim ve Öğretim Dergisi, 1(2), 31-40. Canan-Güngören, Ö., Bektaş, M., Öztürk, E., & Horzum, M. B. (2014). Tablet bilgisayar kabul ölçeği-geçerlik ve güvenirlik çalışması. Eğitim ve Bilim, 39(176), 69-79. Cansaran, A., İdil, Ö. & Kalkan. M. (2006). Fen bilgisi eğitimi anabilim dallarındaki “okul deneyimi” uygulamalarının değerlendirilmesi. Gazi Eğitim Fakültesi Dergisi, 26(1), 83-99. Dallmer, D. (2004). Collaborative relationships in teacher education: A personal narrative of conflicting roles. Curriculum Inquiry, 34(1), 29-45. https://www.aft.org/sites/default/files/pd_principles_2008.pdf%20%20on%2026.07.2017 https://www.aft.org/sites/default/files/pd_principles_2008.pdf%20%20on%2026.07.2017 Hamutoğlu, Güngören, Duman, Horzum, Kıyıcı & Akgün 576 Davis, F. D., Bagozzi, R. P., & Warshaw, P. R. (1989). User acceptance of computer technology: a comparison of two theoretical models. Management Science, 35(8), 982-1003. Ertmer, P. A., Ottenbreit-Leftwich, A. T., Sadik, O., Sendurur, E., & Sendurur, P. (2012). Teacher beliefs and technology integration practices: A critical relationship. Computers & education, 59(2), 423-435. European Commission, (2013). Supporting Teacher Educators: For Better Learning Outcomes. Retrieved from http://ec.europa.eu/dgs/education_culture/repository/education/policy/school/doc/supp ort-teacher-educators_en.pdf. Field, A. (2013). Discovering statistics using IBM SPSS statistics. London: Sage. Fisher, M. (1996). Integrating information technology: Competency recommendations by teachers for teacher training. Journal of Information Technology for Teacher Education, 5(3), 233-238. Flavián, C., Guinalíu, M., & Gurrea, R. (2006). The role played by perceived usability, satisfaction and consumer trust on website loyalty. Information & Management, 43(1), 1-14. Gökçe, E., & Demirhan, C. (2005). Öğretmen adaylarının ve ilköğretim okullarında görev yapan uygulama öğretmenlerinin öğretmenlik uygulaması etkinliklerine ilişkin görüşleri. Ankara Üniversitesi Eğitim Bilimleri Fakültesi Dergisi, 38(1), 43-71. Gömleksiz, M. N., Mercin, L., Bulut, İ., & Atan, U. (2006). Okul deneyimi II. dersine ilişkin öğretmen adaylarının görüşleri (sorunlar ve çözüm önerileri). Eğitim Araştırmaları, 23, 148-158. Green, D., & Pearson, J. M. (2006). Development of a web site usability instrument based on ISO 9241-11. Journal of Computer Information Systems, 47(1), 66-72. Hair, J. F., Black, B., Babin, B., Anderson, R. E., & Tahtam, R. L. (2006). Multivariate data analysis. Upper Saddle River: Prentice Hall. Hixon, E., & So, H. J. (2009). Technology's role in field experiences for preservice teacher training. Journal of Educational Technology & Society, 12(4), 294-304. Jung, I. (2005). ICT-pedagogy integration in teacher training: Application cases worldwide. Educational Technology & Society, 8(2), 94-101. Kırksekiz, A., Uysal, M., İşbulan, O., Akgun, E. A., Kıyıcı, M. & Horzum, M. B. (2015). Okul deneyimi ve öğretmenlik uygulaması derslerine eleştirel bir bakış: Problemler, beklentiler ve çözüm önerileri. Bartın Üniversitesi Eğitim Fakültesi Dergisi, 4(2), 433-451. Kiraz, E. (2002). Öğretmen adaylarının hizmet öncesi mesleki gelişiminde uygulama öğretmenlerinin işlevi. Eğitim Bilimleri ve Uygulama, 1(2), 183-196. http://ec.europa.eu/dgs/education_culture/repository/education/policy/school/doc/support-teacher-educators_en.pdf http://ec.europa.eu/dgs/education_culture/repository/education/policy/school/doc/support-teacher-educators_en.pdf International Online Journal of Education and Teaching (IOJET) 2019, 6(3), 563-580 577 Kline, R. B. (1998). Methodology in the social sciences. Principles and practice of structural equation modeling. New York, NY, US: Guilford Press. Koç, C., & Yıldız, H. (2012). The reflectors of teaching experiences: Diaries. Education, 37(164), 223-236. Lee, Y., Kozar, K. A., & Larsen, K. R.T. (2003). The technology acceptance model: Past, present, and future. Communications of the Association for Information Systems, 50, 752-780. Malaty, G. (2006). What are the reasons behind the success of Finland in PISA. Gazette des Mathématiciens, 108, 59-66. McGee, I. E. (2019). Developing mentor teachers to support student teacher candidates. SRATE Journal, 28(1), 23-30. Ministry of National Education- MEB (2018a). Uygulama öğrencisi değerlendirme sistemi. Retrieved from https://uod.meb.gov.tr/ on 13.06.2019. Ministry of National Education- MEB (2018b). Fakülte kurum işbirliği kapsamındaki öğretmenlik uygulamaları. 13666252-399-E.23603631 sayı ve 07.12.2019 tarihli yazı. Retrieved from https://uod.meb.gov.tr/ on 13.06.2019. Mertler, C. A., & Vanatta, R. A. (2005). Advanced and multivariate statistical methods (3rd Ed.). Glendale, CA: Pyrzcak Publishing. Nielsen, J. (2000). Designing web usability. Indianapolis, IN: New Riders Publishing. OECD (Organization for Economic Cooperation and Development). (2005). Education and training policy: Teachers matter, attracting, developing and retaining effective teachers. Retrieved from https://www.oecd.org/edu/school/34990905.pdf. Özkılıç, R., Bilgin, A., & Kartal, H. (2008). Öğretmenlik uygulaması dersinin öğretmen adaylarının görüşlerine göre değerlendirilmesi. İlköğretim Online, 7(3), 726-737. Paker, T. (2008). Öğretmenlik yygulamasında öğretmen adaylarının uygulama öğretmeni ve uygulama öğretim elemanının yönlendirmesiyle ilgili karşılaştıkları sorunlar. Pamukkale Üniversitesi Eğitim Fakültesi Dergisi, 23(23), 132-139. Parasuraman, A., Zeithaml, V. A., & Malhotra, A. (2005). ES-QUAL a multiple-item scale for assessing electronic service quality. Journal of Service Research,7(3), 213-233. Pearrow, M. (2000). Web site usability. RockLand, MA: Charles River Media. Rıza, E. T. & Hamurcu, H. (2000). Sınıf öğretmenliği bölümü öğrencilerinin okul deneyimi ve öğretmenlik uygulamasına yönelik görüşleri. Pamukkale Üniversitesi Eğitim Fakültesi Dergisi, 8, 1-8. https://uod.meb.gov.tr/ https://uod.meb.gov.tr/ https://www.oecd.org/edu/school/34990905.pdf Hamutoğlu, Güngören, Duman, Horzum, Kıyıcı & Akgün 578 Sarıçoban, A. (2008). Okul deneyimi ve öğretmenlik uygulaması derslerine ilişkin uygulama öğretmenleri ve öğretmen adaylarının görüşleri. Gazi University Journal of Gazi Educational Faculty, 28(3), 31-55. Sánchez-Prieto, J. C., Hernández-García, Á., García-Peñalvo, F. J., Chaparro-Peláez, J., & Olmos-Migueláñez, S. (2019). Break the walls! Second-Order barriers and the acceptance of mLearning by first-year pre-service teachers. Computers in Human Behavior, 95, 158-167. Seçer, Z., Çeliköz, N. & Kayılı, A. G. G. (2010). Okul öncesi öğretmenliği okul uygulamalarında yaşanan sorunlar ve çözüm önerileri. Yüzüncü Yıl Üniversitesi Eğitim Fakültesi Dergisi, 7(1), 128-52. Shneiderman, B. (2004). Designing the user interface: Strategies for effective human- computer interaction (4th Ed.). Reading, MA: Addison-Wesley. Şimşek, N. (1998). Öğretim amaçlı bilgisayar yazılımlarının değerlendirilmesi. Ankara: Siyasal Yayınevi. Şişman, M. (2003). Bir meslek olarak öğretmenlik: Öğretmenliğe giriş. Ankara: PegemA Yayıncılık. Sourdot, L.A., Smith, C., Anderson, G. & Whitworth, J. (2017). The TWUFCL experiment: Authentic engagement with technology for teacher candidates and education professionals. In P. Resta & S. Smith (Eds.), Proceedings of Society for Information Technology & Teacher Education International Conference (pp. 2470-2473). Austin, TX, United States: Association for the Advancement of Computing in Education (AACE). Retrieved from https://www.learntechlib.org/p/177545/. Tabachnick, B. G. & Fidell, L.S. (2007). Using Multivariate Statistics (5th ed.). Boston: Allyn and Bacon. Terhart, E. (2003) Teacher education in Germany: Current state and new perspectives, In: B. Moon, L. Vlasceanu & L. Barrows (Eds). Institutional Approaches to Teacher Education within Higher Education in Europe: Current Models and New Development. Bucharest: UNESCO-CEPES. Venkatesh, V., & Davis, F. D. (2000). A theoretical extension of the technology acceptance model: Four longitudinal field studies. Management Science, 46(2), 186-204. Venkatesh, V., Davis, G. D., & Davis, F. D. (2003). User acceptance of information technology: Towards a unified view. MIS Quarterly, 27(3), 425-478. Vescio, V., Ross, D., & Adams, A. (2008). A review of research on the impact of professional learning communities on teaching practice and student learning. Teaching and Teacher Education, 24(1), 80-91. Wang, Y. S., Wang, H. Y., & Shee, D. Y. (2007). Measuring e-learning systems success in an organizational context: Scale development and validation. Computers in Human Behavior, 23(4), 1792-1808. https://www.learntechlib.org/p/177545/ International Online Journal of Education and Teaching (IOJET) 2019, 6(3), 563-580 579 Wixom, B. H., & Todd, P. A. (2005). A theoretical integration of user satisfaction and technology acceptance. Information systems research, 16(1), 85-102. Yang, Z., Cai, S., Zhou, Z. & Zhou, N. (2005). Development and validation of an instrument to measure user perceived service quality of information presenting Web portals. Information & Management, 42, 575-589. Yükseköğretim Kurulu-YOK, (1998). YÖK Dünya Bankası fakülte-okul işbirliği kılavuzu. Ankara: Öğretmen Eğitimi Dizisi. Hamutoğlu, Güngören, Duman, Horzum, Kıyıcı & Akgün 580 Appendix A. SIDES Overall Satisfaction Scale Items 1. I did not have any difficulties while using the SIDES. 2. I like using the SIDES. 3. I got information about my tasks using SIDES. 4. I managed to easily follow my tasks on the SIDES. 5. Students, lecturers and teachers can communicate easily using SIDES. 6. SIDES calendar add-on can notify me with reminders. Appendix B. SIDES Acceptance Scale Items 1. It is very easy to learn using SIDES for me. 2. I can easily perform what I want to do with SIDES. 3. I can easily learn the skills necessary to use the SIDES. 4. It is easy to use SIDES 5. Using SIDES has improved my productivity in the course 6. Using SIDES has increased my effectiveness in the course 7. I think it is beneficial to use the SIDES 8. It is a good idea to use the SIDES. 9. Using SIDES makes the internship enjoyable 10. Using SIDES is suitable for internship applications 11. SIDES should be used in internship applications 12. I want SIDES to be used in upcoming semesters. 13. I suggest that SIDES should be used in other departments 14. SIDES is necessary for internship application Appendix C. SIDES Usability Scale Items 1. SIDES can be used to support internship applications 2. When I login to SIDES runs error-free 3. I did not have any system crashes while using the SIDES. 4. There are not any system freezes when surfing on the SIDES 5. SIDES makes it easy for me to reach information I needed for teaching application 6. SIDES makes easy all tasks which are necessary for the course 7. I can quickly complete a task using SIDES. 8. SIDES is a well-organized system. 9. Using SIDES is not complicated 10. I have enough skills to use SIDES 11. After learning to use a section of SIDES, it is easy to learn other parts. 12. I can fix whenever I make an erroneous on SIDES 13. Completing a task using SIDES was beneficial for me 14. I have easily completed a task using SIDES 15. I have completed all my duties in the context of teaching practice course with SIDES. 16. SIDES facilitated to achieve goals of teaching practice course 17. SIDES links work correctly 18. Fonts used on SIDES are suitable. 19. Colours used on SIDES are suitable.