International Journal of Interactive Mobile Technologies (iJIM) – eISSN: 1865-7923 – Vol. 15, No. 04, 2021 Short Paper—Improving Questionnaire Reliability using Construct Reliability for Researches… Improving Questionnaire Reliability using Construct Reliability for Researches in Educational Technology https://doi.org/10.3991/ijim.v15i04.20199 Mohd Shafie Rosli () Universiti Teknologi Malaysia, Johor Bahru, Malaysia shafierosli@utm.my Nor Shela Saleh Universiti Tun Hussein Onn Malaysia, Parit Raja, Malaysia Sultan Hammad Alshammari University of Hail, Hail, Saudi Arabia Mohd Mokhzani Ibrahim Universiti Pendidikan Sultan Idris, Tanjung Malim, Malaysia Azri Syazwan Atan, Noor Azean Atan Universiti Teknologi Malaysia, Johor Bahru, Malaysia Abstract—This paper is exploring on maneuver to improve research in- strument reliability in scientific research related to Educational Technology by analyzing the traditional Cronbach’s alpha reliability using SPSS software and the newer statistical tool, AMOS using Construct Reliability (CR) approach. Two sets of data were used as sample to perform the comparison. The first set of data is from a research involving Technology Enhanced Learning Environ- ment. The second data are sampled from research in digital competency. Find- ing from this paper concluded that, conventional approach of using Cronbach’s alpha have lower reliability than the newer approach of using CR. Using Cronbach’s alpha show tendency toward measuring consistency instead of reli- ability. CR offer better definition of reliability and give a robust measurement of reliability in research. This paper had shed light into offering alternative ap- proach to the commonly and widely uses of research reliability especially when it involves questionnaire as instrument. Keywords—Educational Technology, Online Learning, Questionnaire Reliabil- ity, Construct Reliability 1 Introduction Researches in educational technology are various in natures. The researches might be experimental, involving series of interventions to understand the effect of interven- tion. To understand the natural character of the samples without any intervention, iJIM ‒ Vol. 15, No. 04, 2021 109 https://doi.org/10.3991/ijim.v15i04.20199 mailto:example@example.org Short Paper—Improving Questionnaire Reliability using Construct Reliability for Researches… questionnaire-based research is being widely applied. Reliability is a big issue for a questionnaire, without a solid reliability value the instrument deems invalid. Cronbach’s alpha, also known as alpha coefficient, is widely used in educational technology, education as well as social sciences. As technology evolves, construct reliability which also known as composite reliability is taking place as a new reliabil- ity coefficient alternative. Yet, both coefficients generally returned a different value. 2 Problem Statement The main concern for questionnaire-based research is, its instrument reliability. The popular application of reliability test is Cronbach’s alpha (e.g. [1-4]). Calculating alpha is also simpler when compared to other estimates, as only one test is needed [5] and it is easier to be interpreted [6]. The formula for Cronbach’s alpha is with  is the Cronbach’s alpha coefficient, n referring to the number of questions in the instrument. Meanwhile, Vi is the variance of scores and Vtest is the total variance of overall scores. The recent development in educational technology research had gave researchers with the flexibility of adopting a new relia- bility test using CFA (e.g. [7, 8]). The purpose of CFA is to understand the goodness of the researcher’s factor model from the aspect of ensuring all items in the question- naire are representing their respective latent variable as in the measurement model. By manipulating CFA, researcher can gain the value for its item construct reliability (CR), which is comparable to Cronbach’s alpha. The formula for CR is CR is the construct reliability,  is factor loading and  representing the measurement error. Both tests are different statistically, yet somehow have the same function. Therefore, which reliability test is better and could give researcher a much solid needed instrument? Thus, two sets of data will undergo Cronbach’s Alpha and CFA via CR value reliability test to find out which test give a better result. 3 Methodology This research involves two sets of data labelled as Data Set I and Data Set II. The methodological approach for both data sets are: 3.1 Methodology for data set i The same data was undergoing two different reliability tests. The data were origi- nated from the authors’ research on technology enhanced learning environment. Twelve respondents were sampled using random sampling technique for Cronbach’s alpha test. Two hundred respondents were sampled using random sampling technique for measuring CR value using CFA. The sample size is based on the minimum size of sample for CFA research. 110 http://www.i-jim.org Short Paper—Improving Questionnaire Reliability using Construct Reliability for Researches… The internal consistency technique was applied for this purpose of measuring Cronbach’s alpha. The reliability was found to be Cronbach’s alpha = .878. The relia- bility details are as in Table 1. Table 1. The Reliability Data Construct Item Cronbach’s Alpha if Item Deleted Cronbach’s Alpha A A1 .721 .781 A2 .720 A3 .738 A4 .734 B B5 .688 .771 B6 .727 B7 .743 B8 .705 C C9 .811 .824 C10 .763 C11 .760 C12 .780 D D13 .545 .636 D14 .595 D15 .558 D16 .570 Relying on the Cronbach’s alpha value, all the item reliability is excellent, and consistency was recorded to be very high. To make a comparison, the data then un- derwent CFA via Structural Equation Modeling (SEM) and measurement model as in Figure 1 was constructed. Fig. 1. The Measurement Model iJIM ‒ Vol. 15, No. 04, 2021 111 Short Paper—Improving Questionnaire Reliability using Construct Reliability for Researches… The measurement model had returned standardized regression weights as in Table 2. Table 2. Standardized Regression Weights Estimate A1 Construct A .731 A2 Construct A .684 A3 Construct A .685 A4 Construct A .672 B5 Construct B .619 B6 Construct B .546 B7 Construct B .722 B8 Construct B .798 C9 Construct C .729 C10 Construct C .796 C11 Construct C .771 C12 Construct C .653 D13 Construct D .504 D14 Construct D .558 D15 Construct D .546 D16 Construct D .607 Data in Table 2 was later transformed into a square of factor loading, 2 and meas- urement error,  as in Table 3 and CR value for each construct is as in Table 4. Table 3. Square of Factor Loading and Measurement Error Construct Item Square of Factor Loading, 2 Measurement Error, A A1 .534 .466 A2 .468 .532 A3 .469 .531 A4 .452 .548 Sum 1.923 2.077 B B5 .105 .895 B6 .298 .702 B7 .521 .479 B8 .637 .363 Sum 1.561 2.439 C C9 .531 .469 C10 .634 .366 C11 .594 .406 C12 .426 .574 Sum 2.185 1.815 D D13 .254 .746 D14 .331 .669 D15 .298 .702 D16 .368 .632 Sum 1.251 2.749 112 http://www.i-jim.org Short Paper—Improving Questionnaire Reliability using Construct Reliability for Researches… Table 4. Construct Reliability (CR) Value Construct CR Value A .481 B .390 C .546 D .313 3.2 Methodology for data set ii A set of data from the authors’ research on digital competency was used as data set ii. The data also underwent two different reliability tests. Ten respondents were sam- pled via random sampling technique. Through internal consistency technique, the Cronbach’s alpha was found at .868. The reliability values for each construct are as in Table 5. Table 5. Reliability via Cronbach’s Alpha Item Cronbach’s Alpha if Item Deleted Cronbach’s Alpha E1 .840 .868 E2 .837 E3 .849 E4 .848 E5 .840 E6 .840 E7 .890 Items in Table 5 are highly reliable. Then, the data underwent the CFA test. For the first analysis, the model returned χ²df = 2.151, RMR = .026, CFI = .846 and RMSEA = .199 which not fulfilling the requirement of a fit model. Second analysis was con- ducted derived the value of χ²df = 1.318, RMR = .018, CFI = .960 and RMSEA = .109 that meet the minimum parameter of a fit model. Yet, item E7 indicating a factor loading .165, which is, less than the minimum factor loading of .5. To reinforce the finding, the value of Average Variance Extracted (AVE) and CR was calculated as in Table 6. Table 6. Factor Loading, Average Variance Extracted and Construct Reliability Item Factor Loading AVE CR E1 .551 0.321 .752 E2 .763 E3 .587 E4 .541 E5 .571 E6 .611 E7 .165 iJIM ‒ Vol. 15, No. 04, 2021 113 Short Paper—Improving Questionnaire Reliability using Construct Reliability for Researches… The comparison of reliability test between Cronbach’s alpha and CR from CFA shows a deviation. Both tests were conducted for the purpose of measuring reliability, yet it was found that using CR from CFA, the researcher will have much better relia- bility value. An instrument that was qualified as highly reliable using Cronbach’s alpha has been dignified as unreliable using CFA. By using CFA, the reliability value returned is smaller in which gives more precise than using Cronbach’s alpha. 4 Discussion There are two ways of applying CFA commonly found in literature. In a number of researches, CFA was used for the purpose of assessing construct reliability and Cronbach’s alpha for the purpose of measuring instrument internal consistency. While, some other research used CFA to validate their model and Cronbach’s alpha as reliability test. Both approaches are well accepted by the scientific community. Using CFA as a reliability test gives researcher with a much precision and smaller reliability value as reported in this study. Despite the fact that the coefficient Cronbach’s alpha is the most widely used estimator for the purpose of reliability, it has been well criticized for being a lower bound that render the true reliability to be underestimated [9]. Cronbach’s alpha requires the compliance toward classical item-score assumption, tau equivalency assumption and uncorrelated-errors assumption where, when a viola- tion occurs, leading to negatively biased, relatively unbiased and positive biased coef- ficient alpha [7, 10]. Cronbach’s alpha is easy to be misinterpreted and is appropriate to interpret as an estimate of reliability according to the internal consistency between items [10]. Cronbach’s alpha is also prone to the effect of test length [11] and has been subject to so much misunderstanding and confusion [12, 13]. Helms et. al. [14] suggest researchers to calculate composite reliability rather than total-score reliability for the purpose of good practices in analyzing, interpreting and using reliability data. As Cronbach’s alpha is based on total-score reliability approach, it is highly suggested for the researcher to use the construct reliability as did by [15]. It is suitable for future research improvement in educational technology such as tech- nology enhanced learning environment and HOTS as did by [16]. 5 Conclusion Cronbach’s alpha has been well accepted as a reliability test among researchers in educational technology as well as in other social sciences researchers. However, as verified by this study, Cronbach’s alpha normally returning a higher reliability value when compared with the reliability value returned by CR. Despite the low value re- turned by CR, it is believed to be demonstrating a higher precision due to its compo- site nature. Literature had shown that Cronbach’s alpha is subjected toward a number of assumptions that give negative effects when violated where in research, these as- sumptions is delicate to be confirmed. In addition, as Cronbach’s alpha is being a 114 http://www.i-jim.org Short Paper—Improving Questionnaire Reliability using Construct Reliability for Researches… lower bound, it is highly suggested for researchers to use CR as their reliability coef- ficient as a maneuver of intensifying questionnaire-based instrument reliability. 6 Acknowledgement Authors would like to thank Ministry of Education and Universiti Teknologi Ma- laysia for the research funding through UTM Fundamental Research Grant (UTMFR) with Project Number Q.J130000.2553.21H23. 7 References [1] Rosli, M. S., Saleh, N. S., Aris, B., Ahmad, M. H., & Salleh, S. M. (). “Ubiquitous hub for digital natives”, International Journal of Emerging Technologies in Learning, Vol. 11, No. 2, pp 29-34. 2016, https://doi.org/10.3991/ijet.v11i02.4993 [2] Cunha, M., Duarte, J., Cruz, A., & Students 26Th, C., “Learning strategies: Validating a questionnaire, Turkish Online Journal of Educational Technology, 2015, 284-300. [3] Fernández Batanero, J. M., & Torres González, J. A., “Teacher attitudes and best practices with ICT faculty adult continuing education in Andalusia”, Revista Complutense de Edu- cacion, Vol. 26, pp 33-49, 2015, doi:10.5209/rev_RCED.2015.v26.43812 [4] Juned, M., & Adil, M., “Factors influencing adoption of ubiquitous internet amongst stu- dents”, International Journal of Information and Communication Technology Education, Vol. 11, No. 3, pp 62-76, 2015, https://doi.org/10.4018/ijicte.2015070106 [5] Tavakol, M., & Dennick, R., “Making sense of Cronbach's alpha”, International journal of medical education, Vol. 2, pp 53-55, 2011, https://doi.org/10.5116/ijme.4dfb.8dfd [6] Yang, Y., & Green, S. B., “Coefficient Alpha: A Reliability Coefficient for the 21st Centu- ry?”, Journal of Psychoeducational Assessment, Vol. 29, No. 4, pp 377-392, 2011, https://doi.org/10.1177/0734282911406668 [7] Aypay, A., Çelik, H. C., & Sever, M., “Technology acceptance in education: A study of pre-service teachers in Turkey”, Turkish Online Journal of Educational Technology, Vol. 11, No. 4, pp 264-272, 2012 [8] Liu, F., Ritzhaupt, A., & Cavanaugh, C., “Leaders of school technology innovation: A con- firmatory factor analysis of the Change Facilitator Style Questionnaire (CFSQ)”, Journal of Educational Administration, Vol. 51, No. 5, pp 576-593, 2013, https://doi.org/10.1108/ jea-01-2012-0011 [9] Peterson, R. A., & Kim, Y., “On the relationship between coefficient alpha and composite reliability”, Journal of Applied Psychology, Vol. 98, No. 1, pp 194-198, 2013, doi:10.1037/a0030767 [10] Novick, M. R., & Lewis, C., “Coefficient alpha and the reliability of composite measure- ments”, Psychometrika, Vol. 32, No. 1, pp 1-13, 1967, https://doi.org/10.1007/ bf02289400 [11] Schmitt, N., “Uses and abuses of coefficient alpha”, Psychological Assessment, Vol. 8, No. 4, pp 350-353, 1996, https://doi.org/10.1037/1040-3590.8.4.350 [12] Sijtsma, K., “On the use, the misuse, and the very limited usefulness of cronbach's alpha”, Psychometrika, Vol. 74, No. 1, pp 107-120, 2009, https://doi.org/10.1007/s11336-008- 9101-0 iJIM ‒ Vol. 15, No. 04, 2021 115 https://doi.org/10.3991/ijet.v11i02.4993 https://doi.org/10.4018/ijicte.2015070106 https://doi.org/10.5116/ijme.4dfb.8dfd https://doi.org/10.1177/0734282911406668 https://doi.org/10.1108/jea-01-2012-0011 https://doi.org/10.1108/jea-01-2012-0011 https://doi.org/10.1007/bf02289400 https://doi.org/10.1007/bf02289400 https://doi.org/10.1037/1040-3590.8.4.350 https://doi.org/10.1007/s11336-008-9101-0 https://doi.org/10.1007/s11336-008-9101-0 Short Paper—Improving Questionnaire Reliability using Construct Reliability for Researches… [13] Shelby, L. B., “Beyond Cronbach's Alpha: Considering Confirmatory Factor Analysis and Segmentation”, Human Dimensions of Wildlife, Vol. 16, No. 2, pp 142-148, 2011, https://doi.org/10.1080/10871209.2011.537302 [14] Helms, J. E., Henze, K. T., Sass, T. L., & Mifsud, V. A., “Treating Cronbach's Alpha Reli- ability Coefficients as Data in Counseling Research”, The Counseling Psychologist, Vol. 34, No. 5, pp 630-660, 2006, https://doi.org/10.1177/0011000006288308 [15] Aris, B., Gharbaghi, A., Ahmad, M. H., & Rosli, M. S., “A check list for evaluating per- suasive features of mathematics courseware”, International Education Studies, Vol. 6, No. 9, pp 125-134, 2013, https://doi.org/10.5539/ies.v6n9p125 [16] Rosli, M. S., Aris, B., & Ahmad, M. H., “Online intellectual transformation system”, Con- temporary Engineering Sciences, Vol. 8, No. 1-4, pp 39-47, 2015, https://doi.org/10. 12988/ces.2015.412254 8 Authors Mohd Shafie Rosli is a Malaysian active researcher in HOTS and online learning environment with more than a decade of research experience in technological assisted cognitive augmentation. He had supervised a significant number of postgraduate re- searches on the application of online learning environment in the field of education. Email: drshafierosli@gmail.com Nor Shela Saleh is serving Universiti Tun Hussein Onn Malaysia. She is an expert in social sciences statistics and research methodology, contributing to data analysis and literature support. Sultan Hammad Alshammari is serving University of Hail, Saudi Arabia. He re- ceived his PhD in Educational Technology from Universiti Teknologi Malaysia in 2018. He is an expert in questionnaire-based researches in Learning Management System. Mohd Mokhzani Ibrahim is serving Universiti Pendidikan Sultan Idris and re- ceived his PhD from Universiti Teknologi Malaysia in 2018. He had experience con- ducting research about blended learning using Learning Management System. Azri Syazwan Atan is a Master of Philosophy graduate from Universiti Teknologi Malaysia in 2018. His research is about creative thinking skills development using Learning Management System for higher education. Noor Azean Atan is serving Universiti Teknologi Malaysia as Senior Lecturer and had vast experience in online learning as researcher and academic administrator relat- ed to online learning. Article submitted 2020-12-04. Resubmitted 2021-01-09. Final acceptance 2021-01-11. Final version published as submitted by the authors. 116 http://www.i-jim.org https://doi.org/10.1080/10871209.2011.537302 https://doi.org/10.1177/0011000006288308 https://doi.org/10.5539/ies.v6n9p125 https://doi.org/10.12988/ces.2015.412254 https://doi.org/10.12988/ces.2015.412254 file:///E:/IAOE%2020--/IAOE%202021/Review/iJIM/iJIM%2004/Divya/drshafierosli@gmail.com