Australasian Journal of Educational Technology, 2020, 36(3). 26 Perceptions of learning management system quality, satisfaction, and usage: Differences among students of the arts Joyce Hwee Ling Koh University of Otago, New Zealand Rebecca Yen Pei Kan Nanyang Academy of Fine Arts, Singapore The model of information system success has been used to evaluate students’ satisfaction with the system quality, information quality, and service quality of learning management systems. This study extends the model by considering the pedagogical dimensions of instructional quality, learning quality, and interaction quality as well as how perceived usage frequency influences students’ quality perceptions. Through a sample of 376 higher education students, this study validated a five-factor quality structure for the evaluation of learning management systems with respect to the perceptions of students of the arts. Regression analyses showed differences among the quality factors that predicted satisfaction for arts students perceiving infrequent, average frequency, and frequent use. Although the quality of instructional experiences predicted satisfaction for all student groups, information quality significantly predicted satisfaction for only infrequent and average frequency users. System quality was a significant predictor of satisfaction for only average frequency and frequent users, whereas only frequent users perceived the quality of learning outcomes to be a significant predictor of satisfaction. The theoretical, instructional, and institutional implications for higher education institutions are discussed. Implications for practice or policy: • LMS satisfaction can be assessed through system quality, information quality, instructional quality, learning quality and interaction quality. • LMS satisfaction of all users can be enhanced by improving their perceived instructional quality of e-learning activities. • LMS satisfaction for infrequent and average frequency users can be enhanced by improving their perceived information quality of LMS content. • LMS satisfaction for frequent users can be enhanced by improving the learning quality or their perceived learning achievement from e-learning activities. Keywords: learning management system, next-generation digital learning environments, e- learning, e-learning quality, educational technology Introduction A learning management system (LMS) is an Internet-based platform used for delivering and administering e-learning courses (Coates, James, & Baldwin, 2005). The LMS is a standard information technology infrastructure of every higher education institution (Conde et al., 2014) and a critical component of students’ digital learning experiences. However, digital learning appears particularly challenging in the discipline of the arts. This is because artistic development is predominantly studio-based and technology integration tends to be addressed within digital production courses (Mroziak & Bowman, 2016). For artists who do not embrace artistic creation with digital production tools, learning technologies appear incongruent with their artistic creation processes (Snyder & Bulfin, 2007). Better understanding of arts students’ perceptions of their LMS experiences can facilitate the integration of LMS tools to support learning in the discipline of the arts. Students’ perceptions of LMS experiences have been evaluated with the model of information system success (IS model), which posits that users’ perceptions of three factors (system quality, information quality, and service quality) influences their perceptions of satisfaction, which in turns influences their usage (DeLone & McLean, 1992; William & Ephraim, 2003). The three-factor IS model has been statistically validated for LMS evaluation (e.g., Lin, 2007) and can be used as a theoretical model to Australasian Journal of Educational Technology, 2020, 36(3). 27 understand arts students’ LMS experiences. However, more recent LMS evaluation studies have found the need to examine the pedagogical functions of LMSs by extending it beyond the three quality factors (e.g., Mtebe & Raphael, 2018; Ozkan & Koseler, 2009). The IS model is also limited in examining user differences. Even though the model posits that satisfaction influences usage (DeLone & McLean, 1992), it does not address how users’ perceptions of quality and satisfaction are in turn influenced by usage. There is evidence that different degrees of usage influence how LMS users perceive satisfaction and the kinds of quality factors they value (Cheok & Wong, 2015; Horvat, Dobrota, Krsmanovic, & Cudanov, 2015). A more refined understanding of the relationships between LMS quality and satisfaction by usage differences is needed as studies of student LMS preferences indicate their desire for more customisation (Zanjani, Edwards, Nykvist, & Geva, 2017). Understanding these user differences among arts students can also help faculty better optimise LMS features for learning personalisation. This study therefore examined how the perceptions of LMS quality and satisfaction may differ among LMS users who perceive themselves as frequent, average frequency, and infrequent LMS users from a group of 376 arts college students in Singapore. For a more comprehensive LMS evaluation, this study drew upon the three factors of the IS model and added the factors of instructional quality, interaction quality, and learning quality to better capture the pedagogical aspects of e-learning that LMSs support (Ozkan & Koseler, 2009). It first validated a survey instrument created from these LMS quality factors. The different quality factors predicting LMS satisfaction for infrequent, average frequency, and frequent LMS uses were then examined with regression analyses. The implications of these findings for LMS evaluation with the IS model as well as the enhancement of LMS instructional practices in higher education institutions are discussed. Literature review LMS quality factors and satisfaction The IS model (DeLone & McLean, 1992) was originally developed to examine the impact of information systems in business organisations. It theorises that the system quality and information quality of information systems influence user satisfaction and its subsequent usage. A third factor related to the service quality of technical support was subsequently added to the model (William & Ephraim, 2003). By approaching the LMS as an information system, LMS studies have used the three factors of the IS model to examine student perceptions of LMS quality and how it relates to their LMS satisfaction. System quality System quality has been examined in LMS studies as technical factors such as system reliability, response time, ease of access to materials, design of navigation systems, and network speed (Lin, 2007; Pituch & Lee, 2006). These kinds of system-related factors were found to have moderate to strong positive correlations to learner satisfaction (Chen, Lin, & Kinshuk, 2008; Liaw, 2008; Ozkan & Koseler, 2009). Information quality Information quality has been defined as the currency, accuracy, and sufficiency of LMS content (Al- Samarraie, Teng, Alzahrani, & Alalwan, 2018; Lin, 2007; Shee & Wang, 2008). Lin (2007) and Al- Samarraie et al. (2018) found that information quality had stronger positive effects on satisfaction than system quality, whereas Shee and Wang (2008) found stronger positive effects for system quality. Service quality Service quality refers to the availability of technical support for the LMS. Although it was found to have significant positive correlation with LMS satisfaction, its influence on LMS satisfaction was found to be weaker than information quality and system quality (Lin, 2007; Ozkan & Koseler, 2009). William and Ephraim (2003) recommended that quality factors be added or adapted to improve contextualisation of the IS model. Since LMSs support e-learning, LMS studies have attempted to extend the three factors of the IS model with others that better describe students’ pedagogical experiences of LMSs. These factors include instructor quality (e.g., Ozkan & Koseler, 2009), peer interaction (e.g., Chen et al., 2008), instructional design of LMS materials or activities (e.g., Liaw, 2008), and the perceived usefulness or outcomes of LMS activities (e.g., Mtebe & Raphael, 2018). Three issues related to these factors need further consideration. Australasian Journal of Educational Technology, 2020, 36(3). 28 Distinguishing instructional quality from information quality Unlike business information systems, students engage instructionally with LMS content, and this may not be fully captured within the IS model’s conception of information quality as the currency, accuracy, and sufficiency of information (William & Ephraim, 2003). For example, Mtebe and Raphael (2018) redefined information quality as course quality for LMS evaluation, whereas Ozkan and Koseler (2009) included items for content interactivity and instructional design. How the LMS supports learning tasks indicates the instructional quality of students’ LMS experiences (Chen et al., 2008). It may be important to distinguish this from information quality because when evaluated as a separate factor, the design of e-learning materials and activities had a stronger positive influence on students’ satisfaction than LMS technical functionality (Chen et al., 2008; Liaw, 2008). Calls for more discerned considerations of technology-based pedagogies for artistic instruction (Mroziak & Bowman, 2016) also warrant a clearer distinction between the instructional and informational aspects of arts students’ LMS evaluation to be considered in this study. Interaction quality versus instructor quality E-learning systems drive instruction through instructor-student and student-student interactions (Moore, 2007), but the three original factors of the IS model did not cater for user interaction (DeLone & McLean, 1992). Some LMS studies focus on instructor quality in terms of their promptness with instructor-student interactions and found that it had positive effects on student satisfaction (Mtebe & Raphael, 2018; Ozkan & Koseler, 2009). However, e-learning is carried out through a community of inquiry among instructors and students (Garrison, 2007) and the quality of peer interaction was also found to influence learner satisfaction (Chen et al., 2008). An important part of artistic training involves critique and inquiry within artistic communities (Shreeve, Wareing, & Drew, 2008). Therefore, interaction quality that considers both peer and instructor interaction is important during LMS evaluation for students of the arts in this study. Learning quality versus perceived usefulness Students’ LMS experiences need to support their learning goals. Some LMS studies have examined this aspect through practical measures of perceived usefulness such as students’ perceptions of how LMSs contribute to their learning efficiency and academic performance (Mtebe & Raphael, 2018; Sun, Tsai, Finger, Chen, & Yeh, 2008). However, there is recognition of learners as social entities within e-learning systems, and affective aspects of learning such as motivation and enjoyment are also important (Ozkan & Koseler, 2009). Both the practical and motivational perceptions of LMS experiences relate positively with LMS satisfaction (Liaw, 2008; Ozkan & Koseler, 2009; Sun et al., 2008) and influence students’ continual satisfaction with e-learning (Al-Samarraie et al., 2018). As the notion of technology for learning may appear to challenge the existing learning conventions of arts students (Snyder & Bulfin, 2007), both practical and affective outcomes are important as an overall assessment of arts students’ perceived LMS learning quality in this study. The review shows that the factors of system quality, information quality, and service quality constitute the basic facets of LMS evaluation. Articulating the pedagogical dimensions of LMS evaluation with the factors of instructional quality, interaction quality, and learning quality may better capture the nuances that are relevant for the context of artistic learning. The plausibility of this six-factor structure needs to be validated. LMS quality and satisfaction by usage frequency The IS model adopts LMS satisfaction and use as two measures of system success where quality factors drive user satisfaction, which in turn influences system use (DeLone & McLean, 1992). Using this theorised process of system adoption, LMS studies have generally found positive relationships between LMS satisfaction and students’ intentions of use (Liaw, 2008; Lin, 2007). The IS model addresses initial system adoption, but it is also crucial to ensure positive student LMS perceptions beyond that (Al-Samarraie et al., 2018). Students’ usage frequencies, whether actual or perceived, can mediate their perceptions of quality and satisfaction (Cheok & Wong, 2015). There is sporadic evidence that frequent users of LMSs tend to have better perceptions of satisfaction and value peer interaction and cooperation more than low frequency users (Horvat et al., 2015). These findings suggest that students with different degrees of LMS use may have varying LMS perceptions. Such differences need to be better understood as LMS tools are increasingly being developed to support interoperability and user customisation (e.g., Pappas, 2014) in response to conceptions of next-generation digital learning environments (Brown, Dehoney, & Millichap, 2015). On Australasian Journal of Educational Technology, 2020, 36(3). 29 the other hand, students are also reflecting desires for more personalised LMS experiences (Becker et al., 2017; Zanjani et al., 2017). Better understanding of students’ varying LMS perceptions can be obtained by considering the IS model beyond initial system adoption. As the proliferation of technology increasingly shapes artists’ life experiences and work practices (Ferneding, 2007), better design of technology-based pedagogies can help mitigate the conception of technology-based pedagogies as antithesis to the material- based and studio-based practices of the discipline (Wilks, Cutcher, & Wilks, 2012). Developing LMS activities in consideration of arts students’ differing perceptions of LMS quality and satisfaction can better motivate their acceptance of technology as learning tools. From the preceding review, this study drew upon the conceptions of the IS model and proposes a revised perspective for examining arts students’ perceptions of LMS quality, satisfaction, and usage (see Figure 1). Figure 1. Evaluation of LMS quality, satisfaction, and use for arts students Research questions Based on the preceding review, two research questions were therefore examined in this study: (1) Can a six-factor model of LMS quality be statistically validated with respect to the perceptions of students of the arts? (2) How do arts students with different LMS usage frequencies perceive the relationships between LMS quality factors and LMS satisfaction? Methodology Subjects The subjects were 634 students enrolled in the visual and performing arts programs at a private arts college in Singapore. Students were invited to participate in a survey of their LMS learning experiences after a 7- week LMS trial period following which students’ feedback was taken into consideration for system selection. Australasian Journal of Educational Technology, 2020, 36(3). 30 As a formal ethics board for the institution had yet to be set up during the time of data collection, the survey administrators incorporated ethical compliance procedures as part of their data collection implementation. Prior to filling out the online survey, the objectives of study were explained to students and they were also invited to indicate informed consent of voluntary participation. A total of 376 students responded to the survey, constituting a response rate of 59.31%. About 30% of the respondents were performing arts majors (N = 114, 30.3%), whereas the rest were visual arts majors (N = 262, 67.7%). LMS interface Students participating in the study were assigned to trial one of three LMSs: Canvas (n = 123), Brightspace (n = 177), and Blackboard (n = 75), Regardless of the LMS they were assigned to, students had access to similar learning administration and learning applications. The learning administration functions include a modularised course interface, interaction via course announcements, class email, and access to assignment dropboxes whereas learning applications include online discussions, quizzes, and journals (see Figure 2). Sample of the Canvas interface with group discussions Sample of the Brightspace interface with reflection activities Sample of the Blackboard interface with quiz activities Figure 2. Sample features of LMSs Instrumentation and data collection Survey design – LMS quality factors A survey assessing students’ perceptions of LMS quality was designed based on the six factors articulated in Figure 1. The instrument of Ozkan and Koseler (2009) was used as a reference point for instrument Australasian Journal of Educational Technology, 2020, 36(3). 31 development because it had the closest fit to the quality constructs articulated for this study as per Figure 1. This instrument considered systems quality, information quality, and service quality. It also considered learner perspectives in terms of interaction and learning enjoyment. The items of this survey were adapted and redesigned to better reflect current technological developments, product specifications for the LMSs students used, as well as the pedagogical practices relevant for the discipline of the arts. System quality was assessed with 13 questions related to technical reliability, support, visual design, and ease of use and navigation (e.g., “I did not face system errors in this LMS”, “I can find my way around this LMS easily”). To capture the spirit of interoperability as espoused of next-generation LMSs (Becker et al., 2017) and supported within the LMS functions, questions related to access through different devices were added (e.g., “The layout is easy to navigate on mobile devices”). Information quality for this study was assessed with seven questions related to the clarity, sequencing, timeliness, and usefulness of LMS content information (e.g., “The LMS course site is frequently updated with content”). Unlike Ozkan and Koseler (2009), questions related to instructional design were considered within instructional quality to avoid confounding the two variables. Service quality was assessed with four questions related to user support. Besides technical support (e.g., “The learning service to support issues on this LMS is good enough”), it also assessed if the LMS design and instructions provided adequate directions for help-seeking (e.g., “I know what to do if I face learning difficulties in this LMS”). Teaching students to be artists who are capable of independent thought as well as critical engagement within their artistic community is important in the education of the arts (Shreeve et al., 2008). Therefore, the six questions for interaction factors go beyond the considerations of instructor quality by Ozkan and Koseler (2009) to include interaction with instructors (e.g., “I think communicating with my lecturer through this LMS is important”) and peers (e.g., “The LMS provides a good environment to discuss and collaborate with my peers”). Correspondingly, the nine questions for instructional quality were designed to articulate how LMSs support critical pedagogies for arts students, such as independent learning (e.g., “This LMS is valuable for improving learning on my own”), collaborative learning (e.g., “Learning from my peers”), and development of work skills (Kneale, 2009) such as project management (e.g., “This LMS is valuable for managing project collaborations”). Finally, learning quality was assessed with nine questions in terms of the perceived usefulness of their LMS experiences with respect to learning and performance (e.g., “I find that this LMS is a good educational portal that improves my learning”) as well as engagement and motivation (e.g., “This LMS has the potential to make the learning process more engaging and motivating”). The 48 questions related to the LMS quality dimensions were rated on a 5-point Likert scale ranging from 1 – strongly disagree to 5 – strongly agree. Survey design – LMS satisfaction Students’ perceptions of LMS satisfaction was assessed through a 5-point Likert scale ranging from 1 – strongly disagree to 5 – strongly agree. As per the design of Ozkan and Koseler (2009), students’ satisfaction was measured in terms of overall LMS satisfaction and overall LMS effectiveness. A third question pertaining to their willingness to recommend the LMS was added because the survey was implemented as part of an LMS trial and students’ willingness to recommend the LMS constituted an important element of satisfaction within this context. Survey design – Usage frequency As the study consent from students did not include use of their LMS usage statistics, the closest proxy of perceived usage frequency was designed as per Ozkan and Koseler’s (2009) instrument. Students’ perceived LMS usage frequency was rated through the question, “How often did you visit the LMS?” where 0 – Not sure what this is, 1 – Have not used it before, 2 – Less than five times, 3 – Once a week, 4 – Two or more times a week, and 5 – At least once a day. Australasian Journal of Educational Technology, 2020, 36(3). 32 Content review and reliability Content review of the survey was conducted by the faculty and staff of the institution’s educational technology unit as well as an external university faculty in educational technology. High reliabilities were obtained for the constructs: system quality (α = 0.95), information quality (α = 0.96), service quality (α = 0.85), interaction quality (α = 0.94), instructional quality (α = 0.91), learning quality (α = 0.94), and LMS satisfaction (α = 0.87). Data analysis The first research question was answered through an exploratory factor analysis to understand if the proposed factors could be supported by the underlying item structures (Thorndike, 2005). The sample size of 376 satisfied the minimum requirement of five items per variable recommended by (Hair, Black, Babin, Anderson, & Tatham, 2010). The Kaiser-Meyer-Olkin (KMO) measure of sampling adequacy was 0.97, and the Bartlett’s Test of Sphericity was significant (x2 (1128) = 17217.54, p < 0.0001). Maximum likelihood analysis with varimax rotation was used to extract the factors. Similar procedures were also used to establish the factor of LMS satisfaction with KMO of 0.74 and significant Bartlett’s Test of Sphericity (x2 (3) = 566.99, p < 0.0001). Following validation of the factor structure for quality dimensions, the overall trends related to students’ satisfaction with the LMS and their ratings of each quality dimension were analysed with descriptive statistics. Three groups perceiving themselves as infrequent, average, and frequent users were identified from students’ survey responses. The second research question was answered by first examining evidences of relationships between the quality factors and user satisfaction through Pearson’s correlation. Stepwise linear regression was carried out after ascertaining that there were significant correlations among these variables. Regression models were thus created for the three groups of students by their perceived usage frequency. To account for possible effects from the variation of LMSs used by the study participants, each LMS type was converted into a dummy variable and included with the quality dimensions as predictors of user satisfaction in the regression model. This analysis found that all the three LMS types did not significantly predict user satisfaction for the three groups of students. The LMS variables were removed and the regression was carried out for each group of students with user satisfaction as the dependent variable and the quality dimensions as the independent variables. Findings Research question 1 – Validation of quality factors Using exploratory factor analysis, the four items for service quality had factor loadings below 0.50 and were removed, as per Hair et al.’s (2010) recommendation for factor loadings of practical significance. The model yielded six constructs, but the sixth construct comprised two cross-loaded items with factor loadings below 0.50. This factor was removed, and a five-factor model was finalised. Five constructs corresponding to the items designed for system quality, information quality, interaction quality, learning quality, and instructional quality were obtained; 66.47% of the variance was explained by the final model (see Table 1). Australasian Journal of Educational Technology, 2020, 36(3). 33 Table 1 Factor loadings from exploratory factor analysis Factor 1 – System quality Factor 2 – Information quality Factor 3 – Interaction quality Factor 4 – Instructional quality Factor 5 – Learning quality Factor 1 – System quality 1. The layout is easy to navigate on PC/Notebooks .64 2. The layout is easy to navigate on mobile devices .62 3. The most important information on the screen is placed in areas most likely to attract my attention .62 4. Texts and graphics are easy to understand .68 5. Fonts (style, color, saturation) are easy to read on-screen .62 6. I can find my way around this LMS easily (just a few clicks to where I want to go) .75 7. Information in help and documentation features are clearly written .73 8. I can find required information easily on this LMS .73 9. I am able to easily return to the most logical point where I left off in the course site .70 10. I did not face system errors in this LMS .58 11. I was able to access pages within reasonable time .71 12. I was able to access the content easily from any device (tablet, notebook, iOS, Android) .66 13. I generally did not encounter any technical problems to access this LMS .65 Factor 2 – Information quality 14. The LMS clearly states the module learning outcomes/goals of the course .58 15. LMS tools are appropriate to support my learning process .67 16. The LMS design is effective to help understanding of content for this module .61 17. LMS tools (learning units and modules) are self-contained enough to access out of sequence without becoming confused .62 18. LMS tools provide a streamlined access to supplementary materials .63 19. The LMS course site is frequently updated with content .65 20. LMS content is integral to my learning of this module .61 Australasian Journal of Educational Technology, 2020, 36(3). 34 Factor 3 – Interaction quality 21. The LMS provides a good environment to discuss and collaborate with my peers .63 22. LMS discussions are enjoyable .62 23. I think communicating with my lecturer through this LMS is important .69 24. I think communicating with my peers through this LMS is important .77 25. The LMS supports interactivity between learners by discussions, forums etc. .68 Factor 4 – Instructional quality 26. This LMS is valuable for learning from my peers .60 27. This LMS is valuable for saving me time .51 28. This LMS is valuable for assessing material any time from any location .50 29. This LMS is valuable for posting and replying to discussions .66 30. This LMS is valuable for taking tests .69 31. This LMS is valuable for submitting assignments .69 32. This LMS is valuable for managing project collaborations .72 33. This LMS is valuable for communicating with peer students .62 Factor 5 – Learning quality 34. Learning activities foster instructor-student, or student-student interaction .55 35. The course site provides me with opportunities to learn from peers .59 36. This LMS has the potential to make the learning process more engaging and motivating .75 37. Tools available within this LMS facilitate my learning by engaging me with course content .76 38. I am able to learn through several modalities in this LMS (audio, visual and text-based formats) .77 39. The online environment is conducive and enjoyable for learning overall .79 40. I find that this LMS is a good educational portal that improves my learning .84 41. This LMS helped me to become more familiar with the course content .81 42. Technologies are used creatively in ways that go beyond traditional ways of learning .72 Australasian Journal of Educational Technology, 2020, 36(3). 35 The factor structure of the items underlying LMS satisfaction was also validated with 69.64% of the variance explained by the model (see Table 2). Table 2 Factor loadings for items of LMS satisfaction Items Factor 1 – LMS satisfaction 1. Overall, I am satisfied with this LMS .88 2. Overall, this LMS helps me to manage my learning more effectively .81 3. Overall, I will recommend this LMS to students in my program .81 Overall trends Students were generally satisfied with the LMS with respect to the five quality dimensions. On a scale of 1 to 5, average ratings were above 3 for the following: LMS satisfaction (M = 3.25, SD = 0.85), system quality (M = 3.25, SD = 0.77), information quality (M = 3.20, SD = 0.84), interaction quality (M = 3.12, SD = 0.88), instructional quality (M = 3.26, SD = 0.79), and learning quality (M = 3.08, SD = 0.87). Independent sample t tests found no significant differences between visual and performing arts students with respect to their perceptions of overall satisfaction or the five LMS quality dimensions. Influence of frequency of use There were three discernible groups of LMS users by their reported frequency of use. About 35% of the students perceived themselves to be infrequent users who reported using the LMS only five times or less (N = 132, 35.1%), about 34% perceived themselves to be average frequency users who used it at once a week (N = 127, 33.8%), whereas the rest of the students perceived themselves to be frequent users who used it more than once a week (N = 117, 31.1%). One-way ANOVA found significant differences among these three groups of LMS users in terms of perceptions of overall satisfaction (F(2,373) = 19.93, p < 0.0001), system quality (F(2,373) = 18.20, p < 0.0001), information quality (F(2,373) = 27.64, p < 0.0001), interaction quality (F(2,373) = 17.25, p < 0.0001), instructional quality (F(2,373) = 9.90, p < 0.0001), and learning quality (F(2,373) = 13.28, p < 0.0001). Scheffe’s post-hoc tests found that frequent users who perceived themselves as accessing the LMS more than once a week had the highest overall satisfaction among the three groups, whereas those perceiving infrequent use had the lowest satisfaction (Frequent: M = 3.60, Average: M = 3.25, Infrequent: M = 2.95). Similar patterns were observed for system quality (Frequent: M = 3.55, Average: M = 3.24, Infrequent: M = 2.99), information quality (Frequent: M = 3.62, Average: M = 3.14, Infrequent: M = 2.89), and instructional quality (Frequent: M = 3.56, Average: M = 3.27, Infrequent: M = 2.99). Students perceiving frequent use also perceived significantly higher interaction quality (Frequent: M = 3.40, Average: M = 3.09, Infrequent: M = 2.90) and learning quality than those perceiving average or infrequent use (Frequent: M = 3.40, Average: M = 3.01, Infrequent: M = 2.86), whereas there were no significant differences between those perceiving average frequency and infrequent use with respect to these two quality dimensions. Research question 2 – Differences in perceptions of LMS quality and LMS satisfaction Table 3 shows the correlations among the LMS quality dimensions and perceived LMS satisfaction. Table 3 Correlations Perceived satisfaction System quality Information quality Interaction quality Instructional quality Learning quality Perceived LMS satisfaction 1 .65** .65** .57** .67** .50** System quality .65** 1 .81** .70** .60** .57** Information quality .65** .81** 1 .65** .60** .59** Interaction quality .57** .69** .65** 1 .65** .59** Instructional quality .67** .60** .60** .65** 1 .49** Learning quality .50** .57** .59** .59** .49** 1 N = 376. ** p < 0.01 Perceived LMS satisfaction had moderate positive correlation with all the five LMS quality dimensions. Linear regressions were therefore undertaken to understand how LMS quality dimensions contributed to Australasian Journal of Educational Technology, 2020, 36(3). 36 the perceived LMS satisfaction of students who perceived infrequent, average frequency, and frequent use (see Table 4). Table 4 Stepwise regression models by usage frequency Model Predictors B Std. Error Beta Significance Adjusted R2 Infrequent users 1 (Constant) .68 .21 ** .48 Instructional quality .76 .07 .70 *** 2 (Constant) .36 .22 n.s. .53 Instructional quality .61 .08 .56 *** Information quality .26 .08 .25 ** Average frequency users 1 (Constant) .68 .21 ** .48 Instructional quality .76 .07 .70 *** 2 (Constant) 1.24 .23 *** .40 System quality .45 .08 .46 *** Instructional quality .29 .08 .30 *** 3 (Constant) .79 .23 ** .48 System quality .27 .10 .27 ** Instructional quality .24 .08 .25 ** Information quality .26 .09 .28 ** Frequent users 1 (Constant) .57 .27 * .52 System quality .85 .08 .73 *** 2 (Constant) .04 .29 n.s. .58 System quality .64 .09 .55 *** Instructional quality .36 .09 .31 *** 3 (Constant) -.10 .29 n.s. .59 System quality .56 .10 .47 *** Instructional quality .34 .09 .29 *** Learning quality .15 .07 .15 * *p < 0.05; **p < 0.01; ***p < 0.0001; n.s. = not significant Students perceiving infrequent use appeared to give prominence to instructional quality, as indicated by the 48% variance explained by this variable in the regression model. Information quality was also a significant predictor for this group that added another 5% of variance explained to the regression model. For those perceiving average frequency use, system quality appeared to be a critical factor, accounting for 40% of the model variance explained in the regression model. Another 8% of their regression model was explained by instructional quality and information quality. While those perceiving high frequency use also gave prominence to system quality, which explained 52% of the variance of their regression model, instructional quality and learning quality added another 7% to the variance explained. Discussion This study has validated a five-factor model to describe arts students’ perceptions of LMS quality in terms of system, information, and pedagogical dimensions. It has also examined the different student perceptions of quality and satisfaction among students perceiving frequent, average frequency, and infrequent use of the LMS. The following are several implications of this study. Theoretical implications The results of the factor analysis indicate that a five-factor IS model can be statistically supported with respect to the perceptions of arts students. This study attests to the recommendation of the IS model as one needing contextualisation (Mtebe & Raphael, 2018; William & Ephraim, 2003) and the addition of the factors of interaction quality, instructional quality, and learning quality improves the pedagogical evaluation of LMS quality with respect to students of the arts. Although service quality was validated within Australasian Journal of Educational Technology, 2020, 36(3). 37 the factor structures of some studies (Lin, 2007; Mtebe & Raphael, 2018; Ozkan & Koseler, 2009), it was not so in this study. One reason could be that items such as “I know what to do if I face learning difficulties in this LMS” may not clearly indicate the assessment of service quality. Another reason could be that students were undergoing a 7-week LMS trial and may not have encountered the need for service support. These results may also indicate that students’ perceptions of IS quality factors vary according to discipline and context because in another study, it was the factor of perceived usefulness that could not be validated with university students in Tanzania (see Mtebe & Raphael, 2018). Even though the eclectic adoption of quality variables is common in LMS studies (e.g., Al-Samarraie et al., 2018; Asoodar, Vaezi, & Izanloo, 2016), the creators of the IS model recommend that contextualisation efforts be guided by principles underlying factor selection (William & Ephraim, 2003). As in other studies (e.g., Mtebe & Raphael, 2018; Ozkan & Koseler, 2009), this study has sought to enhance the pedagogical dimensions of LMS quality factors. The mixed findings among various LMS studies suggest that more refined pedagogical principles for factor selection might be needed to improve the comparability among LMS evaluation studies. The study also demonstrates how the IS model can be used to explicate differences among system users. When LMS usage was perceived as infrequent, users were satisfied when they undertook learning activities of instructional value and had fuss-free access to course information. Students perceiving average frequency usage were satisfied when they experienced quality instructional activities with a technically reliable LMS that had a clearly organised information structure, whereas those perceiving frequent usage not only required technical reliability and valuable instructional activities but also desired to obtain positive learning outcomes from them. The IS model’s proposition that satisfaction is influenced by user perceptions of system and information quality (DeLone & McLean, 1992) appears more congruent to the perceptions of the average frequency users in this study. Like those of Cheok and Wong (2015), these findings indicate the mediating effects of usage frequency between students’ perceptions of LMS quality and satisfaction. Such effects need to be considered when contextualising the IS model for LMS evaluation. Instructional implications The findings from the regression analyses show that instructional quality significantly predicted LMS satisfaction for all the study participants regardless of their perceived LMS usage frequencies and type of LMS used. Even though there appears to be tensions regarding the role of technology in artistic practice (Wilks et al., 2012), the study results indicate that students of the arts may have different perceptions about using technology for learning versus using technology for artistic practice. With respect to the discipline of the arts, these findings refute the notion of higher education students approaching LMSs as learning administration systems rather than learning systems (Becker et al., 2017). Although university lecturers tend to value LMSs for content sharing and course announcements (Lonn & Teasley, 2009), these findings suggest the need for more deliberate consideration of LMS strategies to foster instructional dimensions such as collaboration and intentionality that are important for meaningful learning (e.g., Koh, 2017). Analysis of student behaviours in massive open online courses found low completion rates with high student drop-offs occurring between the first two modules of a course (Perna et al., 2014). The study results suggest that more deliberate planning of instructional strategies could enhance students’ perceptions of instructional quality from online courses, which could in turn enhance their LMS satisfaction. This could be important for enhancing student retention during online learning. It is also important to enhance information quality as this factor positively predicted LMS satisfaction for students who perceived themselves as having infrequent or average frequency LMS usage. This relates to organising materials for quick access and minimising unnecessary tools and links on LMS course sites (Zanjani et al., 2017). Although Horvat et al. (2015) found that students with high LMS usage frequencies valued peer interaction, it was learning quality rather than interaction quality that predicted satisfaction in the regression model for students perceiving high LMS usage in this study. The use of interactive tools such as quizzes and discussions can help enhance students’ preparation for class (Lonn & Teasley, 2009). These could be ways of enhancing students’ sense of learning achievement and hence their learning quality. This strategy is particularly relevant for students perceiving high LMS usage. Australasian Journal of Educational Technology, 2020, 36(3). 38 Institutional implications Similar to the findings of Horvat et al. (2015), this study found that students perceiving more frequent LMS use had higher levels of LMS satisfaction. On the other hand, there were no significant differences between the learning quality perceptions of students perceiving average frequency and low frequency usage. These results suggest that LMS satisfaction as well as a sense of learning quality achievement with LMSs can be deepened through more usage opportunities. Therefore, sufficient time and opportunities must be given for students to attain system familiarisation and to develop clearer connections between LMS activities and their learning outcomes within institutional curriculum designs. Correspondingly, there is a need for faculty preparation as the study found that instructional quality predicts satisfaction for all student groups. LMS use is highly influenced by how lecturers design their instructional activities. Kirkwood and Price (2014) found that lecturers used technology tools to support their existing pedagogical strategies, which were largely administrative (Dahlstrom, Brooks, & Bichsel, 2014). Supporting faculty with development workshops to build their pedagogical knowledge beyond administrative uses of LMSs is one way to enhance students’ perceptions of instructional quality through lesson design. Limitations and future research This study has several limitations, which can also serve as areas for future research. Firstly, the dimensions of LMS quality were validated with a sample of students of the arts through exploratory factor analysis. The applicability of the survey to other higher education disciplines can be further examined. As the sample size was insufficient for further construct validation through confirmatory factor analysis, this can also be carried out in future studies. Through this process, the validity of service quality for LMS evaluation can be further examined. Secondly, the study used perceived usage frequency as a proxy for system use. Future studies could examine the differences among student groups using actual usage frequencies to validate the findings of this study. Thirdly, the study did not consider how factors such as study load or personal circumstances of students could have influenced their LMS access. Consideration of these factors in future studies through student interviews to triangulate the survey findings could provide a more comprehensive view of LMS quality and usage. Finally, this study has focused on the LMS learning practices of students, but it is recognised that faculty’s pedagogical practices can influence students’ LMS use. Examination of the LMS perceptions of the institution’s faculty was outside the scope of this study. Furthermore, the 7- week LMS trial period could have limited how the lecturers could have explored the system features. Future studies could examine students’ LMS practices in light of faculty pedagogical practices within a full semester to derive more comprehensive insights. Conclusion The study results show that the notion of LMSs as learning systems is already within the perceptions of arts students. These results are encouraging, given that students were participating in a 7-week LMS trial. With an average usage period of 8 years (Dahlstrom et al., 2014), LMS upgrades are a continual feature within the infrastructural plans of higher education institutions, and there is no doubt that upgrades for next- generation digital learning environments that are capable of supporting personalised and community-based learning through learning analytics and interoperable tools (Brown et al., 2015) are underway. Enhancing students’ pedagogical interactions with the LMS is one way of ensuring congruence between the capacities and use of LMSs. A more deliberate strategy than hardware improvement is needed to actualise the full pedagogical potential of digital learning environments. More consideration of the differences among students’ LMS quality perceptions can facilitate the design of pedagogical practices to enhance student satisfaction. References Al-Samarraie, H., Teng, B. K., Alzahrani, A. I., & Alalwan, N. (2018). E-learning continuance satisfaction in higher education: A unified perspective from instructors and students. Studies in Higher Education, 43(11), 2003–2019. https://doi.org/10.1080/03075079.2017.1298088 Asoodar, M., Vaezi, S., & Izanloo, B. (2016). Framework to improve e-learner satisfaction and further strengthen e-learning implementation. Computers in Human Behavior, 63, 704–716. https://doi.org/10.1016/j.chb.2016.05.060 https://doi.org/10.1080/03075079.2017.1298088 https://doi.org/10.1016/j.chb.2016.05.060 Australasian Journal of Educational Technology, 2020, 36(3). 39 Becker, S. A., Cummins, M., Davis, A., Freeman, A., Hall, C. G., & Ananthanarayanan, V. (2017). Horizon Report: 2017 Higher Education Edition. Austin, TX: New Media Consortium. Retrieved from https://library.educause.edu/-/media/files/library/2017/2/2017horizonreporthe.pdf Brown, M., Dehoney, J., & Millichap, N. (2015). The next generation digital learning environment: A report on research. Louisville, CO: EDUCAUSE. Retrieved from https://library.educause.edu/- /media/files/library/2015/4/eli3035-pdf.pdf Chen, N. S., Lin, K. M., & Kinshuk. (2008). Analysing users’ satisfaction with e‐learning using a negative critical incidents approach. Innovations in Education and Teaching International, 45(2), 115–126. https://doi.org/10.1080/14703290801950286 Cheok, M. L., & Wong, S. L. (2015). Predictors of e-Learning satisfaction in teaching and learning for school teachers: A literature review. International Journal of Instruction, 8(1), 75–90. Retrieved from http://www.e-iji.net/dosyalar/iji_2015_1_6.pdf Coates, H., James, R., & Baldwin, G. (2005). A critical examination of the effects of learning management systems on university teaching and learning. Tertiary Education and Management, 11, 19–36. https://doi.org/10.1007/s11233-004-3567-9 Conde, M. Á., García-Peñalvo, F. J., Rodríguez-Conde, M. J., Alier, M., Casany, M. J., & Piguillem, J. (2014). An evolving learning management system for new educational environments using 2.0 tools. Interactive Learning Environments, 22(2), 188–204. https://doi.org/10.1080/10494820.2012.745433 Dahlstrom, E., Brooks, D. C., & Bichsel, J. (2014). The current ecosystem of learning management systems in higher education: Student, faculty, and IT perspectives. Louisville, CO: EDUCAUSE. Retrieved from https://library.educause.edu/-/media/files/library/2014/9/ers1414-pdf.pdf DeLone, W. H., & McLean, E. R. (1992). Information systems success: The quest for the dependent variable. Information Systems Research, 3(1), 60–95. https://doi.org/10.1287/isre.3.1.60 Ferneding, K. (2007). Understanding the message of the medium: Media technologies as an aesthetic. In L. Bresler (Ed.), International handbook of research in arts education (pp. 1331–1354). Dordrecht, The Netherlands: Springer. Garrison, D. R. (2007). Online community of inquiry review: Social, cognitive, and teaching presence issues. Journal of Asynchronous Learning Networks, 11(1), 61–72. https://doi.org/10.24059/olj.v11i1.1737 Hair, J. F., Jr. , Black, W. C., Babin, B. J., Anderson, R. E., & Tatham, R. L. (2010). Factor analysis. In Multivariate data analysis: A global perspective (7th ed., pp. 91–151). Upper Saddle River, NJ: Pearson Education. Horvat, A., Dobrota, M., Krsmanovic, M., & Cudanov, M. (2015). Student perception of Moodle learning management system: A satisfaction and significance analysis. Interactive Learning Environments, 23(4), 515–527. https://doi.org/10.1080/10494820.2013.788033 Kirkwood, A., & Price, L. (2014). Technology-enhanced learning and teaching in higher education: What is ‘enhanced’ and how do we know? A critical literature review. Learning, Media and Technology, 39(1 ), 6–36. https://doi.org/10.1080/17439884.2013.770404 Kneale, P. (2009). Teaching and learning for employability: Knowledge is not the only outcome. In H. Fry (Ed.), A handbook for teaching and learning in higher education (pp. 99–112). New York, NY: Routledge. Koh, J. H. L. (2017). Designing and integrating reusable learning objects for meaningful learning: Cases from a graduate programme. Australasian Journal of Educational Technology, 33(5), 136–151. https://doi.org/10.14742/ajet.3072 Liaw, S. S. (2008). Investigating students’ perceived satisfaction, behavioral intention, and effectiveness of e-learning: A case study of the Blackboard system. Computers & Education, 51(2), 864–873. https://doi.org/10.1016/j.compedu.2007.09.005 Lin, H. F. (2007). Measuring online learning systems success: Applying the updated DeLone and McLean model. Cyberpsychology & Behavior, 10(6), 817–820. https://doi.org/10.1089/cpb.2007.9948 Lonn, S., & Teasley, S. D. (2009). Saving time or innovating practice: Investigating perceptions and uses of Learning Management Systems. Computers & Education, 53(3), 686–694. https://doi.org/10.1016/j.compedu.2009.04.008 Moore, M. G. (2007). The theory of transactional distance. In M. G.Moore (Ed.), The handbook of distance education (2nd ed., pp. 89–108). Mahwah, NJ: Lawrence Erlbaum Associates. Mroziak, J., & Bowman, J. (2016). Music TPACK in higher education. In M. C. Herring, M. J. Koehler, & P. Mishra (Eds.), Handbook of technological pedagogical content knowledge (TPACK) for educators (2nd ed., pp. 285–296). New York, NY: Routledge. https://library.educause.edu/-/media/files/library/2017/2/2017horizonreporthe.pdf https://library.educause.edu/-/media/files/library/2015/4/eli3035-pdf.pdf https://library.educause.edu/-/media/files/library/2015/4/eli3035-pdf.pdf https://doi.org/10.1080/14703290801950286 http://www.e-iji.net/dosyalar/iji_2015_1_6.pdf https://doi.org/10.1007/s11233-004-3567-9 https://doi.org/10.1080/10494820.2012.745433 https://library.educause.edu/-/media/files/library/2014/9/ers1414-pdf.pdf https://doi.org/10.1287/isre.3.1.60 https://doi.org/10.24059/olj.v11i1.1737 https://doi.org/10.1080/10494820.2013.788033 https://doi.org/10.1080/17439884.2013.770404 https://doi.org/10.14742/ajet.3072 https://doi.org/10.1016/j.compedu.2007.09.005 https://doi.org/10.1089/cpb.2007.9948 https://doi.org/10.1016/j.compedu.2009.04.008 Australasian Journal of Educational Technology, 2020, 36(3). 40 Mtebe, J. S., & Raphael, C. (2018). Key factors in learners’ satisfaction with the e-learning system at the University of Dar es Salaam, Tanzania. Australasian Journal of Educational Technology, 34(4), 107– 122. https://doi.org/10.14742/ajet.2993 Ozkan, S., & Koseler, R. (2009). Multi-dimensional students’ evaluation of e-learning systems in the higher education context: An empirical investigation. Computers & Education, 53(4), 1285–1296. https://doi.org/10.1016/j.compedu.2009.06.011 Pappas, C. (2014, January 18). The 20 best learning management systems (2019 update). eLearning Industry. Retrieved from https://elearningindustry.com/the-20-best-learning-management-systems Perna, L. W., Ruby, A., Boruch, R. F., Wang, N., Scull, J., Ahmad, S., & Evans, C. (2014). Moving through MOOCs: Understanding the progression of users in massive open online courses. Educational Researcher, 43(9), 421–432. https://doi.org/10.3102/0013189X14562423 Pituch, K. A., & Lee, Y.-K. (2006). The influence of system characteristics on e-learning use. Computers & Education, 47(2), 222–244. https://doi.org/10.1016/j.compedu.2004.10.007 Shee, D. Y., & Wang, Y. S. (2008). Multi-criteria evaluation of the web-based e-learning system: A methodology based on learner satisfaction and its applications. Computers & Education, 50(3), 894– 905. https://doi.org/10.1016/j.compedu.2006.09.005 Shreeve, A., Wareing, S., & Drew, L. (2008). Key aspects of teaching and learning in the visual arts. In H. Fry, S. Ketteridge, & S. Marshall (Eds.), A handbook for teaching and learning in higher education (3rd ed., pp. 345–362). London, United Kingdom: Kogan Page. Snyder, I., & Bulfin, S. (2007). Digital literacy: What it means for arts education. In L. Bresler (Ed.), International handbook of research in arts education (vol. 16, pp. 1297–1310). Dordrecht, The Netherlands: Springer. Sun, P. C., Tsai, R. J., Finger, G., Chen, Y. Y., & Yeh, D. (2008). What drives a successful e-Learning? An empirical investigation of the critical factors influencing learner satisfaction. Computers & Education, 50(4), 1183–1202. https://doi.org/10.1016/j.compedu.2006.11.007 Thorndike, R. M. (2005). Measurement and evaluation in psychology and education. Upper Saddle River, NJ: Pearson Prentice Hall. Wilks, J., Cutcher, A., & Wilks, S. (2012). Digital technology in the visual arts classroom: An [un] easy partnership. Studies in Art Education, 54(1), 54–65. https://doi.org/10.1080/00393541.2012.11518879 William, H. D., & Ephraim, R. M. (2003). The DeLone and McLean Model of Information Systems Success: A ten-year update. Journal of Management Information Systems, 19(4), 9–30. https://doi.org/10.1080/07421222.2003.11045748 Zanjani, N., Edwards, S. L., Nykvist, S., & Geva, S. (2017). The important elements of LMS design that affect user engagement with e-learning tools within LMSs in the higher education sector. Australasian Journal of Educational Technology, 33(1), 19–31. https://doi.org/10.14742/ajet.2938 Corresponding author: Joyce Hwee Ling Koh, joyce.koh@otago.ac.nz Copyright: Articles published in the Australasian Journal of Educational Technology (AJET) are available under Creative Commons Attribution Non-Commercial No Derivatives Licence (CC BY- NC-ND 4.0). Authors retain copyright in their work and grant AJET right of first publication under CC BY-NC-ND 4.0. Please cite as: Koh, J. H. L., & Kan, R. Y. P. (2020). Perceptions of learning management system quality, satisfaction, and usage: Differences among students of the arts. Australasian Journal of Educational Technology, 36(3), 26–40. https://doi.org/10.14742/ajet.5187 https://doi.org/10.14742/ajet.2993 https://doi.org/10.1016/j.compedu.2009.06.011 https://elearningindustry.com/the-20-best-learning-management-systems https://doi.org/10.3102/0013189X14562423 https://doi.org/10.1016/j.compedu.2004.10.007 https://doi.org/10.1016/j.compedu.2006.09.005 https://doi.org/10.1016/j.compedu.2006.11.007 https://doi.org/10.1080/00393541.2012.11518879 https://doi.org/10.1080/07421222.2003.11045748 https://doi.org/10.14742/ajet.2938 mailto:joyce.koh@otago.ac.nz https://creativecommons.org/licenses/by-nc-nd/4.0/ https://creativecommons.org/licenses/by-nc-nd/4.0/ https://doi.org/10.14742/ajet.5187 Introduction Literature review LMS quality factors and satisfaction System quality Information quality Service quality Distinguishing instructional quality from information quality Interaction quality versus instructor quality Learning quality versus perceived usefulness LMS quality and satisfaction by usage frequency Research questions Methodology Subjects LMS interface Instrumentation and data collection Survey design – LMS quality factors Survey design – LMS satisfaction Survey design – Usage frequency Content review and reliability Data analysis Findings Research question 1 – Validation of quality factors Overall trends Influence of frequency of use Research question 2 – Differences in perceptions of LMS quality and LMS satisfaction Discussion Theoretical implications Instructional implications Institutional implications Limitations and future research Conclusion References