Australasian Journal of Educational Technology, 2020, 36(6). 89 Critical success factors for implementing learning analytics in higher education: A mixed-method inquiry Jo-Anne Clark Griffith University, Australia Yulin Liu Queensland University of Technology, Australia Pedro Isaias The University of New South Wales, Australia Critical success factors (CSFs) have been around since the late 1970s and have been used extensively in information systems implementations. CSFs provide a comprehensive understanding of the multiple layers and dimensions of implementation success. In the specific context of learning analytics (LA), identifying CSFs can maximise the possibilities of an effective implementation and harness the value of converting data into actionable information. This paper proposes a framework that aims to identify and explore the CSFs for the implementation of LA within the higher education sector by examining the viewpoints of higher education professionals. To obtain a rounded insight into stakeholders’ perceptions, we conducted a mixed-method inquiry with factor analysis, profile analysis and thematic analysis of both quantitative and qualitative data collected with an online questionnaire from an international sample. The responses validate five CSFs of LA implementation: strategy and policy at organisational level, information technological readiness, performance and impact evaluation, people’s skills and expertise and data quality. Results also disclose diverse views about the CSFs’ priorities and the associated difficulties and achievements. Implications for practice or policy: • Higher education practitioners should consider CSFs for implementing an LA initiative successfully. • This study validates five dimensions of the CSFs of implementing LA in higher education. • The validated framework enumerates several factors in each of the main dimensions for achieving optimum results. • Stakeholders have diverse opinions about the priorities of CSFs, particularly in organisational commitment, data quality and human capital. Keywords: learning analytics, critical success factors, information systems, mixed methods, technology adoption Introduction The term critical success factors (CSFs), originally developed by the Massachusetts Institute of Technology research team (Rockart, 1979), refers to those factors that are critical to the success of any organisation. If the objectives associated with the factors are not achieved, the organisation will fail – perhaps on a catastrophic scale (Rockart, 1979). A simple definition coined by Freund (1988, p. 20) sees CSFs as “those things that must be done if a company is to be successful”. Appearing widely in the literature in the 1980s, studies have investigated why some organisations were more successful than others. CSF research focused on investigating the components that led to the success of the organisations under study (Ingram et al., 2000). When applied to information systems, CSFs are the conditions that need to be met to assure a system’s success (Poon & Wagner, 2001). In a 2013 study, while in the context of enterprise resource planning implementations, Ram et al. noted that there are few studies that investigate the effect of CSFs on system implementation success as well as organisational performance improvements. Identifying the CSFs in Australasian Journal of Educational Technology, 2020, 36(6). 90 information systems implementation has been the key to whether a system succeeds or not (Miranda et al., 2014). In the context of higher education, there are a number of examples of studies using CSFs for the application of technology (Alhabeeb & Rowley, 2018; Alrasheedi et al., 2015; Isaias et al., 2009; McGill et al., 2014; Miranda et al., 2014; Miranda et al., 2017). The system under focus in this study was the learning analytics (LA) system. LA is “the measurement, collection, analysis and reporting of data about learners and their contexts, for [the] purposes of understanding and optimizing learning and the environments in which it occurs” (Long & Siemens, 2011, p. 34). We are aware that LA is closely related to other concepts, such as educational data mining, data‐driven education, big data in education and educational data science, which have evolved enormously in various education settings in the last decade (Romero & Ventura, 2020). We refer to LA systems throughout this article from the perspective of information systems research. This study aimed to examine the CSFs key that are key to LA systems implementation; the findings could shed light on how to successfully implement these systems within the higher education context. Theoretical background LA in higher education Although the aim of universities is to be innovative and dynamic, a competing demand requires them to operate under very onerous accountability processes, with a top-down management structure imposed upon them (Kenny, 2009). The dichotomy of universities being managed as businesses, where academic activities are managed through strategic controls with the focus on outputs that can be quantified and compared, is troublesome but a certain reality (Reid, 2009). Given the trend of increased accountability in the education sector, there is increasing pressure on universities to perform (Dietz-Uhler & Hurn, 2013). Vast opportunities exist for the use of data in universities, including opportunities to improve decision- making on many levels (Gašević et al., 2015; Slade & Prinsloo, 2013). LA systems are being recognised as valuable education resources that have benefits to learning and teaching in universities (Arnold, Lonn, et al., 2014). Using today’s increased availability of big data around learner activity in various learning environments, LA can take us further than data traditionally available can in the higher education sector (Sclater et al., 2016). As an emerging discipline, LA has drawn the attention of academics, researchers and administrators (Siemens, 2013). Many LA higher education institutional initiatives in the big data age are still at an early stage, and their findings mostly remain unpublished (Sclater et al., 2016). On the other hand, there is a growing body of academic research literature on LA in higher education and its impact on student engagement, academic performance, retention and other measures of student success (Nunn et al., 2016; Banihashem et al., 2018; Colvin et al., 2016; Mangaroska & Giannakos, 2018). LA can be used in a university courses to monitor and predict student performance. Data-driven decision- making can involve making use of data, such as that found in the learning management systems (LMSs), to inform the educator’s judgements rather than their relying on the institution and hunches (Dietz-Uhler & Hurn, 2013; Long & Siemens, 2011; Olmos & Corrin, 2012). Smith et al. (2012) found that the frequency with which students logged in to their LMS, including engaging with the course material, predicted how they would perform in the course, similar to the method Amazon uses in predicting customers’ future purchases from their purchase history (Campbell et al., 2007). LA, although requiring considerable investment, is not a fad that will fade away. With advances in technology, opportunities are growing to personalise students’ learning needs, which will likely lead to greater success in their learning endeavours (Dietz-Uhler & Hurn, 2013). Shum and Ferguson (2012) posited LA should be embedded into the university so that a culture is developed that values the insights LA provide for both organisational strategic planning and improved student learning outcomes. While LA offers new technical opportunities in higher education, there are also challenges, such as ethics, privacy, theoretical foundations and scope and quality of data (Howell et al., 2018). Issues remain that influence the acceptance and adoption of LA among institutions and academics; for instance, educational culture (West et al., 2018), which is indicative of the institutional or national context and can result in bias regarding LA perspectives. Collaboration between academics, students and the university is needed in the development of LA policies and procedures to promote the suitability and widespread adoption of LA in Australasian Journal of Educational Technology, 2020, 36(6). 91 the higher education sector (Howell et al., 2018; Klein et al., 2019). LA is not only a technological phenomenon; it also has legal, moral and cultural implications. The weakest point for institutional capacity probably does not reside in technology or data but in how LA is used and what its effects on practice are (Arnold, Lonn, et al., 2014; Dawson et al., 2018; Lonn et al., 2017; West et al., 2018). This research aimed to uncover the diversity in attitudes towards adopting LA among decision-makers, staff users and LA developers. CFSs for implementing LA The growth in the adoption of LMSs has led to the availability of data sets which can be utilised for LA (Colvin et al., 2016). Data can be mined and analysed using LA systems which can inform education practice. It is therefore important that these systems are implemented within universities in a successful way (Clark & Tuffley, 2019). Greller and Draschler (2012) have noted the importance of carefully planned LA implementations in order for them to be successful. Ferguson et al. (2014) noted at the time of their research that there were few well-documented LA implementation examples existing in the literature; implementations across higher education institutions were typically immature, small in scale and with limited ability to demonstrate institutional impact. This research used an adaptation of a framework originally developed by Miranda et al. (2014), which identifies the CSFs that influence the adoption of e-learning systems. Instead of examining e-learning systems, the framework in this study sought to identify the CSFs pertinent to LA implementations. The original framework theorised the CSF dimensions of organisational, technological, education and training, evaluation, people and data accuracy (Table 1) to examine the factors that influence the successful adoption of e-learning systems (Miranda et al., 2014). Table 1 CSFs for information systems implementation (Miranda et al., 2014, p. 182) CFSs Categories Organisational Top management support and commitment Culture (collaborative, innovative) Communication (involvement of all in sharing information and opportunity for expression) Clear strategic goals Interdepartmental collaboration Motivation Technology New technologies Equipment conditions System complexity and lack of interoperability Hardware, software Education and training Continuous monitoring Training Evaluation Impact of the new system Monitoring and evaluating performance People Team capacity (high competence and expertise of the team) Specialisation Project manager Data accuracy Reliable data from both internal and external sources Maintenance and integrity Confidence in the information provided by the system There are also factors within each dimension that equally influence adoption. The organisational dimension consists of factors related to leadership, management support, communication, strategy, culture, and motivation. The technology dimension focuses on issues related to technological challenges and the condition of the equipment. Education and training looks at the training of users and staff involved in the implementation. The evaluation dimension looks at the overall impact of the system implementation. The people dimension ensures there are the required management and technical skills, and the data accuracy dimension ensures the confidence of the information about the system, maintaining integrity and accuracy Australasian Journal of Educational Technology, 2020, 36(6). 92 (Miranda et al., 2014). This research used the six dimensions of CSFs proposed by Miranda et al. but reformulated them based on an understanding of both the literature and practices of implementing LA in higher education. Hence, in the context of LA system implementation, the modified framework consists of five dimensions: strategy and policy at the organisational level, information technological readiness, performance and impact evaluation, people’s skills and expertise and data quality. Method Mixed-methods approach Following the philosophical framework of critical realism (Scott, 2005), this study adopted an abductive reasoning approach to theory development in order to understand the complex reality. The methodological choice was mixed methods because it is informed by both qualitative and quantitative inquiries, thus increasing the accuracy and confidence in conclusions (Neuman, 2011). An embedded mixed-methods design (Creswell & Plano Clark, 2011) was employed to obtain an in-depth understanding of the perceptions of CSFs of LA among higher education professionals. Both the main quantitative data and supportive qualitative data were concurrently collected via a self-completion online questionnaire of an international sample in 2019. In this study, quantitative analyses consisted of factor analysis of questionnaire items and profile analysis of the factor scores; qualitative analysis comprised the thematic analysis of responses to open-end questions. This study’s embedded mixed-methods design took advantage of those of previous studies mainly using qualitative individual or group interviews (Colvin et al., 2015; Dawson et al., 2018; Howell et al., 2018; Klein et al., 2019), questionnaire surveys (Arnold, Lonn, et al., 2014; Norris & Baer, 2013; Oster et al., 2016; Saint & Gutierrez, 2017; West et al., 2018), and case and policy studies (Arnold, Lynch, et al., 2014; Lonn et al., 2017; Tsai & Gašević, 2017), mostly done at a single institution in one country. Instrument and questionnaire The structured questionnaire included 19 CSF items relating to the implementation of LA in universities, nine items of LA goals (e.g., identifying at-risk students, scholarly research) rated on a 5-point Likert scale of importance from not important at all to extremely important and questions about respondents’ demographic information, including role in working with LA (e.g., academic using LA in teaching, education designer), years of experience in working with LA, country you are working in, and age group. The questionnaire also included five open-ended questions: • “How do you use Learning Analytics in your role?” • “What are the main challenges you have experienced in working with Learning Analytics at your university?” • “What are some of the main successes you can report in working with Learning Analytics at your university?” • “Is there any other factor/s that you see as critical to ensuring a Learning Analytics system is a success at your university?” • “What changes or improvements, if any, would be helpful or useful to you in your Learning Analytics role?” Ethics approval was obtained through the Office of Research Ethics & Integrity at The University of Queensland. The items were internally peer-reviewed to assess face and content validity. Such a combination of both deductive and inductive approaches to item generation ensures thorough domain sampling and sufficient content validity (Hinkin, 1995; Lewis et al., 2005). The final set of 19 items for five CSF dimensions is presented in Table 2. Australasian Journal of Educational Technology, 2020, 36(6). 93 Table 2 Scale for CSFs for implementing LA in universities Dimension Item Strategy and policy at the organisational level (ORG) Managerial commitment and support (ORG1) The culture of the university (ORG2) Open communication channels between all parties involved in implementing and using the system (ORG3) Alignment between LA implementation and strategic goals (ORG4) Collaboration across university departments (ORG5) The motivation behind the implementation; for example, is it a strategic initiative or coming from a teaching perspective? (ORG6) Information technological readiness (TEC) The technologies used for the implementation (TEC1) The condition of equipment (TEC2) System complexity issues (TEC3) Type of hardware and software chosen (TEC4) Usability of software (TEC5) Performance and impact evaluation (EVA) Evaluating the impact of the new system across the university (EVA1) Monitoring and evaluating the system’s performance (EVA2) People’s skills and expertise (PEO) Team competence and expertise (PEO1) Special skills of project team members responsible for implementation (PEO2) Skills of the project manager (PEO3) Data quality (DAT) The system produces reliable data from both internal and external sources (DAT1) The integrity of data is maintained (DAT2) Staff have confidence in the information provided by the system (DAT3) Analyses strategy The analyses of mixed data collected in this study consisted of three major steps (Figure 1). First, the measurement model of the five CSF dimensions (Table 2) was established through exploratory factor analyses (EFA) and confirmatory factor analyses (CFA). Second, profile analysis was performed based on the factor scores of the CSF dimensions in order to explore diversity in respondents’ perceptions of the relative importance of the five CSF dimensions. Finally, explanatory thematic analysis of the qualitative data facilitated the interpretation and comparison of CSF profile groups. Figure 1. Data analysis strategy Measurement modelling employs a multi-method strategy, which follows the psychometrics approach to developing a measurement model (DeVellis, 2012). First, we focused on developing scale items to map the conceptual dimensions of CSF (Table 2). Second, we performed EFA to verify that the identified items serve as indicators of the CSF dimensions. Further, we conducted CFA in the structural equation modelling (SEM) framework to assess how well the items represent CSF constructs in terms of convergent validity Measurement model • EFA • CFA • impute CSF factor scores Profile analysis • non-metric multidimensional scaling of CSF factors • individual regressions • identify profile groups Thematic analysis • generate themes based on coding qualitative data • compare themes between profile groups Australasian Journal of Educational Technology, 2020, 36(6). 94 and discriminant validity of CSF dimensions. Finally, we used correlation analysis to assess the scale’s concurrent validity with various goals of implementing LA in higher education. Profile analysis focused on the diversity in the pattern (i.e., the relative positions) of a set of criterion variables; that is, the five CSF dimensions of implementing LA in higher education. Profile analysis via multidimensional scaling (PAMS) is an exploratory method for identifying major profile patterns, which are represented as dimensions extracted from multidimensional scaling (MDS) analysis. MDS is not a new technique; however, the profile interpretation of dimensions is relatively new (Davison et al., 1996). PAMS defines the latent profiles among individuals using proximities among criteria variables, and then estimates the relationship between latent profiles and observed individuals’ profiles in their patterns. A PAMS model represents an individual observed score as a linear combination of dimensions where the dimensions are the most typical profile patterns present in a population (Kim et al., 2007). A PAMS model that extracts K latent dimensions (i.e., prototypical profiles) from a set of I individuals each observed on V criterion variables has the general form specified in the following equation: 𝑦𝑖(𝑣) = 𝑐𝑖 + ∑ 𝑤𝑖𝑘 𝑥𝑘(𝑣) + 𝑒𝑖(𝑣) 𝐾 𝑘=1 where 𝑦𝑖(𝑣) is an observed score of individual i on criterion variable v; 𝑐𝑖 is a level parameter estimate, which indexes the overall level of individual i’s observed profile and is usually the unweighted average of V variables’ values for individual i; 𝑥𝑘(𝑣) is the location coordinate or scale value for criterion variable v along latent dimension k and is usually obtained from non-metric MDS; 𝑤𝑖𝑘 is the profile match index for individual i on latent dimension k, indexing the degree of correspondence between individual i’s observed profile over V variables and the prototypical profile represented by the V variables’ scale values on latent dimension k; the K profile match indices of individual i are estimated by regressing the individual i’s observed V values onto K dimensions’ scale values with the unweighted least squares method, and the sign of profile match indices indicates that the individual profile exhibits either the prototypical profile or its opposite, the mirror image; and 𝑒𝑖(𝑣) is an error term for individual i on variable v, representing measurement error and systematic deviations from the model. PAMS identifies typical (normative) profiles within the group and simultaneously demonstrates how individuals differ with the individual regression’s model fit (R2) (Hon & Liu, 2016). This approach to profile analysis is exploratory and most suited to situations where normative profiles are derived from data rather than specified by a particular theory (Ding, 2006). Thematic analysis is a widely used and flexible qualitative analytic method that searches for themes or patterns in qualitative data (Braun & Clarke, 2006). The thematic analysis was performed in an analyst- driven way because the qualitative component was embedded in a quantitative survey, forming a mixed- methods research design for this study. Open coding and axial coding of the responses to open-ended questions served as the primary means of analysing the qualitative data. Themes were derived from those codes. Participants Purposive sampling with maximum variation (Byrne, 2001) in participant background was employed in this mixed-method study to reach theoretically relevant and various informants. Academics and professionals engaged in higher education LA from multiple countries were approached through our professional networks in higher education. Small, non-random samples are usually employed in exploratory studies to capture the nature and landscape of issues rather than estimate their statistical prevalence in the population (Gobo, 2004). We aimed at a sample size of 100 for this study considering the costs of data collection, the aim of the study, the complexity of models for testing and the recommended sample sizes for the data analysis procedures to perform. Spector (1992) suggested sample sizes of 100 to 200 for scale development studies. Given high communalities and strong factor loadings, lower sample sizes such as 100 are acceptable for EFA (Tabachnick & Fidell, 2013). For CFA in a SEM framework, depending on the relative complexity of the model to test and distributional properties of measures, a small sample size of 100 can be sufficient in the power of testing overall model fit (Bagozzi & Yi, 2012). PAMS, as a flexible exploratory method, does not restrict distributional assumptions in data or sample size (Ding, 2006). Australasian Journal of Educational Technology, 2020, 36(6). 95 Regarding sample size for qualitative research, data saturation in theoretical sampling usually occurs within the first 30 interviews or fewer (Boddy, 2016; Gentles et al., 2015). The online questionnaire was administrated in June and July 2019. A total of 110 valid responses from universities in 39 countries from all six continents were received. The median questionnaire completion time was 11 minutes with an interquartile range of 15 minutes (Q1 = 7 and Q3 = 22). The majority (65, 59%) of the sample were between 35 and 55 years old, with 8 (7%) respondents between 25 and 34 years, 26 (24%) between 55 and 64 years, and 11 (10%) above 65 years. Half of the sample (56, 51%) were academics using LA in teaching, and the others were educational designers (16, 15%), data scientists (11, 10%) and in other positions (27, 24%). Fifty-eight respondents (52%) had fewer than 5 years of experience in implementing LA in higher education, while the others (52, 48%) had 5 years or more of such experience. Among the 110 responses, there were only nine (0.43% ) missing values in the 19 items of the CSF scale. These missing data were then imputed with mean value replacement. Mahalanobis distances on the 19 items of the CSF scale were calculated to detect multivariate outliers. Although the three largest Mahalanobis distances were statistically significant (chi-square test’s p < 0.001; Tabachnick & Fidell, 2013), they were not substantially larger than the others; thus, all 110 responses were retained for further analysis. Results EFA As shown in Table 3, skewness and kurtosis statistics affirmed univariate normality of the 19 items of the CSF scale, which allowed for a plausible assumption of multivariate normality in the data for factor analyses. The dimensionality was evaluated with EFA in SPSS version 25 with maximum likelihood factor extraction using promax(4) oblique rotation and Kaiser normalisation. The Kaiser-Meyer-Olkin statistic (KMO = .832) and Bartlett’s test of sphericity (approx. χ2 = 929.371, df = 171, p < 0.001) indicated sufficient correlations among the 19 items, reflecting at least one common latent factor. The five-factor solution explained over 55% of total variance, with five eigenvalues over one and sufficient goodness of fit (χ2 = 103.226, df = 86, p = 0.099). More importantly, the five-factor solution showed the simple structure implied by the CSF scale, except for TEC5. The five factors were positively correlated with weak to moderate strength, which indicated discriminant construct validity of the subscales. All five subscales presented sufficient internal consistency according to their high Cronbach’s α scores. Australasian Journal of Educational Technology, 2020, 36(6). 96 Table 3 EFA results of the 19 items of CSFs Item Mean SD Skew Kurt Factor loading Cronbach’s α ORG TEC EVA PEO DAT Organisation (ORG) .86 ORG1 1.94 .92 1.19 1.73 .66 ORG2 2.25 1.02 .74 .23 .76 ORG3 2.15 1.06 .94 .67 .83 ORG4 2.15 .96 .88 .68 .59 ORG5 2.52 1.23 .54 -.66 .72 ORG6 2.14 .90 .93 1.23 .60 Technology (TEC) .55a .85 TEC1 2.01 .91 1.07 1.51 .57 TEC2 2.45 1.07 .61 -.11 .85 TEC3 2.22 .93 .78 .68 .72 TEC4 2.55 1.03 .39 -.41 .84 TEC5 1.75 .91 1.31 1.70 .28 .38 Evaluation (EVA) .45a .46a .74 EVA1 2.03 .82 1.26 3.05 .58 EVA2 1.95 .91 .85 0.71 .64 People (PEO) .28a .46a .19a .74 PEO1 1.77 .70 .67 .44 .50 PEO2 1.91 .78 .62 .04 .84 PEO3 1.98 .82 .43 -.51 .69 Data (DAT) .27a 39a 23a 50a .68 DAT1 1.60 .68 .88 .35 .71 DAT2 1.57 .74 1.29 1.36 .61 DAT3 1.80 .82 .70 -.36 .47 Note. ORG, TEC, EVA, PEO and DAT stand for the five CSF dimensions (Table 2). Factor loadings less than .30 are masked expect for TEC’s loading on TEC5 (.28); a = correlations between factors. CFA To further validate the latent structure of the CSF scale, CFA was conducted using a SEM package, AMOS version 25. Model fit was assessed based on the criteria of a number of goodness-of-fit measures: χ2 test’s p > .05, comparative fit index (CFI) ≥ .95, root mean square error of approximation (RMSEA) ≤ .06 with Pclose > .05, and standardised root-mean-square residual (SRMR) ≤ .08 (Hu & Bentler, 1999). The χ2 test for absolute goodness-of-fit is sensitive to model complexity and sample size. Thus, the normed χ2 statistic over degrees of freedom (df) between one and three (1 < χ2/df < 3) is a practical criterion for acceptable model fit. Maximum likelihood was used as the conventional estimation method for CFA models. Maximum likelihood estimation requires observed variables to follow a multivariate normal distribution. The CSF items are, in fact, on a 5-level Likert ordinal scale, which is acceptable in most practice (Kline, 2016). However, multivariate non-normal data, a large sample size and/or model complexity can inflate the χ2 measure for the model’s goodness-of-fit, leading to an underestimated p value (Kline, 2016). To overcome this issue, the bootstrapping adjustment of the χ2 test’s p value (Bollen & Stine, 1992) implemented in AMOS Version 25 was also used to assess the CFA model fit in this study. The initial CFA model of the 19 items of five CSF dimensions (Table 2) presented an acceptable model fit (χ2 = 191.872, df = 142, p = 0.003, Bollen-Stine bootstrap p = .373; χ2/df = 1.351; CFI = .939; root-mean-square error of approximation (RMSEA) = .057, Pclose = .288; SRMR = .060). Australasian Journal of Educational Technology, 2020, 36(6). 97 Since all the data were collected once in a cross-sectional questionnaire using a common Likert scale, common method variance could be a potential problem (Podsakoff et al., 2003). To assess shared variance among all items, a common factor loading on all 19 items was added to the initial CFA model. The average shared variance was only 6.5% but statistically significant (∆χ2 = 42.652, ∆df = 18, p < .001). Therefore, the common factor was retained in the CFA model to adjust for potential common method bias. The common-method-bias-adjusted CFA model showed an excellent goodness-of-fit (χ2 = 149.219, df = 124, p = 0.061, Bollen-Stine bootstrap p = .545; χ2/df = 1.203; CFI = .969; RMSEA = .043, Pclose = .657; SRMR = .052). The CSF subscales’ reliability was examined using Hancock’s coefficient of maximised reliability (H ≥ .70; Hancock & Mueller, 2001), composite reliability (CR ≥ .70; Malhotra & Dash, 2011) and variance extracted (VE ≥ .50; Fornell & Larcker, 1981). All five subscales displayed good reliability (Table 4). Each construct’s convergent validity was achieved since the factor loadings were significantly different from zero and the standardised loadings were generally above .70. Discriminant validity was also verified when the squared correlation (r2 < 1.0) between two factors was smaller than the two factors’ average VE. Table 4 Common-method-bias-adjusted CFA model of 19 items of CSFs Item β H CR VE Correlationa ORG TEC EVA PEO DAT ORG .855 .851 .488 .699 ORG1 .755 ORG2 .645 ORG3 .715 ORG4 .749 ORG5 .654 ORG6 .665 TEC .845 .830 .498 .569 .706 TEC1 .626 TEC2 .761 TEC3 .797 TEC4 .736 TEC5 .584 EVA .757 .749 .599 .656 .748 .774 EVA1 .731 EVA2 .814 PEO .726 .712 .454 .227 .370 .444 .674 PEO1 .603 PEO2 .755 PEO3 .655 DAT .736 .708 .451 .242 .401 .408 .585 .672 DAT1 .783 DAT2 .657 DAT3 .556 Note. β = standardised factor loadings and all factor loadings were statistically significant (p < .001); H = maximised reliability; CR = composite reliability; VE = variance extracted; a = the square root of VE is shown on the diagonal. Factor scores of the five CSF dimensions were imputed based on the common-method-bias-adjusted CFA model using AMOS version 25 and rescaled to the 5-point Likert scale of importance. Bivariate correlations were then calculated between the five CSF dimensions and nine goals of implementing LA in universities. The chord diagram in Figure 2 compares the strength of these positive correlations. For example, the goal of upholding academics’ accountability for student performance was strongly correlated with the performance and impact evaluation and information technological readiness CSF dimensions. Australasian Journal of Educational Technology, 2020, 36(6). 98 Figure 2. Correlations between nine goals of implementing LA and the five CSF dimensions Profile analysis via multidimensional scaling As the first step of PAMS, nonmetric multidimensional scaling was performed on the proximity data, measured as pairwise Euclidean distances among the five CSF factor scores. The PROXSCAL algorithm in SPSS version 25 was employed with SIMPLEX as the initial solution. The number of latent dimensions for a nonmetric MDS analysis is determined based on the interpretability of resulting prototypical profiles and MDS model fit statistics. The most commonly used badness-of-fit measure is STRESS (S1; Kruskal, 1964). S1 ranges from zero, indicating a good fit, to 1.00, indicating a misfit; that is, the rank order in the data could not be reproduced by the MDS model. The common arbitrary cut-off values of S1 are 0.2, fair; 0.1, good; 0.05, excellent; and 0.025, nearly perfect. In this case, the one-dimension MDS model was taken as the optimal solution, with an S1 of 0.006. Figure 3 illustrates the prototypical profile of the five CSFs. The scale values do not have any particular meaningful metric and simply represent the typical relative positions among the five CSF factor scores. The scale values with larger magnitude indicate salient criterion variables of the profile; in this case, strategy and policy at the organisational level, people’s skills and expertise and data quality. Australasian Journal of Educational Technology, 2020, 36(6). 99 Figure 3. The prototypical profile of the five critical success factors The prototypical profile was then used to predict the 110 respondents’ observed profiles on the five CSF scores with simple linear regression. The individual regression’s model fit (R2) indicated how much of the individual’s profile pattern was predicted by the prototypical profile, and the sign of the profile match index (PMI; i.e., regression coefficient) signified if the individual’s profile pattern was a mirror image of the prototypical profile. As shown in the side-by-side histograms in Figure 4, the distribution of individual model fit was comparable between the positive and negative PMI groups. A total of 36 individuals with low model fit (R2 < 20%) were considered as low-fit individuals in this study. Figure 4. Distribution of individual model fit in positive and negative PMI groups Figure 5 contrasts the typical profile patterns (with group average within individual standardised profile scores) between the positive PMI (n = 56) and negative PMI (n = 18) groups, without the 36 low-fit or atypical individuals. The two groups had divergent opinions on three CSFs dimensions: strategy and policy at the organisational level, data quality and people’s skills and expertise. One group emphasised the Evaluation Data People Technology Organisation S c a le V a lu e o f Im p o rt a n c e CSF High Medium Low Australasian Journal of Educational Technology, 2020, 36(6). 100 importance of data quality and people’s competence, whereas the other group focused on organisational strategy. The information technological readiness and performance and impact evaluation dimensions seemed to be placed in the middle area in both groups. No statistically significant association was found between the CSF profile grouping and being an academic (χ2 = 3.036, df = 2, p = .219) or having 5+ years of experience in working with LA (χ2 = 1.821, df = 2, p = .402). Figure 5. Two typical profile patterns of the five CSF dimensions Explanatory thematic analysis One of the major themes emerging from the open-ended survey questions was that LA implementations needed managerial support. This finding aligns with the strategy and policy organisational-level dimension used in the model (Miranda et al., 2014). Responses varied from senior executives providing financial support to ensuring the LA systems were strategically supporting the institutional goals. Some respondents reported little or no support for LA systems. Some universities reported that their systems were like silos – not interconnected and not supported. Another major issue involved systems’ performance issues and technical support. One respondent reported there were issues with bandwidth at their university. This respondent suggested that LA systems implementations require teamwork consisting of people in roles such as instructional designer, programmer, academician, content designer and domain expert. Along with a myriad of Internet connection issues, respondents felt that there was a lack of training and competent users in the area at their university. Two dimensions are relevant here: technology, and education and training (Miranda et al., 2014). A key issue that has arisen with the use of LA systems is data interpretation. Are the data from the LA system being interpreted correctly? This issue was prominent within the respondent data and aligns with the data quality dimension (Miranda et al., 2014). One respondent reported the problem of having 11 official languages at their university: We have to ensure that they understand a question before providing an answer and not just merely ticking a box to get it done. Interpreting the data from the systems in a trusted way for educational advices and instructional use. Also, if data are extracted from different sources or systems, it is important to ensure there are no data redundancy or accuracy errors. In many cases, data needed to be cleaned. Another reported issue is how data were interpreted by the teaching staff; teaching staff need skills in interpretation and data analysis. Respondents also reported that students do not trust data. Evaluation Data People Technology Organisation CSF Organization Data-People Relatively Important In-between Relatively Unimportant Australasian Journal of Educational Technology, 2020, 36(6). 101 Findings show it is also important to have a policy around data access. It was reported that ethical applications for research take time and need to be processed and this could get in the way of research with student data. Ethical issues such as data access and the use of data were at the forefront of reported responses. As one respondent noted, “Only a few people see it as a fruitful investigation, most are afraid of privacy issues.” A lack of clear standards and policy was also reported. Many were concerned about the privacy issues of data use created through the LA implementations. The findings also show that LA implementations are still in their infancy. It was reported that many implementations were still in the initial stages of adoption. One respondent expressed, “it is still far from exploring all the possibilities that LA can offer”. It is therefore important that these implementations are evaluated, as per the evaluation dimension (Miranda et al., 2014). Respondents reported that it can be difficult bringing all the stakeholders together on LA implementations to make them successful. Dedicated LA departments may be the solution to this problem (Clark & Tuffley, 2019). Discussion As information becomes an increasingly valuable commodity, education is investing in the adoption of LA systems to take advantage of the data that is produced during all the stages of the learning process, from enrolment to course completion. These systems have benefits that extend both to teaching and learning (Arnold, Lonn, et al., 2014). As happens with all innovations, LA requires the meticulous consideration of all stakeholders involved in order to assess how it can best serve their particular objectives. In exploring the CSFs of LA implementation, this research intended to reflect on the conditions that can assist the higher education sector to successfully harness the value of LA systems. The insights that were obtained via the questionnaire posit that the implementation of LA systems requires five key factors to be successful: strategy and policy at the organisational level, information technological readiness, performance and impact evaluation, people’s skills and expertise and data quality. The results presented the point of view of higher education professionals, including academics, educational designers and data scientists, about the use of LA systems, their successes and challenges and suggestions for improvements. The respondents also provided their input regarding the theoretical framework that this study proposed. Concerning the quantitative data, its analysis reveals a validation of all five categories of the framework, as was supported by Miranda et al. (2014). There was a greater incidence of agreement on data quality and people’s skills and expertise categories, revealing concern for these key aspects of implementation. Data quality is at the centre of LA systems’ success. It is essential to ensure the reliability of all data sources, to maintain the integrity of the data and to guarantee that users have confidence in the information that the LA system offers. These results not only highlight the importance of these aspects but they can also be used as a springboard to question data scientists about the most suitable measures to ensure data quality. Additionally, educational institutions can use the respondents’ input to invest in more solid data maintenance strategies and query their staff about their levels of confidence with the system that they use, in order to take appropriate action. Respondents’ agreement concerning the people’s skills and expertise category, in turn, discloses a preoccupation with the skills of those involved in the project. It focuses on the need for a competent team with adequate expertise, skills of the project team heading the implementation process of the system and the ability of the project manager. This emphasis on people underlines the essential role of human intervention in the adoption of technology, which significantly determines its success. Empowering existing staff through relevant training and recruiting new personnel are some of the actions that educational administrations can take to ensure that the people involved in the implementation process have the required skills to use the system to benefit the institution and attain the objectives for which it was employed. The qualitative analysis of the open-ended questions equally provided validation for the framework. The participants’ responses underlined the significance of the strategy and policy at the organisational level by referring to the need for managerial support both financially and strategically. Institutional support can manifest through a variety of measures, such as providing the necessary funding and guaranteeing that the LA systems are strategically supportive of the institutional objectives. The fact that some respondents Australasian Journal of Educational Technology, 2020, 36(6). 102 reported a lack of organisational sustenance for LA systems should prompt institutional action to guarantee that the implementation of these systems is not falling short of their goals due to the scarcity of support. With respect to information technological readiness, the aspects mentioned by the participants pertained to basic challenges of bandwidth and Internet connection constrictions, performance of the systems themselves and technical support. These types of problems are common to other types of technology adoption and seem to be recurrent in higher education institutions. If the higher education sector is to move forward with the deployment of learning technologies, there needs to be a considerable infrastructural upgrade allowing institutional-wide adoption and promoting scalability. This is true not only for LA systems but also for other types of systems and educational technologies and applications. In terms of the performance and impact evaluation dimension (Miranda et al., 2014), its importance was evident in the need to explore all the possibilities of LA. Data quality was equally present in the qualitative analysis through the contribution of some respondents who mentioned data interpretation, reliability and maintenance. The accuracy of data interpretation was highlighted in some responses, as was the need to clean the data and the lack of confidence that students have in the data. This last aspect is particularly important since a lack of trust in the data will be counterproductive, and the system will fail to offer students the benefits that they should derive from LA. When numerating other factors not included in the framework that lead to the effective deployment of LA systems, the participants volunteered data about the need for more training and skilled users, and the significance of ethical and privacy issues. Privacy and ethics were mentioned by some respondents as being central to successful implementation. The participants alerted to the existence of privacy and ethics and to the fact that they sometimes result in people’s misinterpretation of LA not as a valuable examination of data but as a potentially intrusive process. Conclusion LA systems need to be implemented in a successful manner (Clark & Tuffley, 2019) if they are to serve the purposes for which they were adopted. In proposing a framework composed of five main CSFs, this study adds to a growing body of research concerned with the effectiveness of LA systems implementation. More than a technological and strategic tendency, the rising significance of LA is justified by the volume of information that is available in education and the fact that such data can provide a richness of information capable of improving the learning process and enhancing the formal education experience. The results presented in this study reflect the need for a multilayered and multidisciplinary implementation approach guided by five key interconnected factors: strategy and policy at the organisational level, information technological readiness, performance and impact evaluation, people’s skills and expertise and data quality. This study contributes with data about LA systems implementation in higher education, exploring the factors that potentiate its success from the viewpoint of higher education professionals. Despite all efforts to offer a comprehensive framework, future research in the examination of CSFs will be essential to – equally – explore the viewpoint of higher education management and students (West et al., 2020) to provide a more inclusive perspective on implementation. Another limitation of this study concerns the sample size, which, in future research ventures, could be expanded to cover a higher number and variety of higher education stakeholders. Also, the framework provides a depiction of the factors but focuses more deeply on identifying what they are rather than on how they can be implemented. Prospective research (e.g., more in-depth qualitative inquiries into students’ perspectives, feedback and involvement in designing LA systems) is required to examine in detail each of the factors and assess how they can be operationalised in practice. Australasian Journal of Educational Technology, 2020, 36(6). 103 References Alhabeeb, A., & Rowley, J. (2018). E-learning critical success factors: Comparing perspectives from academic staff and students. Computers & Education, 127, 1–12. https://doi.org/10.1016/j.compedu.2018.08.007 Alrasheedi, M., Capretz, L. F., & Raza, A. (2015). Management’s perspective on critical success factors affecting mobile learning in higher education institutions: An empirical study. Journal of Educational Computing Research, 54(2), 253–274. https://doi.org/10.1177/0735633115620387 Arnold, K., Lonn, S., & Pistilli, M. (2014). An exercise in institutional reflection: The Learning Analytics Readiness Instrument (LARI). In A. Pardo & S. Teasley (Eds.), Proceedings of the 4th International Conference on Learning Analytics and Knowledge (pp. 163–167). Association for Computing Machinery. https://doi.org/10.1145/2567574.2567621 Arnold, K., Lynch, G., Huston, D., Wong, L., Jom, L., & Olsen, C. (2014). Building institutional capacities and competencies for systemic learning analytics initiatives. In A. Pardo & S. Teasley (Eds.), Proceedings of the 4th International Conference on Learning Analytics and Knowledge (pp. 257–260). Association of Computing Machinery. https://doi.org/10.1145/2567574.2567593 Bagozzi, R. P., & Yi, Y. (2012). Specification, evaluation, and interpretation of structural equation models. Journal of Academy of Marketing Science, 40, 8–34. https://doi.org/10.1007/s11747-011- 0278-x Banihashem, S. K., Aliabadi, K., Ardakani, S. P., Delaver, A., & Ahmadabadi, M. N. (2018). Learning analytics: A critical literature review. Interdisciplinary Journal of Virtual Learning in Medical Sciences, 9(2), Article e63024. https://doi.org/10.5812/ijvlms.63024 Boddy, C. R. (2016). Sample size for qualitative research. Qualitative Market Research: An International Journal, 19(4), 426–443. https://doi.org/10.1108/QMR-06-2016-0053 Bollen, K. A., & Stine, R. A. (1992). Bootstrapping goodness-of-fit measures in structural equation models. Sociological Methods & Research, 21(2), 205–229. https://doi.org/10.1177/0049124192021002004 Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101. https://doi.org/10.1191/1478088706qp063oa Byrne, M. (2001). Sampling for qualitative research. AORN Journal, 73(2), 494–498. https://doi.org/10.1016/S0001-2092(06)61990-X Campbell, J. P., DeBlois, P. B., & Oblinger, D. G. (2007). Academic analytics: A new tool for a new era. EDUCAUSE Review, 42(4), 40–57. http://www.educause.edu/EDUCAUSE+Review/EDUCAUSEReviewMagazineVolume42/Academic AnalyticsANewToolforaN/161749 Clark, J., & Tuffley, D. (2019). Learning analytics implementations in universities: Towards a model of success using multiple case studies. In Y. W. Chew, K. M. Chan, & A. Alphonso (Eds.), Proceedings of the the 36th International Conference of Innovation, Practice and Research in the Use of Educational Technologies in Tertiary Education (pp. 82–92). http://2019conference.ascilite.org/assets/proceedings/ASCILITE-2019-Proceedings-Final.pdf Colvin, C., Rogers, T., Wade, A., Dawson, S., Gašević, D., Shum, S. B., Nelson, K., Alexander, S., Lockyer, L., Kennedy, G., Corrin, L., & Fisher, J. (2015). Student retention and learning analytics: A snapshot of Australian practices and a framework for advancement. Australian Government Office for Learning and Teaching. http://www.olt.gov.au/project-student-retention-and-learning-analytics- snapshot-current- australian-practices-and-framework Creswell, J. W., & Plano Clark, V. L. (2011). Designing and conducting mixed methods research (2nd ed.). Sage Publications. Davison, M. L., Gasser, M., & Ding, S. (1996). Identifying major profile patterns in a population: An exploratory study of WAIS and GATB patterns. Psychological Assessment, 8, 26–31. https://doi.org/10.1037/1040-3590.8.1.26 Dawson, S., Poquet, O., Colvin, C., Rogers, T., Pardo, A., & Gašević, D. (2018). Rethinking learning analytics adoption through complexity leadership theory. In Proceedings of the 8th international conference on learning analytics and knowledge (pp. 236–244). Association for Computing Machinery. https://doi.org/10.1145/3170358.3170375 DeVellis, R. F. (2012). Scale development: Theory and applications (3rd ed.). Sage Publications. https://doi.org/10.1016/j.compedu.2018.08.007 https://doi.org/10.1177/0735633115620387 https://doi.org/10.1145/2567574.2567621 .%20https:/doi.org/10.1145/2567574.2567593 https://doi.org/10.1007/s11747-011-0278-x https://doi.org/10.1007/s11747-011-0278-x https://doi.org/10.5812/ijvlms.63024 https://doi.org/10.1108/QMR-06-2016-0053 https://doi.org/10.1177/0049124192021002004 https://doi.org/10.1191/1478088706qp063oa https://doi.org/10.1016/S0001-2092(06)61990-X http://www.educause.edu/EDUCAUSE+Review/EDUCAUSEReviewMagazineVolume42/AcademicAnalyticsANewToolforaN/161749 http://www.educause.edu/EDUCAUSE+Review/EDUCAUSEReviewMagazineVolume42/AcademicAnalyticsANewToolforaN/161749 http://2019conference.ascilite.org/assets/proceedings/ASCILITE-2019-Proceedings-Final.pdf http://www.olt.gov.au/project-student-retention-and-learning-analytics-snapshot-current-%20australian-practices-and-framework http://www.olt.gov.au/project-student-retention-and-learning-analytics-snapshot-current-%20australian-practices-and-framework https://doi.org/10.1037/1040-3590.8.1.26 https://doi.org/10.1145/3170358.3170375 Australasian Journal of Educational Technology, 2020, 36(6). 104 Dietz-Uhler, B., Hurn, J. E. (2013). Using learning analytics to predict (and improve) student success: A faculty perspective. Journal of Interactive Online Learning, 12(1), 17–26. https://www.ncolr.org/issues/jiol/v12/n1/using-learning-analytics-to-predict-and-improve-student- success.html Ding, C. S. (2006). Multidimensional scaling modelling approach to latent profile analysis in psychological research. International Journal of Psychology, 41(3), 226–238. https://doi.org/10.1080/00207590500412219 Ferguson, R., Macfayden, L. P., Clow, D., Tynan, B., Alexander, S., & Dawson, S. (2014). Setting learning analytics in context: Overcoming the barriers to large-scale adoption. Journal of Learning Analytics, 1(3), 120–144. https://doi.org/10.18608/jla.2014.13.7 Fornell, C., & Larcker, D. F. (1981). Evaluating structural equation models with unobservable variables and measurement error. Journal of Marketing Research, 18(1), 39–50. https://doi.org/10.2307/3151312 Freund, Y. (1988). Critical success factors. Planning Review, 16(4), 20–23. https://doi.org/10.1108/eb054225 Gašević, D., Dawson, S. & Siemens, G. (2015). Let’s not forget: Learning analytics are about learning. TECHTRENDS, 59, 64–71. https://doi.org/10.1007/s11528-014-0822-x Gentles, S. J., Charles, C., Ploeg, J., & McKibbon, K. (2015). Sampling in qualitative research: Insights from an overview of the methods literature. The Qualitative Report, 20(11), 1772–1789. https://nsuworks.nova.edu/tqr/vol20/iss11/5 Gobo, G. (2004). Sampling, representativeness and generalizability. In C. Seale, G. Gobo, J. F. Gubrium, & D. Silverman (Eds.), Qualitative research practice (pp. 405–426). Sage Publications. https://doi.org/10.4135/9781848608191.d32 Greller, W., & Drachsler, H. (2012). Translating learning into numbers: A generic framework for learning analytics. Journal of Educational Technology & Society, 15(3), 42–57. https://psycnet.apa.org/record/2012-33665-003 Hancock, G. R., & Mueller, R. O. (2001). Rethinking construct reliability within latent variable systems. In R. Cudeck, S. Du Toit, & D. Sörbom (Eds.), Structural equation modeling: Present and future. A Festschrift in honor of Karl Jöreskog (pp.195–216). Scientific Software International. Hinkin, T. R. (1995). A review of scale development practices in the study of organizations. Journal of Management, 21(5), 967–988. https://doi.org/10.1016/0149-2063(95)90050-0 Hon, C. K. H., & Liu, Y. (2016). Exploring typical and atypical safety climate perceptions of practitioners in the repair, maintenance, minor Alteration and addition (RMAA) sector in Hong Kong. International Journal of Environmental Research and Public Health, 13, Article 935. https://doi.org/10.3390/ijerph13100935 Howell, J. A., Roberts, L. D., Seaman, K., & Gibson, D. C. (2018). Are we on our way to becoming a “helicopter university”? Academics’ views on learning analytics. Technology, Knowledge and Learning, 23, 1–20. https://doi.org/10.1007/s10758-017-9329-9 Hu, L. T., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6, 1–55. https://doi.org/10.1080/10705519909540118 Ingram, H., Biermann, K., Cannon, J., Neil, J., & Waddle, C. (2000). Internalizing action learning: A company perspective. Establishing critical success factors for action learning courses. International Journal of Contemporary Hospitality Management, 12(2), 107–114. https://doi.org/10.1108/09596110010307369 Isaias, P., Miranda, P., & Pifano, S. (2009). Critical success factors for web 2.0: A reference framework. In A. A. Ozok & P. Zaphiris (Eds.), Online communities and social computing (pp. 354–363). Springer. https://doi.org/10.1007/978-3-642-02774-1_39 Kenny, J. D. (2009). Managing a modern university: Is it time for a rethink? Higher Education Research and Development, 28(6), 629–642. https://doi.org/10.1080/07294360903206934 Kim, S.K., Davison, M. L., & Frisby, C. L. (2007). Confirmatory factor analysis and profile analysis via multidimensional scaling. Multivariate Behavioral Research, 42(1), 1–32. https://doi.org/10.1080/00273170701328973 Klein, C., Lester, J., Rangwala, H., & Johri, A. (2019). Learning analytics tools in higher education: Adoption at the intersection of institutional commitment and individual action. The Review of Higher Education, 42(2), 565–593. https://doi.org/10.1353/rhe.2019.0007 Kline, R. B. (2016). Principles and practice of structural equation modeling (4th ed.). Guilford Press. https://www.ncolr.org/issues/jiol/v12/n1/using-learning-analytics-to-predict-and-improve-student-success.html https://www.ncolr.org/issues/jiol/v12/n1/using-learning-analytics-to-predict-and-improve-student-success.html https://doi.org/10.1080/00207590500412219 https://doi.org/10.18608/jla.2014.13.7 https://doi.org/10.2307/3151312 https://www.emerald.com/insight/publication/issn/0094-064X https://doi.org/10.1108/eb054225 https://doi.org/10.1007/s11528-014-0822-x https://doi.org/10.4135/9781848608191.d32 https://psycnet.apa.org/record/2012-33665-003 https://doi.org/10.1016/0149-2063(95)90050-0 https://doi.org/10.3390/ijerph13100935 https://doi.org/10.1007/s10758-017-9329-9 https://doi.org/10.1108/09596110010307369 https://doi.org/10.1007/978-3-642-02774-1_39 https://doi.org/10.1080/07294360903206934 https://doi.org/10.1080/00273170701328973 https://doi.org/10.1353/rhe.2019.0007 Australasian Journal of Educational Technology, 2020, 36(6). 105 Kruskal, J. B. (1964). Multidimensional scaling by optimizing goodness of fit to a nonmetric hypothesis. Psychometrika, 29, 1–27. https://doi.org/10.1007/BF02289565 Lewis, B. R., Templeton, G. F., & Byrd, T. A. (2005). A methodology for construct development in MIS research. European Journal of Information Systems, 14(4), 388–400. https://doi.org/10.1057/palgrave.ejis.3000552 Long, P. D., & Siemens, G. (2011, September 12). Penetrating the fog: Analytics in learning and education. EDUCAUSE Review, 46(5). http://www.educause.edu/ero/article/penetrating-fog-analytics- learning-and-education Lonn, S., McKay, T. A., & Teasley, S. D. (2017). Cultivating institutional capacities for learning analytics. New Directions for Higher Education, 179, 53–63. https://doi.org/10.1002/he.20243 Malhotra, N. K., & Dash, S. (2011). Marketing research: An applied orientation. Pearson Publishing. Mangaroska, K., & Giannakos, M. (2018). Learning analytics for learning design: A systematic literature review of analytics-driven design to enhance learning. IEEE Transactions on Learning Technologies, 12(4), 516–534. https://doi.org/10.1109/TLT.2018.2868673 McGill, T. J., Klobas, J. E., & Renzi, S. (2014). Critical success factors for the continuation of e-learning initiatives. The Internet and Higher Education, 22, 24–36. https://doi.org/10.1016/j.iheduc.2014.04.001 Miranda, P., Isaias P., Costa, C. J. (2014). From information systems to e-learning 3.0 systems’ critical success factors: A framework proposal. In P. Zaphiris & A. Ioannou (Eds.), Learning and collaboration technologies: Designing and developing novel learning experiences (pp. 180–191). Springer. https://doi.org/10.1007/978-3-319-07482-5_18 Miranda, P., Isaias, P., Costa, C. J., & Pifano, S. (2017). Validation of an e-learning 3.0 critical success factors framework: A qualitative research. Journal of Information Technology Education: Research, 16(1), 339–363. https://doi.org/10.28945/3865 Neuman, W. (2011). Social research methods: Qualitative and quantitative approaches. Pearson Education. Norris, D. M., & Baer, L. L. (2013, February 25). Building organizational capacity for analytics. EDUCAUSE Learning Initiative, 7–56. https://library.educause.edu/resources/2013/2/building- organizational-capacity-for-analytics Nunn, S., Avella, J. T., Kanai, T., & Kebritchi, M. (2016). Learning analytics methods, benefits, and challenges in higher education: A systematic literature review. Online Learning, 20(2), 13–29. https://www.doi.org/10.24059/olj.v20i2.790 Olmos, M., & Corrin, L. (2012). Learning analytics: A case study of the process of designs of visualizations. Journal of Asynchronous Learning Networks, 16(3), 39–49. https://doi.org/10.24059/olj.v16i3.273 Oster, M., Lonn, S., Pistilli, M. D., & Brown, M. G. (2016). The learning analytics readiness instrument. In Proceedings of the Sixth International Conference on Learning Analytics and Knowledge (pp. 173– 182). Association for Computing Machinery. https://doi.org/10.1145/2883851.2883925 Podsakoff, P. M., MacKenzie, S. B., Lee, J. Y., & Podsakoff, N. P. (2003). Common method biases in behavioral research: A critical review of the literature and recommended remedies. Journal of Applied Psychology, 88(5), 879–903. https://doi.org/10.1037/0021-9010.88.5.879 Poon, P., & Wagner, C. (2001). Critical success factors revisited: Success and failure cases of information systems for senior executives. Decision Support Systems, 30, 393–418. https://doi.org/10.1016/S0167- 9236(00)00069-5 Ram, J., Corkindale, D., & Wu, M-L. (2013). Implementation critical success factors (CSFs) for ERP: Do they contribute to implementation success and post-implementation performance? International Journal of Production Economics, 144(1), 157–174. https://doi.org/10.1016/j.ijpe.2013.01.032 Reid, I. C. (2009). The contradictory managerialism of university quality assurance. Journal of Education Policy, 24(5), 575–593. https://doi.org/10.1080/02680930903131242 Rockart, J. (1979, March). Chief executives define their own data needs. Harvard Business Review, 81– 92. https://hbr.org/1979/03/chief-executives-define-their-own-data-needs Romero, C., & Ventura, S. (2020). Educational data mining and learning analytics: An updated survey. WIREs Data Mining and Knowledge Discovery, 10(3), Article e1355. https://doi.org/10.1002/widm.1355 Saint, J., & Gutierrez, A. (2017). Adoption of learning analytics in the UK: Identification of key factors using the TOE framework (Regent’s working paper 1703). Regent’s University London. https://www.regents.ac.uk/sites/default/files/2018-11/rwp1703-saint-j-guttierrez-a.pdf https://doi.org/10.1007/BF02289565 https://doi.org/10.1057/palgrave.ejis.3000552 http://www.educause.edu/ero/article/penetrating-fog-analytics-learning-and-education http://www.educause.edu/ero/article/penetrating-fog-analytics-learning-and-education https://doi.org/10.1002/he.20243 https://doi.org/10.1109/TLT.2018.2868673 https://doi.org/10.1016/j.iheduc.2014.04.001 https://doi.org/10.1007/978-3-319-07482-5_18 https://doi.org/10.28945/3865 https://library.educause.edu/resources/2013/2/building-organizational-capacity-for-analytics https://library.educause.edu/resources/2013/2/building-organizational-capacity-for-analytics https://www.doi.org/10.24059/olj.v20i2.790 https://doi.org/10.1145/2883851.2883925 https://doi.org/10.1037/0021-9010.88.5.879 https://doi.org/10.1016/j.ijpe.2013.01.032 https://doi.org/10.1080/02680930903131242 https://hbr.org/1979/03/chief-executives-define-their-own-data-needs https://doi.org/10.1002/widm.1355 https://www.regents.ac.uk/sites/default/files/2018-11/rwp1703-saint-j-guttierrez-a.pdf Australasian Journal of Educational Technology, 2020, 36(6). 106 Sclater, N., Peasgood, A., & Mullan, J. (2016). Learning analytics in higher education: A review of UK and international practice. JISC. https://www.jisc.ac.uk/reports/learning-analytics-in-higher- education Scott, D. (2005). Critical realism and empirical research methods in education. Journal of Philosophy of Education, 39(4), 633–646. https://doi.org/10.1111/j.1467-9752.2005.00460.x Shum, S. B., & Ferguson, R. (2012). Social learning analytics. Educational Technology & Society, 15(3), 3–26. https://www.learntechlib.org/p/74970/ Siemens, G. (2013). Learning analytics: The emergence of a discipline. American Behavioral Scientist, 57(10), 138–1400. https://doi.org/10.1177/0002764213498851 Slade, S., & Prinsloo, P. (2013). Learning analytics: Ethical issues and dilemmas. American Behavioral Scientist, 57(10), 1510–1529. https://doi.org/10.1177/0002764213479366 Smith, V.C., Lange, A., & Huston, D.R. (2012). Predictive modeling to forecast student outcomes and drive effective interventions in online community college courses. Journal of Asynchronous Learning Networks, 16(3), 51–61. https://doi.org/10.24059/olj.v16i3.275 Spector, P. E. (1992). Summated rating scale construction: An introduction. Sage Publications. Tabachnick, B. G., & Fidell, L. S. (2013). Using multivariate statistics (6th ed). Pearson Education. Tsai, Y. S., & Gašević, D. (2017). Learning analytics in higher education: Challenges and policies: A review of eight learning analytics policies. In I. Molenaar, X. Ochoa, & S. Dawson (Eds.), Proceedings of the Seventh International Learning Analytics and Knowledge Conference (pp. 233– 242). Association for Computing Machinery. https://doi.org/10.1145/3027385.3027400 West, D., Tasir, Z., Luzeckyj, A., Kew, S. N., Toohey, D., Abdullah., Z., Searle, B., Jumaatm N. F., & Price, R. (2018). Learning analytics experience among academics in Australia and Malaysia: A comparison. Australasian Journal of Educational Technology, 34(3), 122–139. https://doi.org/10.14742/ajet.3836 West, D., Luzeckyj, A., Toohey, D., Vanderlelie, J., & Searle, B. (2020). Do academics and university administrators really know better? The ethics of positioning student perspectives in learning analytics. Australasian Journal of Educational Technology, 36(2), 60–70. https://doi.org/10.14742/ajet.4653 Corresponding author: Jo-Anne Clark, jo-anne.clark@griffithuni.edu.au Copyright: Articles published in the Australasian Journal of Educational Technology (AJET) are available under Creative Commons Attribution Non-Commercial No Derivatives Licence (CC BY-NC-ND 4.0). Authors retain copyright in their work and grant AJET right of first publication under CC BY-NC-ND 4.0. Please cite as: Clark, J., Liu,, Y., & Isasias, P. (2020). Critical success factors for implementing learning analytics in higher education: A mixed-method inquiry. Australasian Journal of Educational Technology, 36(6), 89-106, https://doi.org/10.14742/ajet.6164 https://doi.org/10.1111/j.1467-9752.2005.00460.x https://www.learntechlib.org/p/74970/ https://doi.org/10.1177/0002764213498851 https://doi.org/10.1177/0002764213479366 https://doi.org/10.24059/olj.v16i3.275 https://doi.org/10.1145/3027385.3027400 https://doi.org/10.14742/ajet.3836 https://doi.org/10.14742/ajet.4653 mailto:jo-anne.clark@griffithuni.edu.au https://creativecommons.org/licenses/by-nc-nd/4.0/ https://doi.org/10.14742/ajet.6164 Introduction Theoretical background LA in higher education CFSs for implementing LA Method Mixed-methods approach Instrument and questionnaire Analyses strategy Participants Results EFA CFA Profile analysis via multidimensional scaling Explanatory thematic analysis Discussion Conclusion References