Australasian Journal of Educational Technology, 2022, 38(1). 136 Massive open online courses for professional certificate programs? Perspectives on professional learners’ longitudinal participation patterns Hengtao Tang University of South Carolina Wanli Xing University of Florida Massive open online courses (MOOCs) have been integrated into higher education systems as an option for delivering online professional degree and certificate programs; however, concerns about whether employed professionals actively participate in MOOCs remain unresolved. Some researchers have described learners’ employment as the major cause of attrition from MOOCs, but research has not addressed how employed learners interact with MOOCs over time. Understanding employed professionals’ trajectory of participation patterns across course time is thus essential to improving the effectiveness of MOOCs. This study investigated the log data of learner participation to explore how attrition occurs in a professional MOOC, focusing on whether students’ employment status was associated with learner participation. The results revealed learners’ longitudinal participation patterns and confirmed the impact of sustained engagement on course performance. The study also found that employed learners were more likely than their peers without jobs to become cramming learners with initially infrequent engagement in a course but investing intensive time at the end for certificates. We discuss practical implications for designing and facilitating large- scale professional degree and certificate programs in higher education institutions. Implications for practice or policy: • Educators can apply MOOCs with a lower weekly workload and a slower pace to support employees’ professional development. • Educators should develop professional learners’ interests in the course topic to avoid only cramming for the course certificates. • Educators may consider longitudinal patterns of learner participation when assessing learner performance. Keywords: MOOCs, longitudinal clustering, professional certificate, temporal dimension, participation, performance Introduction To date, the controversy about massive open online courses (MOOCs) has transitioned from their potential of disrupting higher education into being avenues for universities to offer professional certificates and degrees. MOOCs allow higher education institutions to deliver certificates and degrees online to a large scale of professionals seeking a more affordable and more flexible option of improving their professional skillset and credential (Reich & Ruipérez-Valiente, 2019). However, research holds that professional learners’ employment status seems to be a barrier to consistent participation in MOOCs: studies indicate many employed learners’ work schedules leave them with too little time to participate in a MOOC (Cisel, 2014; Morris et al., 2015). Many learners enrolled in MOOCs hold full-time jobs, which may force them to plan the time dedicated to learning around work schedules (Littenberg-Tobias & Reich, 2020) and thus make it challenging to follow the course schedule or maintain the retention in MOOCs. Many employed professionals thus gradually disengage from an enrolled MOOC despite their initially active participation. Gradually disengaging learners are those who actively participate in a MOOC during the first several weeks, but their participation in the course starts to taper off over time (Tang et al., 2018). Attrition undermines the effectiveness of a MOOC (Zhang et al., 2016), and if gradually disengaging learners have been identified and provided with efficient instructional supports before any disengagement occurs, such learners might continue to participate (Xing et al., 2016). Supporting gradually disengaging learners requires Australasian Journal of Educational Technology, 2022, 38(1). 137 understanding how their participation patterns drop over time in MOOCs. Indeed, evidence arguing that learners with full-time jobs less actively participate in enrolled MOOCs and earn lower grades than learners who are not employed stems mainly from studies that enumerate the total frequency of learner participation patterns as learner engagement (Cisel, 2014; Morris et al., 2015; Tang & Bao, 2020, in press). In contrast, studies that looked at temporal patterns of learner participation found employment status does not predict such patterns (Shirvani Boroujeni, Kidzinski, et al., 2016; Shirvani Boroujeni, Sharma, et al., 2016). Using a time series analysis of learner activities, researchers (Shirvani Boroujeni, Kidzinski, et al., 2016; Shirvani Boroujeni, Sharma, et al., 2016) found that employed learners actually maintain more routine participation in MOOCs on weekends or on weekday evenings than unemployed learners. This regular pattern over time may amount to less overall participation but may not be a barrier to effective participation (Shirvani Boroujeni, Kidzinski, et al., 2016; Shirvani Boroujeni, Sharma, et al., 2016). To address this question the better to create supports for employed and unemployed learners alike, it is necessary to include the time dimension of learner participation in an analysis so as to reveal the longitudinal patterns of participation. To this end, we assembled participation feature sets into week-level units to build more effective models with better capabilities of representing time-series data in a continuous dataset, in line with past research (Gardner & Brooks, 2018; Xing et al., 2016). We used longitudinal k-means clustering algorithm (KmL) to provide a granular view of how learners’ employment status influenced gradual disengagement in MOOCs. Moving away from using self-reported data such as surveys and interviews collected after the course adjourns, we sought to understand log data about learner behaviour in a MOOC and support learners in a timely and efficient manner. For the analysis, we employed KmL to investigate the longitudinal pattern of learner participation in a MOOC and identified the gradually disengaging group. By comparing the course performance across three different groups, we found that gradual disengagement was detrimental to learner performance. Then we investigated whether learners’ employment status was associated with gradual disengagement. The findings contributed to a granular understanding of gradually disengaging learners in MOOCs and provided implications for higher education stakeholders on offering MOOCs as a more cost-effective and flexible alternative to professional certificate and degree programs (Kizilcec et al., 2020). Literature review Potentials of MOOCs as professional degree programs Higher education institutions are increasing online delivery of their programs in response to the escalating demand for higher education degrees and certificates (Littenberg-Tobias & Reich, 2020). MOOC providers have continued to establish partnerships with universities and other equivalent institutions to seek options for integrating MOOCs to supplement teaching and learning into higher education systems (Radford et al., 2014; Reich & Ruipérez-Valiente, 2019). In particular, many higher education institutions have adopted MOOCs as an alternative to traditional certificate and degree programs (Kizilcec et al., 2020). Traditional professional certificate and degree programs offered by higher education institutions are demanding in terms of time and cost for many employed professionals (Radford et al., 2014). In contrast, MOOCs have the advantage of providing professional learners with cost-effective access to high-quality educational resources offered by elite universities and also on-demand options for taking various courses (Tang & Carr-Chellman, 2016; Tang & Wang, 2019). This allows more employed professionals to afford the cost to take courses and the flexibility to arrange study time around their busy work schedules and social responsibilities (Littenberg-Tobias & Reich, 2020). Particularly for employed professionals who can only dedicate time to learning during off-work hours, flexible options in timetables are more likely to help them maintain a high quality of learning (Reich & Ruipérez-Valiente, 2019). On the other hand, MOOCs have the potential to serve unlimited numbers of online learners, which may allow a massive number of professional learners to enhance their credentials online at a lower cost (Littenberg-Tobias & Reich, 2020; Tang et al., 2020; Xing et al., 2019). Concerns about employed professionals’ participation in MOOCs It is worth noting, however, that flexibility is meaningful only for professional learners if they are autonomous and self-regulated online learners (Houlden & Veletsianos, 2019). Research on the effect of learners’ employment status on learner performance argues that professional learners with a full-time job Australasian Journal of Educational Technology, 2022, 38(1). 138 usually yield high attrition and a low performance in MOOCs (Cisel, 2014; Morris et al., 2015). Specifically, time constraints make full-time employees less likely to earn high grades in MOOCs (Cisel, 2014) or to complete them (Morris et al., 2015) than learners without a job (e.g., who are unemployed, retired and/or students). Existing concerns about employed professionals’ learner performance seemingly undermine MOOCs’ potential in reinforcing professional development and accreditation (Cisel, 2014; Morris et al., 2015; Tang, 2020). Ascertaining how learners’ employment status constrains their engagement and performance in MOOCs thus has significant potential to improve the effectiveness of MOOCs as professional certificate and degree programs. On the other hand, proponents of MOOC-based professional certificate and degree programs counter that professional learners have recorded a more consistent pattern of participation during off-work time (e.g., weekday evenings, weekends) despite a relatively lower overall participation (Shirvani Boroujeni, Kidzinski, et al., 2016; Shirvani Boroujeni, Sharma, et al., 2016). Research on the relationship between learner participation and learner performance in MOOCs also claims that low total frequency of participation in MOOCs does not always yield low performance, but a persistent trajectory of learner participation across time matters for learner performance in MOOCs (Tang, 2021a, 2021b). Accordingly, the influence of learner participation at different points on learner performance may vary (Tang et al., 2018, Tang et al., 2019); therefore, temporal variations in professional learners’ longitudinal participation patterns in MOOCs should be considered in order to ameliorate the time constraints of MOOCs as professional certificate and degree programs. Temporal dimension of learner participation Time is a significant dimension of learning (Barbera et al., 2015). This temporal dimension reflects an event-based understanding that learning is a cumulative process of events spanning a period of time, and each event makes different contributions to learner performance. (Reimann, 2009). In a temporal view, learning is not a constant variable but a dynamic process wherein learner engagement might fluctuate (Perna et al., 2014; Tang et al., 2019). However, Barbera et al. (2015) has argued that research does not usually address the temporal dimension of learning but considers learning as a static variable (i.e., variable-based learning). Similar research has relied on the total frequency of learner participation to relate learner participation to their performance, which might yield unreliable findings (Molenaar, 2014). For example, some gradually disengaging learners recorded large numbers of participation traits in the first several weeks, but seldom participated after the second week (Tang, 2018; Tang et al., 2018). Without understanding how participation varies over time, studies might have assigned this type of gradually disengaging learners to actively participating groups and thereby provided ineffective instructional support. Thus, whether the contention rom studies relying solely on summative measures of participation patterns is still tenable remains uncertain. A more granular understanding of learner participation calls for insights on how participation patterns evolve over time. Using week-by-week analysis, Canal et al. (2015) found that, for online learners with initial infrequent participation, cramming for exam success was counterproductive, especially given that more active participants always earned a higher score than cramming learners. This finding in part concurs that each event during the process of learning influences the outcome distinctively. Although Zhu et al. (2016) also found that active participation in the forum interactions during the second week resulted in a higher score in the quiz of that week, their week-by-week analysis suggested that this correlation might discontinue or reverse at some subsequent points, such as Weeks 5 or 6. Xing et al. (2016) applied an ensemble learning approach to build a predictive model that was more effective in detecting struggling learners and their gradually disengaged patterns in MOOCs. In building the model, they appended features of the previous weeks to the feature sets of the current week to represent the time-series data and to improve the model performance. Taking time into consideration, Tang et al. (2019) clustered online learners into three groups based on their longitudinal trajectory of learner engagement and identified the first period of an online course as the key time point that differentiates gradually disengaging learners from others. Those studies produced a more granular view of how the relationship between learner engagement and their performance changes over time. Conducting a temporal (e.g., week-by-week) analysis of whether the correlation exists between learner participation and learner performance is critical. This study thus sought to identify longitudinal patterns in learner participation: from patterns in learner participation over time, researchers can determine how the temporal dimension of participation influences learner performance and Australasian Journal of Educational Technology, 2022, 38(1). 139 identify how learners’ employment status constraints their longitudinal participation. Therefore, this study investigated the following two research questions with their hypothesis (H) respectively: (1) What is the influence of learners’ employment status on their longitudinal patterns of participation in a MOOC? H1: There is a difference in the longitudinal pattern of learner participation between employed professional and learners without a job in a MOOC. (2) What is the influence of learners’ longitudinal patterns of participation on their performance in a MOOC? H2: Learners who have a consistently higher longitudinal pattern of participation are more likely to outperform peers without such a pattern. Methodology Dataset The anonymised MOOC dataset used in this study was derived from an 8-week course offered by the Canvas Network. We received an approval of institutional review board protocol before accessing the dataset. The theme focused on project management and aimed to help professionals learn about the concepts, techniques and principles of project management. The course consisted of 11 modules: four were provided in the first week, and one was offered in each of the remaining weeks. The main components of each module included video lectures, assignments, quizzes and discussion forums. The course also included a total of 12 quizzes: one quiz at the end of each module and a final quiz in the final week. The data were retrieved from two main sources: clickstream and the Canvas application programming interface. The clickstream data recorded the time stamps of the learners’ visits to specific pages (e.g., lectures, forums and quizzes). The remaining data were from JavaScript object notation and obtained from the Canvas application programming interface. These data included details regarding the learners’ quiz scores and forum traits. In total, the dataset included the participation records and quiz score of 640 valid learners. Variables The study investigated three variables: learner participation, learner performance and employment status: • Learner participation. To represent the time-series data, the study used weekly patterns of learner participation as units of analysis. The weekly participation patterns of each learner included the sum of their page views, forum posts and replies and attempts and submissions for each assessment (see Table 1). • Learner performance. The study retrieved the grades that each learner earned on all the quizzes. The average grade of each learner was used to characterise their performance. • Employment status. Learner employment status was binary in this study: employed or unemployed. The relevant data were obtained by coding the posts from the first forum designed especially for self-introduction. A total of 444 threads were posted by 300 learners, but 144 of these threads were excluded from coding because they were comments without information about the learner’s employment status. The constant comparative method was then used to code the remaining 300 threads (Glaser & Strauss, 1967). Employment status (employed or unemployed) was the coding scheme (see Table 2). To ensure adequate inter-rater reliability, each of us individually coded 30 posts (10% of the total number of posts) and calculated Cohen’s (1960) kappa value (k = 0.92). Then, one of us coded the remaining posts individually. In total, 56 void posts were removed because they were not about the learner’s employment status (see Table 2). We obtained the employment-status data for 244 learners. Australasian Journal of Educational Technology, 2022, 38(1). 140 Table 1 Week-by-week patterns of learner participation Week Mean Range SD 1 29.34 401 48.97 2 32.37 393 56.46 3 30.90 513 59.80 4 30.27 299 50.75 5 30.55 472 61.23 6 19.22 498 47.53 7 20.90 445 47.94 8 33.23 627 76.86 Table 2 Coding schemes of learners’ employment status in the forum posts Code Excerpts from forum posts Employed Currently I am working as operational manager for an export company. Although I have some experience in projects I would like to acquire some Professional know-how in dealing with projects. Unemployed I am still a University student and have no substantial work project management experience. I am currently studying my master degree of Bioscience enterprise; therefore I am interested in managing biotechnology projects in the future. Void Hi, Mi [My] name is [name], and I am interested in Project Management. Data analysis Prior to the data analysis, the dataset was normalised. The result after normalisation between 0 and 1 is shown in Figure 1. To analyse the data, the study primarily used KmL (see Figure 2). Because KmL is an unsupervised clustering algorithm, the first step in the data analysis was to determine the optimal number of clusters. To increase its reliability, this study used a pseudo-T-square analysis and an iterative self- organising data analysis technique algorithm to achieve a 2-step verification. The pseudo-T-square analysis provides the optimal number of clusters when its curve demonstrates the greatest fluctuation (Edens et al., 1999), and an ISODATA algorithm yields its optimal value at the point at which its curve apparently bends (Milligan & Cooper, 1985). The KmL grouped learners with different longitudinal patterns of participation into three clusters – see Figure 3. Figure 1. Longitudinal trajectory of participation after normalisation between 0 and 1 Australasian Journal of Educational Technology, 2022, 38(1). 141 Figure 2. Sample illustration of KmL (a) (b) Figure 3. (a) The results of the pseudo-T-square and (b) the results of the ISODATA algorithm show the clustering number is 3. The second step in the data analysis was to use the KmL to group the participants into three clusters according to their longitudinal patterns of participation across the 8 weeks (see Figure 2). The KmL used Euclidean distance to evaluate the similarity of two participants – Dist(Xa,Xb): Dist(𝑋𝑎,𝑋𝑏) = √ 1 𝑇 ∑ (𝑋𝑎𝑡 − 𝑋𝑏𝑡) 2 𝑇 𝑡=1 A smaller value for Euclidean distance between two participants (a and b in the equation) meant they were more likely to be assigned to the same cluster (Genolini & Falissard, 2010). The study then investigated the degree to which different longitudinal patterns of participation affected learner performance. A chi-square test with post-hoc testing (e.g., adjusted standardised residuals) was conducted to determine how learners’ longitudinal pattern of participation differed by their employment status. Then, an analysis of variance (ANOVA) and a Tukey’s honestly significant difference post-test were used to detect any significant differences in learner performance among the three clusters. Results The participants were grouped into three clusters according to their longitudinal patterns of participation (see Figure 4 and Table 3). As described below, these three clusters had recorded participation patterns that changed over time and that significantly differed from each other. Australasian Journal of Educational Technology, 2022, 38(1). 142 Cluster A, the seldom engaging cluster, constituted 62.7% of the participants (n = 401). These learners demonstrated much lower participation rates throughout the course than the other two clusters. Their participation reached its peak during the first week and then decreased. While the last week was the only week when they were not the least engaged cluster, their participation was below average every week. Cluster B, the gradually disengaging cluster, constituted 22.3% of the participants (n = 143). Most of the learners in this cluster were highly engaged from the beginning to the middle of the course – indeed, some of them were the most engaged of all of the students in the first 5 weeks. However, their participation gradually decreased after the third week and then dramatically dropped after the fifth week. Particularly, most of the learners in this cluster were seldom engaged in the last week and their participation was the lowest among all three of the clusters. Cluster C, the cramming cluster, constituted 15.0% of the participants (n = 96). They demonstrated a much higher level of participation at the end of the course than at the beginning. This cluster was the second most-engaged in the first 5 weeks after Cluster A, but over the duration of the course, they maintained a relatively regular participation pattern, one without any significant increases or decreases. Learners in this cluster demonstrated a slight decrease in participation between Week 4 and Week 6; their lowest level of participation came in Week 6. After Week 6, however, this cluster became the most engaged, with their course participation levels increasing continuously until the end of the course. This cluster participated extremely actively during the last week, which was also the peak week for this cluster. Figure 4. Three clusters of learners differentiated by their longitudinal patterns of participation Table 3 Three clusters of learners differentiated by their longitudinal patterns of participation Week Cluster A (62.70%) Cluster B (22.34%) Cluster C (15%) Mean Range SD Mean Range SD Mean Range SD 1 14.14 263 28.75 59.36 401 70.14 48.12 235 51.05 2 9.02 223 24.24 76.52 329 69.98 64.14 393 73.42 3 5.89 118 15.01 82.30 513 86.88 58.78 393 66.47 4 7.32 203 18.77 72.42 299 62.15 63.34 278 63.75 5 10.28 294 29.79 77.10 472 90.72 45.90 324 62.83 6 8.26 259 24.82 31.43 498 71.66 46.81 270 58.54 7 6.23 138 18.97 30.55 287 58.50 67.83 445 74.55 8 10.58 627 46.65 10.54 89 19.42 161.68 523 101.96 Australasian Journal of Educational Technology, 2022, 38(1). 143 What is the influence of learners’ employment status on their longitudinal patterns of participation in a MOOC? We analysed the coded posts from the self-introductory forum. We then applied a chi-square test to determine how longitudinal patterns of learner participation differed by their employment status. The result (χ2 = 6.070; df = 2; p < .05) confirmed that learners with jobs maintained different longitudinal patterns of participation from their peers without jobs. This finding did not reject H1. The effect size for the chi-square analysis showed this was association between two variables, Cramer’s V = 0.158, Phi = 0.158, p < .05. The post-hoc test results indicated the adjusted residuals for both types of learners in Cluster C were more than 2.0 (see Table 4). For the learners with jobs, the number of cases in Cluster C was significantly larger than would be expected if the null hypothesis was true, with a significance level of .05. The learners without jobs were less likely to become cramming learners who had initially infrequent engagement but then boosted participation at the end of the course. Table 4 Crosstabulation of each employment status as reported in the coded forum posts Clusters Measures Unemployed Employed Total Cluster A Count 21 95 116 Expected count 17.1 98.9 116.0 Adjusted residual 1.4 -1.4 Cluster B Count 13 64 77 Expected count 11.4 65.6 77.0 Adjusted residual 0.6 -0.6 Cluster C Count 2 49 51 Expected count 7.5 43.5 51.0 Adjusted residual -2.5 2.5 Total 36 208 244 What is the influence of learners’ longitudinal patterns of participation on their performance in a MOOC? An ANOVA test was conducted to determine how gradual disengagement patterns influenced learner performance (see Table 5). The result confirmed that the three clusters differed significantly in their performance (F = 476.3, p < .05). The result of the Tukey’s honestly significant difference post-test revealed that while Clusters B and C significantly outperformed Cluster A, no significant difference in performance was observed between Clusters B and C (see Table 6). This finding did not reject H2. Table 5 Average quiz grades for the three clusters Cluster(s) Mean Range SD A (n = 401) 1.92 15.25 3.48 B (n = 143) 10.28 13.11 4.27 C (n = 96) 12.13 15.08 3.27 Table 6 Difference in learner performance among the clusters form Tukey’s honestly significant difference post- test results Clusters Mean difference 95% family-wise confidence level Significance Lower bound Upper bound B-A 9.03 8.21 9.85 p < 0.001 C-A 9.69 8.74 10.64 p < 0.001 C-B 0.66 -0.45 1.77 0.34 Discussion MOOCs have been increasingly integrated in professional certificate and degree programs through the partnership established between MOOC providers and higher education institutions, but whether this Australasian Journal of Educational Technology, 2022, 38(1). 144 flexible option can benefit employed professionals remain disputed due to employed professionals’ high attrition rates and low performance in enrolled MOOCs (Littenberg-Tobias & Reich, 2020; Reich & Ruipérez-Valiente, 2019). To determine why such attrition occurs, this study went beyond counting total frequency of learner participation and analysed learners’ week-by-week participation patterns. The study revealed three clusters of the longitudinal participation patterns – including gradually disengaging learners, cramming learners and seldom engaging learners. It also investigated the relationship between learners’ longitudinal patterns of participation and their performance. While the gradually disengaging and cramming learners outperformed the seldom engaging learners, no significant difference in performance was found between these two outperforming clusters. Finally, the study revealed learners might maintain different participation patterns depending on whether they had a job or not. In particular, employed learners were more likely than their peers without a job to cram for a satisfying grade or certificates at the end of a MOOC. Additionally, the findings support that learner participation fluctuates over time regardless of learner employment status, echoing the findings of Shirvani Boroujeni, Kidzinski, et al. (2016) and Shirvani Boroujeni, Sharma, et al. (2016). While prior research that did not consider changes in participation over time has concluded that learners with jobs tend to complete less in a MOOC and earn lower grades than their unemployed peers (Cisel, 2014; Morris et al., 2015), this study finds that these employed learners actually are more likely to become cramming learners who invest intensive time and effort at the end to earn a certificate from a MOOC. This finding refutes the argument of Shirvani Boroujeni, Kidzinski, et al. (2016) and Shirvani Boroujeni, Sharma, et al. (2016) that employed learners maintained more routine participation compared to their peers without jobs. This might result from the course subject of the focal MOOC being about project management, which might mean employed learners register for this MOOC as a professional development opportunity. This finding further questions the potential of adopting MOOCs as supplementary resources for professional development (Radford et al., 2014), especially given that cramming learners did not outperform those gradually disengaging learners. To reinforce this potential of MOOCs, understanding how to motivate employed learners to consistently interact with the course content becomes necessary. Furthermore, employed professionals might have too little time to complete a MOOC, and this can affect how they interact with the course over time. The finding also adds to Kizilcec et al.'s (2013) argument that such factors as personal commitment, work conflict and course workload predict disengagement. To mitigate the influence of learners’ external commitments, Kizilcec et al.’s (2013) propositions on offering MOOCs in which students can pace themselves or participate at a slower pace might present value to learners with jobs. Furthermore, the findings resonate with those of Canal et al. (2015) in that the dramatic increase in learner participation observed at the end of the course did not help the cramming cluster to accomplish more. This raises concerns about the motivations of learners who enroll in MOOCs. We speculate that the learners in the cramming group were not intrinsically motivated. Further studies into the motivational status of online learners could seek to support these cramming learners to develop intrinsic motivations towards MOOCs. This study also concurs with Zhang et al.’s (2016) finding that disengagement undermines the effectiveness of MOOCs. While the gradually disengaging cluster was more involved overall than was the cramming cluster, they performed no better than did the cramming cluster, probably as a result of their gradual disengagement. Likewise, the cramming cluster was unable to outperform the gradually disengaging cluster because the cramming cluster exhibited a low level of learner participation in the first several weeks. Indeed, research has indicated that only learners with a constantly active participation pattern often perform at higher levels than their peers (Canal et al., 2015; Tang et al., 2018). From another perspective, this finding further supports the idea that to perform exceptionally well in online courses, learners may have to maintain consistently high levels of participation. The findings also reinforce the idea that to fully understand learning, one must possess a granular view of how it unfolds over time (Chen et al., 2016; Knight et al., 2017; Molenaar, 2014). Echoing Xing et al. (2016), this study also argues that counting the total frequency of learner participation is unable to capture gradual disengagement from MOOCs. Taking the example of this study, the gradually disengaging cluster was the most active group until the fifth week. However, if we focus only on the total frequency of participation, this cluster of learners might be considered active learners overall, which is not appropriate for the design of interventions to prevent student attrition (Xing et al., 2016). This study also offers practical recommendations that may help educators to improve the performance of their online learners. First, it is important to consider longitudinal patterns of learner participation when Australasian Journal of Educational Technology, 2022, 38(1). 145 assessing learner performance (Kizilcec et al., 2013; Molenaar, 2014). Only by examining participation over time can educators accurately evaluate learner performance and provide efficient and timely interventions. Second, educators might use MOOCs to support employees’ professional development, but these employees might desire a relatively lower weekly workload and slower pace in a MOOC to ensure they can efficiently manage their time and stay engaged (Kizilece et al., 2013). Third, MOOC educators and offering institutions might tailor the course topics and contents to students’ interests and practical needs. For example, educators could use project-based learning theory to frame their courses and help students to relate course outcomes to their career trajectory needs. Fourth, this study also confirmed that cramming (e.g., for a final exam grade) does not help online learners. To improve learner performance and make MOOCs more effective, educators could seek to develop student interest in the course topic and explain its relevance for their students’ chosen professions (Cisel, 2014; Tang et al., 2018). Limitations and future research This study faced several limitations. First, the course did not include formal assessments – only quizzes. The study used learners’ average score on the quizzes to assess their individual performance in the course, but a formal overall assessment in future research might provide richer data on how learner performance correlates with longitudinal participation patterns. Second, learners have a wide range of intentions in registering for a MOOC, but the limitation of the dataset prevented additional investigations on individual demographic information such as the motivational factors and their influence on learner participation patterns. For future research, it might be useful to include some formative assessment for learners to provide more enriched data. Third, this study examined a single course on a single course platform. Future research could produce findings with greater validity by using multiple datasets obtained from larger numbers of courses and course platforms. Conclusions This exploratory study identified learners’ longitudinal profiles of their participation patterns in MOOCs and provided significant implications for higher education institutions on offering MOOCs as professional certificate and degree programs. As a result, it provides a granular understanding of gradual disengagement from MOOCs, and it offers insightful recommendations to online educators for facilitating online courses. Because disengagement undermines the success of MOOCs (Zhang et al., 2016), helping learners to remain always actively engaged is especially important. This study also reveals employed learners are more likely than learners without jobs to cram for a certificate or a higher grade at the end of MOOCs; it remains uncertain whether such courses efficiently support the professional development of employees. It is also worth noting that timely and efficient interventions are required to maximise the value of MOOCs to online learners (Perna et al., 2014; Tang et al., 2018; Tang & Wang, 2019; Xing et al., 2016). To increase the validity of these findings, future studies could use large-scale datasets collected from multiple courses offered by different providers. In addition, future research might use better validated measures to evaluate learner performance. Acknowledgements We appreciate the support provided by Jared Stein from Instructure of Canvas. References Barbera, E., Gros, B., & Kirschner, P. A. (2015). Paradox of time in research on educational technology. Time & Society, 24(1), 96–108. https://doi.org/10.1177/0961463X14522178 Canal, L., Ghislandi, P., & Micciolo, R. (2015). Pattern of accesses over time in an online asynchronous forum and academic achievements. British Journal of Educational Technology, 46(3), 619–628. https://doi.org/10.1111/bjet.12158 Chen, B., Wise, A. F., Knight, S., & Cheng, B. H. (2016). Putting temporal analytics into practice: The 5th international workshop on temporality in learning data. In S. Dawson, H. Drachsler, & C. Rose (Eds.), Proceedings of the Sixth International Conference on Learning Analytics & Knowledge (pp. 488–489). Association for Computing Machinery. https://doi.org/10.1145/2883851.2883865 https://doi.org/10.1177/0961463X14522178 https://doi.org/10.1111/bjet.12158 https://doi.org/10.1145/2883851.2883865 Australasian Journal of Educational Technology, 2022, 38(1). 146 Cisel, M. (2014, February 10–12). Analyzing completion rates in the first French xMOOC [Paper presentation]. Second European MOOCs Stakeholders Summit, Lausanne, Switzerland. Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20, 37–46. https://doi.org/10.1177/001316446002000104 Edens, J. F., Cavell, T. A., & Hughes, J. N. (1999). The self‐systems of aggressive children: A cluster‐ analytic investigation. Journal of Child Psychology and Psychiatry, 40(3), 441–453. https://doi.org/10.1111/1469-7610.00461 Gardner, J., & Brooks, C. (2018). Student success prediction in MOOCs. User Modeling and User- Adapted Interaction, 28(2), 127–203. https://doi.org/10.1007/s11257-018-9203-z Genolini, C., & Falissard, B. (2010). KmL: k–means for longitudinal data. Computational Statistics, 25(2), 317–328. https://doi.org/10.1007/s00180-009-0178-4 Glaser, B., & Strauss, A. (1967). The discovery of grounded theory: Strategies for qualitative research. Aldine. Houlden, S., & Veletsianos, G. (2019). A posthumanist critique of flexible online learning and its “anytime anyplace” claims. British Journal of Educational Technology, 50(3), 1005–1018. https://doi.org/10.1111/bjet.12779 Kizilcec, R. F., Piech, C., & Schneider, E. (2013). Deconstructing disengagement: Analyzing learner subpopulations in massive open online courses. In D. Suthers, K. Verbert, E. Duval, & X. Ochoa (Eds.), Proceedings of the Third International Conference on Learning Analytics and Knowledge (pp. 170–179). Association for Computing Machinery. https://doi.org/10.1145/2460296.2460330 Kizilcec, R. F., Reich, J., Yeomans, M., Dann, C., Brunskill, E., Lopez, G., & Tingley, D. (2020). Scaling up behavioral science interventions in online education. Proceedings of the National Academy of Sciences, 117(26), 14900–14905. https://doi.org/10.1073/pnas.1921417117 Knight, S., Wise, A. F., & Chen, B. (2017). Time for change: Why learning analytics needs temporal analysis. Journal of Learning Analytics, 4(3), 7–17. https://doi.org/10.18608/jla.2017.43.2 Littenberg–Tobias, J., & Reich, J. (2020). Evaluating access, quality equity in online learning: A case study of a MOOC–based blended professional degree program. The Internet and Higher Education, 47, Article 100759. https://doi.org/10.1016/j.iheduc.2020.100759 Milligan, G. W., & Cooper, M. C. (1985). An examination of procedures for determining the number of clusters in a data set. Psychometrika, 50(2), 159–179. https://doi.org/10.1007/BF02294245 Molenaar, I. (2014). Advances in temporal analysis in learning and instruction. Frontline Learning Research, 2(4), 15–24. https://doi.org/10.14786/flr.v2i4.118 Morris, N. P., Hotchkiss, S., & Swinnerton, B. (2015, May 18–20). Can demographic information predict MOOC learner outcomes? [Paper presentation]. Third European MOOCs Stakeholders Summit, Mons, Belgium. Perna, L. W., Ruby, A., Boruch, R. F., Wang, N., Scull, J., Ahmad, S., & Evans, C. (2014). Moving through MOOCs: Understanding the progression of users in massive open online courses. Educational Researcher, 43(9), 421–432. https://doi.org/10.3102/0013189X14562423 Radford, A. W., Robles, J., Cataylo, S., Horn, L., Thornton, J., & Whitfield, K. E. (2014). The employer potential of MOOCs: A mixed–methods study of human resource professionals’ thinking on MOOCs. The International Review of Research in Open and Distributed Learning, 15(5). https://doi.org/10.19173/irrodl.v15i5.1842 Reich, J., & Ruipérez–Valiente, J. A. (2019). The MOOC pivot. Science, 363(6423), 130–131. https://doi.org/10.1126/science.aav7958 Reimann, P. (2009). Time is precious: Variable-and event-centred approaches to process analysis in CSCL research. International Journal of Computer-Supported Collaborative Learning, 4(3), 239– 257. https://doi.org/10.1007/s11412-009-9070-z Shirvani Boroujeni, M., Kidzinski, L., & Dillenbourg, P. (2016). How employment constrains participation in MOOCs? In T. Barnes, M. Chi, & M. Feng (Eds.), Proceedings of the 9th International Conference on Educational Data Mining (pp. 376–377). International Educational Data Mining Society. http://www.educationaldatamining.org/EDM2016/proceedings/edm2016_proceedings.pdf Shirvani Boroujeni, M., Sharma, K., Kidziński, Ł., Lucignano, L., & Dillenbourg, P. (2016). How to quantify student’s regularity? In European Conference on Technology Enhanced Learning (pp. 277– 291). Springer. https://doi.org/10.1007/978–3–319–45153–4_21 Tang, H. (2018). Exploring self–regulated learner profiles in MOOCs: A comparative study. (Publication No. 13871931) [Doctoral dissertation, The Pennsylvania State University]. ProQuest Dissertations Publishing. https://doi.org/10.1177/001316446002000104 https://doi.org/10.1111/1469-7610.00461 https://doi.org/10.1007/s11257-018-9203-z https://doi.org/10.1007/s00180-009-0178-4 https://doi.org/10.1111/bjet.12779 https://doi.org/10.1145/2460296.2460330 https://doi.org/10.1073/pnas.1921417117 https://doi.org/10.1016/j.iheduc.2020.100759 https://doi.org/10.1007/BF02294245 https://doi.org/10.14786/flr.v2i4.118 https://doi.org/10.3102/0013189X14562423 https://doi.org/10.19173/irrodl.v15i5.1842 https://doi.org/10.1126/science.aav7958 https://doi.org/10.1007/s11412-009-9070-z http://www.educationaldatamining.org/EDM2016/proceedings/edm2016_proceedings.pdf https://doi.org/10.1007/978–3–319–45153–4_21 Australasian Journal of Educational Technology, 2022, 38(1). 147 Tang, H. (2020). A qualitative inquiry of k–12 teachers’ experience with open educational practices: Perceived benefits and barriers of implementing open educational resources. The International Review of Research in Open and Distributed Learning, 21(3), 211–229. https://doi.org/10.19173/irrodl.v21i3.4750 Tang, H. (2021a). Implementing open educational resources in digital education. Educational Technology Research and Development, 69(1), 389–392. https://doi.org/10.1007/s11423–020–09879–x Tang, H. (2021b). Person–centered analysis of self–regulated learner profiles in MOOCs: A cultural perspective. Educational Technology Research and Development, 69(2), 1247–1269. https://doi.org/10.1007/s11423-021-09939-w Tang, H., & Bao, Y. (2020). Social justice and K–12 teachers’ effective use of OER: A cross–cultural comparison by nations. Journal of Interactive Media in Education, 2020(1), 1–13. https://doi.org/10.5334/jime.576 Tang, H., & Bao, Y. (in press). Latent class analysis of K–12 teachers’ barriers to implementing OER. Distance Education. Tang, H., & Carr–Chellman, A. (2016). Massive open online courses and educational equality in China: A qualitative inquiry. Journal of Educational Technology Development and Exchange, 9(1), 49–66. https://doi.org/10.18785/jetde.0901.04 Tang, H., Lin, Y. J., & Qian, Y. (2020). Understanding K‐12 teachers’ intention to adopt open educational resources: A mixed methods inquiry. British Journal of Educational Technology, 51(6), 2558–2572. https://doi.org/10.1111/bjet.12937 Tang, H., & Wang, S. (2019). Revisiting the myth of disruptive innovation: An exploratory study of Chinese Learners’ Perception of Massive Open Online Courses. In S. Wang, L. Lin, T. Hartsell, H. Zhan, & J. Beedle (Eds.), Proceedings of the 2019 Eighth International Conference on Educational Innovation through Technology (pp. 71–77). The Institute of Electrical and Electronics Engineers. https://doi.org/10.1109/EITT.2019.00023 Tang, H., Xing, W., & Pei, B. (2018). Exploring the temporal dimension of forum participation in MOOCs. Distance Education, 39(3), 353–372. https://doi.org/10.1080/01587919.2018.1476841 Tang, H., Xing, W., & Pei, B. (2019). Time really matters: Understanding the temporal dimension of online learning using educational data mining. Journal of Educational Computing Research, 57(5), 1326–1347. https://doi.org/10.1177/0735633118784705 Xing, W., Chen, X., Stein, J., & Marcinkowski, M. (2016). Temporal predication of dropouts in MOOCs: Reaching the low hanging fruit through stacking generalization. Computers in Human Behavior, 58, 119–129. https://doi.org/10.1016/j.chb.2015.12.007 Xing, W., Tang, H., & Pei, B. (2019). Beyond positive and negative emotions: Looking into the role of achievement emotions in discussion forums of MOOCs. The Internet and Higher Education, 43, Article 100690. https://doi.org/10.1016/j.iheduc.2019.100690 Zhang, J., Skryabin, M., & Song, X. (2016). Understanding the dynamics of MOOC discussion forums with simulation investigation for empirical network analysis (SIENA). Distance Education, 37(3), 270–286. https://doi.org/10.1080/01587919.2016.1226230 Zhu, M., Bergner, Y., Zhang, Y., Baker, R., Wang, Y., & Paquette, L. (2016). Longitudinal engagement, performance, and social connectivity: a MOOC case study using exponential random graph models. In Proceedings of the Sixth International Conference on Learning Analytics & Knowledge (pp. 223– 230). Association for Computing Machinery. https://doi.org/10.1145/2883851.2883934 Corresponding author: Hengtao Tang, htang@mailbox.sc.edu Copyright: Articles published in the Australasian Journal of Educational Technology (AJET) are available under Creative Commons Attribution Non-Commercial No Derivatives Licence (CC BY-NC-ND 4.0). Authors retain copyright in their work and grant AJET right of first publication under CC BY-NC-ND 4.0. Please cite as: Tang, H., & Xing, W. (2022). Massive open online courses for professional certificate programs? Perspectives on professional learners’ longitudinal participation patterns. Australasian Journal of Educational Technology, 38(1), 136-147. https://doi.org/10.14742/ajet.5768 https://doi.org/10.19173/irrodl.v21i3.4750 https://doi.org/10.1007/s11423-021-09939-w https://doi.org/10.5334/jime.576 https://doi.org/10.18785/jetde.0901.04 https://doi.org/10.1111/bjet.12937 https://doi.org/10.1109/EITT.2019.00023 https://doi.org/10.1177/0735633118784705 https://doi.org/10.1016/j.chb.2015.12.007 https://doi.org/10.1016/j.iheduc.2019.100690 https://doi.org/10.1080/01587919.2016.1226230 https://doi.org/10.1145/2883851.2883934 mailto:htang@mailbox.sc.edu https://creativecommons.org/licenses/by-nc-nd/4.0/ https://doi.org/10.14742/ajet.5768 Introduction Literature review Potentials of MOOCs as professional degree programs Concerns about employed professionals’ participation in MOOCs Temporal dimension of learner participation Methodology Dataset Variables Data analysis Results What is the influence of learners’ longitudinal patterns of participation on their performance in a MOOC? Discussion Conclusions Acknowledgements References