An empirical investigation of the antecedents of learner-centered outcome measures in MOOCs

Abstract

This research revealed the antecedes of two learner-centered outcome measures of success in massive open online courses (MOOCs): learner satisfaction and learner intention-fulfillment. Previous studies used success criteria from formal education contexts placing retention and completion rates as the ultimate outcome measures. We argue that the suggested learner-centered outcomes are more appropriate for measuring success in non-formal lifelong learning settings because they are focused on the learner’s intentions, rather than the intentions of the course developer. The behavioural measures of 125 MOOC participants who answered a pre- and a post-questionnaire were harvested. The analysis revealed that learner satisfaction was directly affected by: the importance of the MOOC’s benefits; online self-regulated learning - goal setting; number of video lectures accessed; and, perceived course usability. Age and the number of quizzes accessed indirectly effected learner satisfaction, through perceived course usability and through number of video lectures accessed. Intention-fulfillment was directly affected by: gender; the importance of the MOOC’s benefits; online self-regulated learning - goal setting; the number of quizzes accessed; the duration of participation; and, perceived course usability. Previous experience with MOOCs and the importance of MOOC’s benefits, indirectly affected intention-fulfillment through the number of quizzes accessed and perceived course usability.

Introduction

Lifelong learning received extensive support from recent technological developments such as online learning in general, and MOOCs in particular (Kalz, 2015). This development is accompanied by controversy. One key criticism of MOOCs is the high drop-out rates (Gardner & Brooks, 2018; Reich & Ruipérez-Valiente, 2019). These rates are, on average, 93% (Chuang & Ho, 2016; Jordan, 2014; Margaryan, Bianco, & Littlejohn, 2015). Furthermore, most MOOC participants who earn certificates for completing the course are experienced learners with a strong academic background (Christensen et al., 2013; Daily, 2014; Guo & Reinecke, 2014; Hansen & Reich, 2015; Koller, Ng, Chuong, & Zhenghao, 2013; Reich & Ruipérez-Valiente, 2019). Though it is true that MOOC dropout rates are very high, the question is whether completion rate is the appropriate measure for evaluating the success of this new form of lifelong learning. Completion rate is a success criterion borrowed from formal education contexts where students enroll in courses with the goal of completing them, and of satisfying the learning outcomes defined by the instructor. Rather, students may enroll in MOOCs for a variety of reasons (Littlejohn, Hood, Milligan, & Mustain, 2016; Onah, Sinclair, & Boyatt, 2014; Wang & Baker, 2018), and MOOC participants may have a variety of expected learning outcomes. For example, MOOC participants may achieve their learning goals by engaging in only a segment of the course (Ho et al., 2015; Liyanagunawardena, Parslow, & Williams, 2013). It has been proposed that the success of lifelong learning in MOOCs should be evaluated not through traditional, instructor-focused measures such as completion rates, but rather through more learner-centered measures such as learner satisfaction and the fulfillment of learner intentions (Henderikx, Kreijns, & Kalz, 2017; Reich, 2014).

Learner satisfaction, and intention fulfillment

Learner satisfaction reflects students’ perception of their learning experience (Kuo, Walker, Schroder, & Belland, 2014; Littlejohn et al., 2016) and is defined as a student’s overall positive assessment of his or her learning experience (Keller, 1983). While some authors have found positive correlations of student satisfaction with post-secondary student success (Chang & Smith, 2008), and a positive relationship between learning satisfaction and the intention to use e-learning (Liaw & Huang, 2011; Roca, Chiu, & Martínez, 2006), a recent study of student data of the Open University by Rienties and Toetenel (2016) has found that retention and satisfaction are not correlated. The authors explain these findings from a formal distance education context with the fact that learning is not always fun, and requires effort. While this explanation is relevant in the context of degree-seeking learners in formal education, the role of learner satisfaction in the open learning context of MOOCs should not be underestimated, as these learners participate in the courses for different reasons than degree-seeking students in formal education.

Another success criterion that has been proposed is learner intention-fulfillment. Intention-fulfillment emerges as a promising success measure of open courses and MOOCs, since it takes into account the personal objectives that the learners intend to achieve, rather than external success criteria (Henderikx et al., 2017). In MOOCs and in other forms of open education, a successful learning experience can take a variety of forms, ranging from viewing a single lecture, attaining a specific skill, or studying a topic of interest, to studying a whole course and fulfilling all of its requirements.

This study focuses on learner satisfaction and learner intention-fulfillment as two learner-centered success measures and examines the factors that impact these subjective success measures in the context of a mid-sized (circa 2000 participants) MOOC on the recent history of the Middle-East. The goal of this study is to identify key factors contributing to MOOC learner satisfaction and intention-fulfillment. We examine how these two dependent variables are predicted by personal learner characteristics (demographic characteristics, previous experience with MOOCs), learner dispositions (self-regulated learning, course outcome beliefs), learner behaviour in the MOOC (e.g. number of video lectures accessed, number of quizzes accessed), and perceived course usability (e.g. ease of navigation, website usability). Understanding the predictors of the two success criteria, learner satisfaction and intention-fulfillment, will contribute to theories of learner motivation and behavior in open online environments, as well as help create more personalized courses and provide lifelong learners with better support in open learning contexts.

The rest of the paper is organized as follows. After a review of related work on possible predictors of the learner-centered outcome variables, we propose research hypotheses. This is followed by the research model, a Method section detailing the research methods, the participants, and the instruments used for data collection, and a Data analysis section. We conclude with the results and a discussion of the findings.

Predictors of learner satisfaction and of intention-fulfillment

Demographic background

MOOC learners are a heterogeneous group, comprising of male and female participants of all ages, from across the world, with different educational, socio-economic and psychological characteristics (Chuang & Ho, 2016; Koller, Ng, Do, & Chen, 2013). The diversity of MOOC learners has been discussed in several studies. Some earlier studies did not identify an influence of gender on achievement or on completion rates (Breslow, Pritchard, & DeBoer, 2013; Cisel, 2014; R. Kizilcec, Piech, & Schneider, 2013; Morris, Hotchkiss, & Swinnerton, 2015), while other studies, such as Garrido, Koepke, Anderson, and Mena (2016) found that women are more likely than men to complete a MOOC or obtain certification.

Furthermore, there are inconsistent findings about the association between age and academic achievement. Guo and Reinecke (2014) found a positive correlation between age and grades, while Breslow et al. (2013) did not find such a correlation. In an examination of completion rates, Morris et al. (2015) found that course completers were on average older, while those who dropped out in the first week of the course were on average the youngest group. Based on these findings, we propose the following hypotheses:

• Hypothesis 1 Gender will be associated with learner behaviour in the course, and with the course outcomes: learner satisfaction and participant intention-fulfillment.

• Hypothesis 2 Age will be positively associated with a higher level of participant activity in the course, with higher perceived usability and with better course outcomes: learner satisfaction and participant intention-fulfillment.

Previous experience in MOOCs

Scholars highlight the high level of previous knowledge and competencies needed to be successful in a MOOC (Santos, Costa, & Aparicio, 2014). Most of the participants who earn certificates for completing MOOCs are experienced learners with a strong academic background (Christensen et al., 2013; Daily, 2014; Guo & Reinecke, 2014; Hansen & Reich, 2015; Koller, Ng, Chuong, & Zhenghao, 2013). Based on these findings we propose the following hypothesis:

• Hypothesis 3 Previous experience in MOOCs will be positively associated with a higher level of participant activity in the course, and with better course outcomes: learner satisfaction and participant intention-fulfillment.

Course outcome beliefs

Course outcome beliefs is a variable that describes the expectations learners have regarding the outcomes of participating in the MOOC. A person may believe that taking a MOOC will result in positive outcomes such as more opportunities in the labor market or negative outcomes such as losing leisure time or creating stress. Those outcome beliefs can affect the learner’s behaviour as well as the learner’s evaluation of the course.

• Hypothesis 4 Positive course outcome beliefs will be positively associated with a higher level of participant activity in the course, and with better course outcomes: learner satisfaction and participant intention-fulfillment.

Self-regulated learning

MOOCs and other forms of open online education are not only open in access, but also open for participants to choose their learning behavior, their learning path and their learning schedule (Kizilcec, Perez-Sanagustín, & Maldonado, 2017; Margaryan et al., 2015; Van den Beemt, Buijs, & Van der Aalst, 2018). The self-paced nature of online courses treats the learner as an active agent, and provides learners with the freedom to select and control the resources and tools that they are using. Thus, online learning requires a high level of self-regulation. Zimmerman (2000) defines self-regulation as “self-generated thoughts, feelings, and actions that are planned and cyclically adapted to the attainment of personal goals” (p. 14). Self-regulation is a context-specific process. In the context of learning, self-regulated learning (SRL) is defined as students’ proactive actions aimed at acquiring and applying information or skills that involve setting goals, self-monitoring, time management and regulating one’s efforts towards learning goal fulfillment (Järvelä, Malmberg, & Koivuniemi, 2016; Reimann, Markauskaite, & Bannert, 2014; Tabuenca, Kalz, Drachsler, & Specht, 2015; Zimmerman, 1990). Several studies found a positive correlation between SRL and satisfaction in online courses (Artino, 2007; Li, 2019; Puzziferro, 2008). Learners, who do not regulate their learning process, may experience increased dissatisfaction (Sun & Rueda, 2012). As well, goal setting and strategic planning were positively predict goal attainment in MOOCs and help seeking negatively predicts goal attainment (Kizilcec et al., 2017). These findings lead us to the following hypothesis:

• Hypothesis 5 Higher self-regulated learning capabilities will be positively associated with a higher level of participant activity in the course, and with better course outcomes: learner satisfaction and participant intention-fulfillment.

Perceived course usability

The usability of the course website as perceived by the user (perceived course usability) is influenced by factors such as the user’s perception of the course website, and the organization of the course materials into logical and understandable components (Eom, Wen, & Ashill, 2006). Usability refers to whether a system can be used with effectiveness and efficiency to enable users to achieve specified goals in a particular context of use (ISO 9241-11, 1998). Usability affects students’ learning effectiveness and overall learning experience, and the level of usability affects the satisfaction level and the learning outcomes of distance learners (Eom et al., 2006). We thus hypothesize that:

• Hypothesis 6 The level of perceived course-usability will be positively associated with better course outcomes: participant learner satisfaction and intention-fulfillment.

Learning behaviour in MOOCs

In open learning environments like MOOCs learners can study when, where and how they wish, alone or with others, and with fewer restrictions on time or space compared to traditional online-courses. Learning behaviour in MOOCs is mostly visible through access and usage patterns of different types of resources. The participants can learn in different learning sequences by watching video lectures and by interacting with different MOOC resources (Van den Beemt et al., 2018). The learning path can deviate from a linear course, learners can start the courses later than the original launch date, can view lectures several times, and do exercises and take quizzes several times. For example, initial findings suggest that successful MOOC certificate earners view only 78% of the course content and skip the rest (Guo & Reinecke, 2014). Successful certificate earners are also more engaged in non-linear navigation behaviour than non-certificate earners. They “jump” backwards to revisit earlier lectures or assessments up to three times more often than non-certificate earners (Guo & Reinecke, 2014). Davis, Chen, Hauff, and Houben (2016) showed that learners who succesfully passed the course are more interested in their quiz scores than learners who did not, and they used progress tracking tools more often. Such learning patterns show that learners who sucesfully pass the course use better self-regulated learning strategies than those who did not pass the course. Thus, we hypothesize that:

• Hypothesis 7 The number and variety of course activities performed, and the time spent on the course will be positively associated with the participant’s perceived course usability, and with better course outcomes: learner satisfaction and intention-fulfillment.

In summary, the objective of this study was to identify how MOOC participant characteristics and pre-course disposition affect participant learning behaviour in the course, as well as how these predictors affect the perceived course usability, and finally, how all of these variables predict the learning outcomes: learner satisfaction and intention-fulfilment. Figure 1 illustrates the research model of the study.

Fig. 1

A model of key factors influencing learner satisfaction and intention-fulfillment in MOOCs. The row above the model presents the timeline and the three stages of data collection

Method

Participants

Participants in a nine-week massive open online course (MOOC) of the Open University in Israel were surveyed for this study. The open access course was built on a Moodle platform and dealt with the political and sociological aspects of the “Middle East in our times”. A participant was defined as a person who enrolled to the MOOC and who participated in at least one activity in the course. The course was open and free to the Hebrew speaking public without any prerequisites, and did not award academic credit other than a certificate of participation. Each week, a new topic was opened to the participants and the participants were able to watch video lectures, answer multiple choice questions and quizzes, and respond to discussion questions in discussions forums. Participants who fulfilled all of the course assignments received a certificate of participation without charge.

During the course, participants’ online activities were recorded in a log-file. All MOOC participants were invited to respond to two questionnaires: a pre- and a post-course online survey. The survey was not mandatory. Of the 2007 participants who enrolled to the course and participated in at least one activity, 377 (18.7%) participants responded to the pre-questionnaire and 190 (9.5%) participants responded to the post-questionnaire. In total, 125 (6.2%) participants took part in all three stages of the study by answering the pre-questionnaire, participating in at least one MOOC activity, and answering the post-questionnaire. This paper reports findings on this group.

The participants in this study were diverse. Ages ranged from 18 years old to 85 (M = 61, SD = 14.01). Course participants under the age of 18 were not included in the study. 56% of the participants were male and 44% were female. On average, the participants rated themselves as highly skilled internet users on a scale of 1–7 (Very low to very high Internet skills) (M = 6.23, SD = 0.65), though a majority (63.7%) reported that this MOOC was their first online learning experience. The sample of participants included in the study is demographically similar to the population which enrolled in the MOOC: Age and gender of the participants were compared to demographics reported by 457 (23%) course enrollees who responded to a short survey at the beginning of the course that included questions about the gender and the age of the participants. The participant pool was not significantly different from the enrollee pool: A Chi-square test reveals that the gender distribution in the two samples was similar (Chi²₍₁₎ = 0.04, p = .84, Male_{brief survey} = 57%), and t-test for independent samples revealed that there are no differences in the participants age (t₍₅₈₀₎ = 0.83, p = .41, M_{brief survey} = 59.97, SD = 16.02). The similarity in gender and age between the large sample and the research sample enables us to generalize the research results beyond the sample of participants who met all of the inclusion criteria.

Assessments and measures

As explained in Participants section, this study comprised three stages of data collection: a pre-course questionnaire, behavioural data collected from log-files, and a post-course questionnaire. All MOOC participants were invited to answer the pre-course questionnaire via email immediately after they had enrolled to the MOOC, and a reminder was sent after 1 week to those who did not yet respond to the questionnaire. In addition, an invitation to participate in the pre-course questionnaire was posted to the MOOC bulletin board. All participants were informed that responding to questionnaire is voluntary, and signed an informed consent before taking the questionnaire. Similarly, on the last week of the MOOC, all enrollees were invited to the post-questionnaire by email, with a reminder after 1 week. A unique identifier connected the survey responses and the behavioural data. An anonymization process had been implemented ahead of the statistical analysis.

Pre-questionnaire

The pre-questionnaire included three sections – demographics, course outcome beliefs, and online SRL.

Demographics

Participants reported gender, age, and number of MOOCs previously taken. Gender was a two-category variable with male coded as 1 and female coded as 2. Age was reported in years. Previous experience with MOOCs was measured by the number of MOOCs that the participants took up to the time of the survey. The variable was coded as a dummy variable – Took (‘1’) or did not take (‘0’) at least one MOOC in the past.

Course outcome beliefs

Two indices measured course outcome beliefs. (A) Importance of the benefits of participating in the MOOC (‘importance of MOOC’s benefits’): Eighteen items including statements such as ‘will increase my chances in the labour market’, and ‘will allow me to do my job better’ (Cronbach’s alpha = .92). (B) Importance of the disadvantages of participating in the MOOC (‘importance of MOOC’s disadvantages’): Five items including statements such as ‘will limit my free time with family and friends’, ‘will force me to buy a new multimedia computer’ (Cronbach’s alpha = .74).

Online SRL

Two indices measuring online self-regulation were adapted from the OLSQ (Barnard, Lan, To, Paton, & Lai, 2009). (A) Goal setting: The ability to set goals for the learning process (e.g. ‘I maintain a high standard of learning in my online courses’, and ‘I set short-term (daily or weekly) goals as well as long-term goals (monthly or for the semester) ‘) (Cronbach’s alpha = .87). (B) Environmental structuring: The ability to arrange the location for studying (e.g. ‘I choose the location where I study to avoid too much distraction’, ‘I find a comfortable place to study’) (Cronbach’s alpha = .85). Each index included five items. The other four self-regulation scales had not been used in order to decrease the load on the participants. All the indices were on a 7-point Likert scale.

Behavioural measurements

The behavioural measurements were extracted from the log file of the course. The measurements included: (A) The number of video lectures that the participant accessed during the course (‘number of videos accessed’). The participants could access the same video lecture more than once. (B) The number of quizzes that the participant accessed during the course (‘number of quizzes accessed’). The quizzes were self-evaluation activities that enabled participants to assess their knowledge. The participants could access the same quiz more than once. (C) The number of discussion forums that the participant accessed during the course (‘number of forums accessed’). Participants who asked to get a certificate of course completion were asked to post at least two comments to a weekly discussion forum. (D) The duration of time the MOOC was taken (‘duration of participation’). This measure was calculated by subtracting the time of the last log-on of the participant in the course, from the time of the first log-on. (E) The total number of MOOC activities the participant participated in, including the number of lectures, quizzes and forums accessed (‘number of activities accessed’). This behavioural measurements were log-transformed in order to get a normal distribution of the variable. (F) Whether the participant received a certificate of course completion (‘receiving completion certificate’). The criteria for receiving the certificate were based on completing a minimal quota of course activities.

The post-questionnaire

The post-questionnaire included three sections – perceived course usability, learner satisfaction and intention-fulfillment.

Perceived course usability

Seventeen items on 7-point Likert scale ranging from 1 ‘totally don’t agree’ to 7 ‘strongly agree’, including items such as ‘It is easy to learn to use this MOOC virtual learning environment’, ‘I know where to go in this MOOC virtual learning environment’ (Cronbach’s alpha = .84).

Learner satisfaction

Single item on a Likert scale ranging from 1 ‘very unsatisfied’ to 7 ‘extremely satisfied’: ‘How satisfied have you been with this MOOC?’

Intention-fulfillment

Four items on 7-point Likert scale ranging from 1 ‘totally don’t agree’ to 7 ‘strongly agree’, including items such as: ‘I achieved my personal learning goals by participating in this MOOC’, ‘the MOOC met my expectations’ (Cronbach’s alpha = .89).

Data analysis

Bivariate correlation analyses (Pearson product moment) were performed in order to identify the predictors of learner satisfaction and intention-fulfillment. The Pearson correlation coefficient (r), ranging between − 1 and + 1, indicates the strength and the direction of the relationship of each independent variable with the other independent and dependent variables.

As a preliminary step in preparation for the linear regression analysis, the correlations between the independent variables were evaluated to identify multicollinearity. Afterwards, stepwise hierarchical linear regression models were performed with learner satisfaction and intention-fulfillment as the dependent variables. Independent variables with higher than bivariate correlation of .60 were entered into the same regression model in a stepwise manner in order to avoid violation of the regression assumptions. Furthermore, in every regression analysis, variance inflation factor (VIF) and tolerance values were checked in order to find evidence of multicollinearity.

Finally, structural equation model-based PLS methodology was employed to examine the shared effect of the independent variables on each other and on the dependent variables: learner satisfaction and intention-fulfillment. PLS is well suited for this research because it is useful for early stages of theory building and testing (Chin, 1998). To reduce the model complexity, only variables that had been identified as significant predictors in the linear regression were entered the model.

Results

The study’s two dependent variables are the outcome measures of success: learner satisfaction and intention-fulfillment. Table 1 presents the results of a Pearson bivariate correlation analyses between the independent variables and the two dependent variables. Appendix includes the Pearson bivariate correlations between all the research variables. Furthermore, learner satisfaction and intention-fulfillment were found to be highly correlated (r = .78, p < .001).

Table 1 Pearson correlations between the predictor variables and the two dependent variables

Learner satisfaction

A hierarchical linear regression to predict the level of learner satisfaction was performed in four stages. In the first stage, demographic predictors – age, gender and previous experience with MOOCs entered to the regression in Enter mode. In the following steps the variables were entered in a stepwise method in order to avoid multicollinearity. The second step included predictors from the pre-questionnaire that were found to be in correlation with learner satisfaction (see Table 1). The variables entered were: the importance of MOOC’s benefits, the importance of MOOC’s disadvantages, the level of online SRL in environmental structuring and the level of SRL in goal setting. In the third step, behavioural indices that were in correlation with learner satisfaction (see Table 1) were entered. The variables entered were – the number of videos accessed, the number of forums accessed, the number of quizzes accessed, and the duration of participation. In the fourth step, the perceived level of the course’s usability was entered into the analysis. The results of the four stages are presented in Table 2.

Table 2 Stepwise linear regression predicting learner satisfaction, performed in four stages

As seen in Table 2, the more the participants set goals for their online learning and the more they perceived the importance of the benefits of taking the MOOC as high, the more they reported higher satisfaction from the course. From the behavioural measurements, the more video lectures the participant accessed, the higher their level of course satisfaction. Lastly, the higher the participants’ perceived course usability, the more they reported satisfaction from the course. All the variables together explained 42% of the variance of the learner satisfaction variable.

Intention-fulfillment

A hierarchical linear regression was performed in four stages in order to predict the level of participant intention-fulfillment. In the first stage, the three demographic predictors age, gender and previous experience with MOOCs, were entered to the regression as control variables. In the following three stages, the variables were entered in a stepwise manner in order to avoid multicollinearity. In the second stage, predictors from the pre-survey that were found to correlate with the level of intention-fulfillment (see Table 1) were entered. These variables were: the importance MOOC’s benefits, and the level of online SRL: environmental structuring and goal setting. In the third stage, behavioural predictors that were found to correlate with the level of intention-fulfillment (see Table 1) were entered. The variables entered were: number of videos accessed, number of forums accessed, number of quizzes accessed, and duration of participation. In the fourth stage the perceived course’s usability, was entered. The results of the four stages are presented in Table 3.

Table 3 Stepwise linear regression predicting intention-fulfillment, performed in four stages

As seen in Table 3, female participants reported a higher level of intention fulfillment than male participants. The more participants set goals for their online learning and the more they perceived the importance of the benefits of taking a MOOC to be high, the more they reported higher intention-fulfillment.

From the behavioural measurements, the longer the duration of participation, and the higher the number of quizzes accessed during the course, the more they reported higher intention-fulfillment. Lastly, the higher the participants` perceived course usability, the more they reported that their intentions were fulfilled. All the variables together explained 37% of the variance of the intention-fulfillment variable.

Prediction of learner satisfaction and intention-fulfilment with SEM analysis

Structural equation modeling (SEM) - based PLS methodology with maximum-likelihood estimation was conducted using Amos 22 in order to model the relationships between the variables. Missing variables were rare and were imputed using regression estimation (Schreiber, 2008). The variables for the SEM analysis were selected based on the significant correlations identified in Table 1, and the significant coefficients identified in Tables 2 and 3. The results are presented in Fig. 2. All paths in the model are significant, and non-significant paths were removed. Sample size had been found sufficient for the number of the variables that had entered into the model (Bentler & Chou, 1987; Tabachnick & Fidell, 2001). The model goodness of fit is satisfactory (Chi²₍₃₃₎ = 40.29, p = .18, CFI = .97, TLI = .94, NFI = .88, RMSEA = .03). The model explained 36% of the variance of intention-fulfilment and 25% of the variance of the learner satisfaction. The results of the SEM analysis demonstrate several phenomena on the effects of the study’s independent variables on learner satisfaction and intention-fulfillment.

Fig. 2

SEM analysis presenting the significant factors influencing learner satisfaction and intention-fulfillment. Note: All the paths are significant at a p < .05 level. The estimations are presented using a standardized coefficient

The demographic variables influence the dependent variables (DVs) in several ways. Gender had a direct effect on intention-fulfilment, but not on the level of satisfaction. Female participants report that they fulfil their intentions more than males, but there are no significant differences between female and male participants in the level of learner satisfaction. On the other hand, age did not have a direct effect on the DVs, but rather affected the number of videos lectures the participants accessed. There was a positive correlation between participant age and the number of video lectures viewed. Further to that, the number of video lectures viewed was positively correlated with the level of learner satisfaction. In summary, older age predicted viewing more video lectures, which in turn predicted a high level of learner satisfaction. In contrast, age did not predict the level of intention-fulfilment, neither directly or indirectly.

The number of MOOCs previously taken did not directly predict the DVs. Rather, it predicted the number of quizzes the participant accessed. Those who participated in at least one MOOC in the past accessed more quizzes than those who did not participate in a MOOC in the past. The number of quizzes accessed was directly positively correlated with the level of intention-fulfilment. It was also indirectly correlated with the level of intention-fulfillment, as it was mediated by the level of perceived usability of the course.

The positive outcome beliefs and the goal setting variable affected the DVs directly as well as indirectly. The level of the importance of MOOC’s benefits positively affected the level of intention-fulfilment, and thus participants who expected to gain benefits from their participation in the MOOC were more likely to report that they fulfilled their intentions. Another interesting finding is that of the importance of MOOC’s benefits on the number of weekly quizzes that the participant accessed: Participants who expected a positive outcome from the MOOC, accessed more quizzes than those who didn’t expect positive outcomes. The number of quizzes accessed, as was already mentioned, is positively correlated with intention-fulfilment. Note though that the number of quizzes accessed had no significant mediating effect on the connection between the importance of MOOCs’ benefits and the level of intention-fulfilment (Sobel Z = 1.35, p = .09). Interestingly, the level importance of MOOC’s disadvantages did not have an effect on the DVs and for that reason is not shown in the final model.

Participants, who regulated their learning by setting goals, reported higher levels of learner satisfaction and intention-fulfilment without any mediation by the behavioural variables. On the other hand, the ability to self-regulate learning by structuring the learning environment, did not affect the DVs and is thus not shown in the model.

Surprisingly, the duration of participation was not affected by any pre-course variables. On the other hand, duration of participation positively affected the level of intention-fulfilment, but not learner satisfaction. Lastly, the perceived course usability was predicted only by the number of quizzes the participant accessed, and it was positively correlated with both learner satisfaction and intention-fulfilment.

Discussion

The goal of this research was to better understand the predictors of two important learner-centered outcome measures of success in massive open online courses: learner satisfaction and learner intention-fulfillment. In contrast with previous studies, which focused on the fulfillment of the course developers’ intentions and placed retention and completion rates as the ultimate outcome measures, we place learner satisfaction and learner intention-fulfillment as alternative course outcome measures, which are more appropriate for measuring success in the non-formal lifelong learning context of MOOCs.

Participants in a mid-sized MOOC filled out pre-and post-questionnaires and data about their behaviour in the MOOC were collected from the course log files. This study used educational data mining and learning analytic techniques to understand how participants’ demographics, their pre-course characteristics when entering the course, their actual behaviour in the course and their perceived course usability predict the two learner-centered outcome variables which describe the learners’ level of satisfaction and the extent to which the MOOC enabled them to fulfill their intentions. Furthermore, despite the relatively high correlation between these two outcome variables (r = .78), our findings distinguished between two distinct pathways though which the participants achieved these outcomes. These pathways are presented in Fig. 2, and elaborated below.

Learner satisfaction was directly and positively affected by four variables: two pre-course variables: the importance of the benefits of taking a MOOC, and online SRL-goal setting; one behavioural variable: number of video lectures accessed; and, one post course variable, perceived course usability. Furthermore, there were two indirect effects on learner satisfaction, through perceived course usability and through number of video lectures accessed. The first path begins with previous experience with MOOCs, which positively influenced the number of quizzes, and which, in turn, positively affects perceived course usability. The second path shows that age positively affected the number of video lectures accessed.

Intention-fulfillment was directly and positively affected by six variables. Gender directly affected the level of intention-fulfilment. The two pre-course variables were: the importance of the benefits of taking a MOOC, and online SRL-goal setting; two behavioural variables were the number of quizzes and the duration of time taking the MOOC; and, the post-course variable - perceived course usability. Female participants reported higher levels of intention-fulfillment. Furthermore, previous experience with MOOCs and the importance of the advantages of taking MOOCs, indirectly affected intention-fulfillment through the number of quizzes and the perceived course usability.

Our findings shed new light on the role of the demographic variables on course outcomes. Similarly to the findings of Garrido et al. (2016) who found that women are more likely than men to complete a MOOC or obtain certification, our findings demonstrate that gender had an effect on one of the learner centric outcome variables, by positively affecting the intention-fulfillment variable. Females had a higher level of intention-fulfillment than men did. Further research should explore whether this can be generalized beyond the specific context of this MOOC. In regards to age, our findings are similar to those of Morris et al. (2015) who found that older participants persist in their online studies more then young participants. Similarly, in our study, age was not a direct predictor of course outcomes, but rather predicted a behavioural variable that reflects progress in the course: the number of video lectures that the participants accessed during the course, which in turn predicted learner satisfaction.

As can be seen in Fig. 2, the level of importance of the benefits of participating in the MOOC predicted both of the learner-centered outcome variables. It had a direct positive influence on both satisfaction and intention-fulfillment, as well as an indirect positive influence on the number of quizzes taken, which in turn influenced intention-fulfillment directly and satisfaction indirectly. The MOOC did not provide any credit beyond a certificate of completion, and we thus can see how lifelong learners who give the advantages provided by the MOOC a higher value, are likelier to invest more in the course, and to achieve positive outcomes. An applied implication of this finding is the importance of clearly delineating the MOOCs benefits and contributions in a way that allows participants to evaluate the relevance of the MOOC for their personal goals.

A strong and positive impact of goal setting on course outcomes was identified. As Zimmerman (2002) mentioned, the ability to set learning goals is an internal structure that is based on the learner abilities, and can be learned throughout one’s life. Interestingly, our findings did not identify that those correlations were mediated by any of the behavioural variables.

Another thought-provoking finding of this study is the difference between the behavioural variables that influenced learner satisfaction and those that influenced intention-fulfillment. The number of video lectures accessed positively predicted learner satisfaction, while the level of intention-fulfillment was directly predicted by the number of weekly quizzes accessed, and by course duration. Accessing video lectures is a passive learning behaviour, while taking self-assessment quizzes, and to a lesser extent persisting in the course, are more active aspects of learner behaviour. A possible insight is that more active course components, such as self-assessment quizzes that provide participants with feedback on their achievements and understanding, assist learners who are focused not only on enjoying the course (i.e. learner satisfaction) but also on using the course to fulfill the personal intentions they had when they set out to study the MOOC (intention-fulfillment).

Finally, perceived course usability was a strong predictor of both course outcomes. This finding reflects the fact that a course with poor usability will delay the learner’s progress, and decrease the personal benefits from participating in it (Eom et al., 2006). The only direct predictor of perceived course usability was the number of quizzes taken, which, as discussed in the previous paragraph, is also an important predictor of the key outcome variables.

Several limitations of this study can help drive future research. First, additional factors such as ICT skills and educational background should be examined as predictors of the course outcomes. In our study, those measures showed insufficient variability and could not be included in the analysis. Secondly, participants in our study were a unique sub-group of participants who had chosen to answer the pre- and the post-questionnaire, and not a random sample of the MOOC participants. This limitation is typical for MOOC studies that use self-reported questionnaires (Breslow et al., 2013; Kizilcec & Halawa, 2015). Nevertheless, as mentioned in the Method section, a comparison of the sample’s demographic characteristic with the demographics of the course’s population did not identify any significant differences. Since the MOOC that had been analyzed was in Hebrew, only Hebrew speaking participants had been able to participate in it. Those limitations reduce the external validity of the results. Future research should develop non-responsive methods to investigate the antecede of the two dependent variables - learner satisfaction and learner intention-fulfillment.

Conclusions

In conclusion, although the correlation between learner satisfaction and intention-fulfillment is high, the behavioural predictors for the two constructs are different. While the level of learner satisfaction was predicted by the number of video lectures accessed, the learner intention-fulfillment was predicted by the number of quizzes and by the duration of participation in the course. We can see that although these two outcome variables are important, and although they show a high level of correlation, our findings distinguish between the antecedents of these outcomes. The level of satisfaction is determined mainly by the lectures and not by other learning aspects such as evaluation mechanisms, while intention-fulfillment is determined mainly by components that allow participants to self-assess their learning activities.

Finally, following the critiques of Reich (2015) who stated that research to date had little impact on educational practice and the critiques of Pardo, Han, and Ellis (2016) who pointed out that using educational data mining without a theoretical framework reduces the ability of translating the results into a meaningful pedagogical guidance, we would like to suggest that the educational impact of our results is that they propose a deeper, theory-supported, understanding of student perception of the courses and of their outcomes. This emphasis on the student’s perspective is essential when discussing lifelong learning. Our findings demonstrate the importance of learners’ ability to set goals in order to self-regulate their learning, and the importance of clearly stating the benefits of the MOOC, while providing participants with tools to evaluate their achievements during the course. Course designers and developers should not only develop excellent learning materials, but also assist MOOC participants to set their goals and to evaluate the potential benefits of the course.

Appendix

Table 4 Pearson correlations between all the research variables