Dispositional learning analytics and formative assessment: an inseparable twinship

Abstract

Educational innovations, particularly those in online education and technology-enhanced learning, some accelerated by the recent pandemic, take centre stage in this journal. Examples include the resurgence of the flipped classroom methodology, supported by instructional technology, the utilization of formative assessment with technological assistance to provide effective learning feedback, and the integration of learning analytics within technology-enhanced learning. Despite empirical studies addressing these advancements individually, they tend to focus predominantly on the initial phase of learning feedback: where the learner currently stands. This paper contends that isolated depictions of formative assessment overlook the broader progress in technology-enhanced learning. Our contribution highlights the significance of dispositional learning analytics, which combines learning and learner data to offer insights into learners' current status, future trajectory, and methods to facilitate their advancement. We advocate for strengthening the integration of formative assessment data within learning analytics to fully harness their combined potential. An empirical study is presented to illustrate the predictive value of formative practice behaviour on module performance and explore the link between students' learning dispositions and their engagement with e-tutorials. By delineating student profiles based on practice behaviour and examining variations in learning dispositions, our aim is to enhance our comprehension of effectively supporting learners in technology-enhanced educational environments.

Introduction

In addition to all the damage that the COVID-19 pandemic has inflicted on education worldwide, the pandemic has certainly also had a favourable effect on educational innovations. In this journal, these innovations have been extensively discussed, in the field of online education and the use of technology-enhanced learning (Sum & Oancea, 2022). The resurgence of the flipped classroom methodology, which involves transferring learning activities from inside the classroom to outside the classroom with the aid of instructional technology, has been notably highlighted (Divjak et al., 2022). When initial learning occurs beyond direct teacher supervision, the provision of effective learning feedback becomes essential. Formative Assessment (FA), once again supported by technology, largely addresses this need for learning feedback (Lu & Cutumis, 2022). However, to provide learning feedback that not only aids learners in their learning but also assists teachers in their teaching, the integration of learning analytics (LA) alongside the utilization of appropriate LA dashboards is necessary (Kaliisa et al., 2023).

The surge in the deployment of these forms of Technology-Enhanced Learning (TEL) has been extensively documented, both in this journal and beyond. However, these developments are almost always described in isolation: the success of LA sec, or of formative assessment, or flipped learning. In William's groundbreaking work on formative assessment (Wiliam, 2011; Wiliam & Thompson, 2008), the suggestion is made that formative assessment can be understood through three primary processes: identifying the current state of learners' understanding (“where the learner is now”), determining their destination (“where the learner is going”), and charting the path to reach it (“how to get there”). These processes are facilitated by three groups of actors: teachers, learners, and peers. The intersection of these dimensions yields five fundamental strategies for formative assessment. These strategies span from the dissemination of information by various actors regarding learners' current status to the mobilization of learners by different actors to facilitate their progress towards the learning objectives. Yet, it appears that applications of learning analytics primarily centre on the process of understanding “where the learner is now”, and lack the understanding how formative assessment could be used “where the learner is going” and “how to get there”. In this contribution, we will argue that such isolated descriptions of formative assessment fail to capture the broader advancements in the field of TEL: it is precisely the interplay of individual functionalities that leads to added value.

Our contribution is built on the powerful role that dispositional learning analytics (DLA) can play in learning. What distinguishes DLA from conventional LA is its distinctive integration of learning data and learner data: the traces of learning activities, as found in all LA applications, are coupled with self-reported survey data on learners, aligning with the established research tradition in education (Buckingham Shum & Deakin Crick, 2012; Tempelaar et al., 2015). The potential role of LA in fostering dispositions, or future competences as termed by Kleimola and Leppisaari (2022), represents one facet of the issue. The complementary aspect concerns the role learning dispositions can play in the learning process, where the learner is now, where they are going, and how to get there, which constitutes the primary focus of our contribution. Our intent is to reinforce and expand the pivotal role of FA data within LA. The synergy between LA and FA is undeniable: effective LA implementations necessitate FA data as a principal driving force, while FA’s full potential is unlocked only when integrated within an LA context that offers comprehensive feedback extending beyond cognitive metrics (Knight et al., 2020; Kohnke et al., 2022). Within the scope of furnishing such comprehensive feedback, we hypothesize that learning dispositions, denoted by the ‘D’ in DLA, assume a vital and irreplaceable function.

The argumentative framework we adopt in this empirical study consists of two sequential steps. The first step involves demonstrating the significance of FA (Formative Assessment), assuming the form of practicing in test-driven e-tutorials in this study. What predictive value does this practice behaviour have for module performance? And, crucially from the standpoint of early intervention, what predictive value does this practice behaviour measured at the beginning of the module have for later module performance? In this first step, variable-oriented models are employed, obtained through path analysis. In the second step, where we utilize person-oriented models, the question is posed: what is the relationship between students' learning dispositions and the intensity and timing of their use of the e-tutorials? To do this, we distinguish between different student profiles based on observed practice behaviour in the e-tutorials, and subsequently investigate how learning dispositions vary among those profiles.

Literature review

Dispositional learning analytics (DLA)

To reveal the inseparable connection between dispositional learning analytics (DLA) and formative assessment (FA), the first step is to define DLA. DLA introduces a LA framework that amalgamates learning data originating from learning activities, obtained through traces within technology-enhanced learning systems, with dispositions representing learner-centric data encompassing student learning competences (Kleimola & Leppisaari, 2022): learning cognitions, behaviours, values, and attitudes, gauged via self-report surveys (Buckingham Shum & Deakin Crick, 2012). In works by Buckingham Shum and Deakin Crick (2012) and Buckingham Shum and Ferguson (2012), learner data is sourced from a dedicated survey instrument meticulously crafted to assess learning power—the interplay of skills, experiences, social interactions, values, and attitudes shaping the learning engagement. Drawing inspiration from the systematic approach presented in studies by Buckingham Shum and their colleagues, this study aims to put dispositions into operation using instruments derived from contemporary educational research, thus fortifying the link with educational theory. This endeavour builds upon the author’s prior investigations into the influence of learning dispositions on students’ educational decisions within blended or hybrid learning contexts (Tempelaar et al., 2015, 2017, 2018). In these prior publications, we demonstrated that information obtained from trace data is highly complementary to information obtained from survey data, which strongly advocates for the use of both types of data in LA applications. The only restriction on the use of surveys is that they somewhat disrupt the natural learning process, and their implementation must therefore be appropriately integrated into the learning context.

Regarding the other data component of LA: within trace data, we typically distinguish between process and product data (Tempelaar, 2023). Process data pertains to activities, such as accessing a theory page explaining a mathematical principle or beginning to solve a mathematical problem. Product data pertains to cognitive outcomes, such as proficiency in solving a mathematical problem or the results of computerized assessments. For all three data components used in DLA—trace data of both product and process types, as well as disposition data—the balance between timeliness and predictive power is crucial (Tempelaar, 2023). To provide meaningful learning feedback, this feedback must be predictive of learning outcomes. Generally, the longer we take, the more predictive and actionable the data collected throughout the module becomes. However, taking action requires time, necessitating a balance between the time spent collecting powerful data and the time available for taking action.

There has been some debate about the type of data that should be included in LA studies—whether all data with predictive power or only objective data like trace data, excluding subjective data collected through surveys (Tempelaar, 2023). This dispute has never been fully resolved, though many contemporary LA studies incorporate both types of data. However, studies that use disposition data with as rich a composition as the trace data applied remain relatively scarce.

Assessment as-, for-, and of learning

That same balance between timeliness and informativeness applies even more strongly to assessment data. Contemporary research distinguishes three types of assessments based on their relationship with learning: assessment of, for, and as learning (Schellekens et al., 2021). Assessment of learning, traditionally known as summative assessment, evaluates what students have learned and is typically administered at the end of a module. Assessment for learning, also known as formative assessment, aims to provide feedback to guide students in their learning. Assessment as learning shares the functionality of formative assessment in guiding learning but emphasizes student involvement in the assessment process as part of self-directed learning. As is clear from these definitions, assessment of learning data typically does not meet the requirements of timeliness. However, the pursuit of prompt feedback does not negate the utility of all assessment data. Prior research has underscored that even early assessment data contains predictive power related to module performance (Hlosta et al., 2022; Tempelaar et al., 2015).

Among the timeliest forms of assessment data are those derived from learning activities, especially in scenarios where assessments guide the learning process. This is particularly relevant for digital learning platforms based on mastery learning. From the very beginning of a module, learning activity data embodies the principles of both 'assessment as learning' and 'assessment for learning' (Tempelaar, 2020). Both assessment types are formative in nature where assessment is a learning tool to develop and enhance metacognition (as), or to provide comprehensive feedback that allows to discern and/or adapt learning behaviour (for).

A further specification of assessment for learning is offered by the theoretical approach of Black and Wiliam (2009), who conclude (p. 26): “…that any evidence of formative interaction must be analysed as reflecting a teacher’s chosen plan to develop learning, the formative interactions which that teacher carries out contingently within the framework of that plan […] and the internal cognitive and affective models of each student of which the responses and broader participation of students provide only indirect evidence”. To understand and apply formative assessment to its full potential, the perspective of the teacher as well as the student, and the interrelation between the two are crucial. Furthermore, building on multiple theories, Black and Wiliam (2009) indicate that comprehensive feedback regarding learning behaviour best informs learning and instruction (so both the teacher and the learner) if it provides insight in the current position of the learner in their learning, in where the learner is going, and finally in what the learner needs to do to get there.

Within the scope of our study, two e-tutorial learning platforms grounded in mastery learning not only provide data for assessment as learning but also provide assessment for learning data, exemplified by mastery scores students achieve in practicing module topics. In the realm of predicting assessment of learning data—specifically, performance in quizzes administered every week throughout the module and the final exam—it is noted that assessment of learning data (such as quiz results) dominates assessment for learning data in terms of predictive power, and in turn, assessment for learning data dominates assessment as learning data in predictive power. However, higher predictive power comes always with a lengthier time delay. The synergy of these three assessment types—assessment as learning, assessment for learning, and assessment of learning—culminates in the most effective contributions to actionable learning feedback.

Assessment and learning analytics

Two recent special issues have delved into exploring the relationship between LA and assessment in education: Gasevic et al. (2022) and Rakovic et al. (2023). Empirical studies in these special issues that focus on the role of FA consistently yield results that harmonize with the findings of a decade-long investigation into the joint influence of FA and LA, as conducted by the authors. These findings suggest that data from FA possess significant predictive capabilities for both module performance and student attrition. This indicates that any application of LA can greatly benefit from incorporating FA data (Tempelaar, 2020, 2021; Tempelaar et al., 2013, 2015, 2018). The introductions of both special sections employ a classification framework for LA research involving assessment, initially introduced in Gasevic et al. (2022). This framework comprises three categories: analytics for assessment, analytics of assessment, and measurement validity. When it comes to FA, the first two categories naturally coincide: FA data stands as a pivotal component of LA applications, and at the same time, to fully harness the potential impact of FA, the outcomes must be seamlessly integrated into the learning feedback generated by LA. Yet, it appears that learning analytics applications predominantly concentrate on discerning the learner's current position in the learning process, the instructional process of “where the learner is now” (Wiliam, 2011; Wiliam & Thompson, 2008), underexposing the instructional processes of “where the learner is going” and “how to get there”.

To address these last two questions, it is beneficial to use richer data than just the trace data of learning behaviour in digital systems, which is the primary data source in most LA applications. Dispositional learning analytics (DLA) provides one such enhancement (Buckingham Shum & Deakin Crick, 2012; Tempelaar et al., 2018, 2021). By integrating trace data from digital learning platforms with self-report surveys on learning dispositions, personalized feedback can be generated for students. This feedback not only identifies shortcomings in learning performance but also links performance gaps to learning dispositions, such as suboptimal use of learning strategies (Tempelaar et al., 2018, 2021).

Learning dispositions in the current study

The learning dispositions investigated in this study to incorporate in our DLA application emanate from the aforementioned considerations: namely, the essential learning proficiencies required for students to excel within a Problem-Based Learning (PBL)-focused curriculum. Our approach involves the inclusion of dispositions spanning cognitive, behavioural, and affective domains. Within this framework, we encompass cognitive and behavioural dimensions through motivation and engagement data, derived from Martin’s (2007) 'Motivation and Engagement Wheel', as well as achievement goal setting (Elliot et al., 2011). Further enriching this spectrum are two dispositions encapsulating both affective and behavioural facets: epistemic and achievement learning emotions (Pekrun & Linnenbrink-Garcia, 2012). Furthermore, our exploration encompasses mind-set measures, specifically encompassing entity and incremental beliefs (Dweck, 2006), which span the complete range of affective, behavioural, and cognitive components, denoted as a-b-c. The position of dispositions in our LA based modelling endeavour is thereby complementary Kleimola and Leppisaari (2022) approach. Instead of viewing disposition as competences that result from the model, we consider learning dispositions as precursors within our model.

Research questions

The research focus of this study naturally aligns with the two stages of LA applications: signal and act. The signal stage involves collecting, analysing, and interpreting learning data, while the act stage aims to enhance learning through interventions for and by students. As mentioned in the introduction, our first research inquiry pertains to the signalling phase, demonstrating the predictive value of formative assessment (FA) for module performance. The second inquiry, corresponding to the act stage, explores how students' learning dispositions can be leveraged to overcome suboptimal learning patterns.

In the first phase, the focus lies on observing learning processes and identifying instances where such processes could benefit from assistance or are susceptible to attrition, whereby we aim to demonstrate that by sharing both learning dispositions data and FA data with students could help to support their learning processes and learning outcomes. In the context of our specific question for the initial phase, we will concentrate on investigating the role that data from FA can play in this signalling process. A crucial aspect of this signalling is its timeliness, allowing ample room for the subsequent phase, which is the action phase.

Regarding the action phase, our inquiry pertains to the potential roles that learners' dispositions towards learning can assume in shaping interventions. While numerous LA applications excel at identifying students who are at risk of dropping out, many struggle to propose the subsequent course of action. Addressing inadequate learning dispositions emerges as a logical contender for such a follow-up step. This prompts the emergence of the research question for the second phase: Can we establish a connection between predictions of academic performance and the current state of students' learning dispositions?

Methods

Context and setting

This study was conducted in a large-scale introductory mathematics and statistics module, specifically designed for first-year undergraduate students enrolled in a business and economics program in the Netherlands. The module spanned over eight weeks, with a study load of 20 h per week. It was a compulsory module for all first-year students, often posing challenges for those with a limited background in mathematics. The educational approach employed a blended or hybrid learning model, based on the principles of flipped class design (Divjak et al., 2022).

Both e-tutorial systems, SOWISO and MSL, employ a test-driven learning and practice approach. Each step in the learning process begins with a problem that students are encouraged to solve. If a student struggles with a problem, they can request hints to guide them step-by-step or ask for a fully worked-out example. After receiving feedback, a new version of the problem, based on different parameters, is presented to allow the student to demonstrate their newly acquired skills. Students can choose from the following feedback strategies (see also Sect. “E-tutorial formative assessment data”, Fig. 2 specifically):

• Check: An unstructured problem-solving approach that provides correctness feedback after the problem is solved.

• Hint: A tutored problem-solving approach that offers feedback and tips to help with the various problem-solving steps.

• Solution: A worked examples approach.

• Theory: A brief explanation of the mathematical principle.

The main component of the blend was face-to-face PBL, where students learned in small groups of 14 students, facilitated by a subject expert tutor. Attendance in tutorial groups was mandatory and required around 2 × 2 h per week. Weekly lectures introduced key concepts, and the remaining 14 h were designated for self-study. Self-study was supported by printed materials such as textbooks and two interactive e-tutorials: Sowiso (https://sowiso.nl/) and MyStatLab (Nguyen et al., 2016; Rienties et al., 2019; Tempelaar, 2020; Tempelaar et al., 2015, 2017). This student-centered approach aimed to empower students to take responsibility for their own learning choices.

In line with the principles of PBL and suggestions by Black and Wiliam (2009) of providing formative feedback on how to help learners how to get there, feedback from the interactive tutorials and performance in the biweekly quizzes was shared with students and tutors, who provided guidance and discussion in tutorial sessions. During their interactions with students in tutorial groups, tutors played a crucial role in providing guidance and support. They engaged in bilateral communication with students, where they addressed the implications of feedback received and explore potential strategies for improvement. However, it is important to note that these prompting interactions occurred within tutorial sessions and were not formally recorded.

The learning process in this module can be divided into three distinct phases based on the timing of learning activities. In phase 1, students prepared for the tutorial session held each week. During the face-to-face tutorial sessions, students engaged in solving advanced mathematical and statistical problems, requiring prior self-study to actively participate in discussions. Phase 1 was not formally assessed but served as a preparation phase for subsequent activities.

Phase 2 involved preparing for the weekly quiz session, which took place at the end of each module week, except the first week. These were primarily formative in nature, providing students with feedback on their mastery of the mathematical and statistical topics covered that week at the end of the weekly learning cycle. However, they also contained a summative component, contributing 12.5% of the total score. The quizzes were administered online and included test items drawn from the same item pools used in the e-tutorials. This approach aimed to encourage students with limited prior knowledge to make extensive use of the e-tutorials.

Phase 3 focused on preparing for the final exam, which took place in the eighth week of the module. This phase involved formal, graded assessments, with the final exam accounting for 85% of the total module score. For a visual representation of the learning phases and corresponding sessions, please refer to Fig. 1.

Fig. 1

Flowchart of learning phases and session

Students made timing decisions regarding the amount of preparation dedicated to each of the three phases. Aligned with the principles of student-centred instruction, there are various forms of preparation that students can engage in. Most of these forms are not formally recorded and stored in a database. However, the use of e-tutorials for learning and solving practice questions were tracked and monitored. In this study, we consider solving practice questions in the e-tutorials as two out of the three aspects FA (i.e., where is the learner now, where is the learning going, and providing insights into “how to get there”), and the mastery score achieved through successful solving questions in these e-tutorials is regarded as the outcome of that FA. Each content topic was associated with three outcome measures: mastery in each of the three learning phases. To alleviate the pressure of quizzes for students with test anxiety and discourage procrastination until the week before the final exam, students had the opportunity to partially compensate for a lower quiz score by achieving good mastery scores in the second learning phase. On average, this compensation contributed 2.5% to the total module score. Consequently, although the primary nature of practicing in the e-tutorials is formative, it did have a minimal summative implication.

Participants

This study included a total of 1175 first-year students from the 2022/2023 academic year who actively participated in the e-tutorial Sowiso. Among these students, 37% were female and 63% were male. Additionally, 17% of the students held a Dutch high school diploma, while 83% were international students. The international student population was diverse, with a significant representation from neighbouring countries such as Germany (33%) and Belgium (16%), as well as other European countries. Furthermore, 6% of the students came from outside of Europe.

It is important to note that high school systems across Europe vary significantly, particularly in terms of mathematics and statistics education. For example, the Dutch high school system places emphasis on statistics, whereas many other countries may not cover this topic in their high school programs. Moreover, different countries have multiple levels of math education in high school, tailored for sciences, social sciences, or humanities. For admission into the international business and economics program, prior mathematics education preparing for social sciences is required. Additionally, 36% of the students in the sample followed the highest track in high school, contributing to the diversity in prior knowledge among the students.

Given this diversity, it was crucial to design the first module in a flexible manner, allowing for individualized learning paths and providing frequent, interactive feedback on students' learning strategies and tasks.

In addition to a final written exam, the assessment of students included a student project where students statistically analysed their own personal learning disposition data. For this purpose, students completed several questionnaires addressing affective, behavioural, and cognitive aspects of their aptitudes at the beginning of the module. Again these data provided insights in terms of the three FA functions (i.e., where is the learner now in terms of learning dispositions, where is the learning going, and providing insights into “how to get there”). For example, if a student's self-evaluations regarding adaptive motivational abilities such as planning and task management are found to be disappointing, the tutor and student will engage in a conversation focused on improvement. Subsequently, students received personal data sets to work on their project in the following weeks.

E-tutorial formative assessment data

Data for FA were collected from two e-tutorial systems: Sowiso for mathematics and MyStatLab for statistics. Both systems are based on the mastery learning instructional method (Tempelaar et al., 2017). However, there are substantial differences between the two systems in terms of data collection capabilities. Sowiso provides detailed time-stamps for each individual event initiated by the student, as well as mastery data, allowing for the integration of temporality in learning model design. On the other hand, MyStatLab lacks this level of detailed trace data. Given that mathematics is generally perceived as more challenging than statistics by most students, there was higher participation in the Sowiso e-tutorial. Therefore, for this study, the analysis focused exclusively on Sowiso data.

The Sowiso e-tutorial was structured around assignments or exercises that students were expected to solve. An example of such an exercise is provided in Fig. 2. Following the mastery learning principle, students were first introduced to a new topic through explanatory pages (Theory; see Fig. 2). They then solved exercises using worked-out examples (Solution; see Fig. 2) and hints (Hint; see Fig. 2) as learning aids. The final step was to solve the exercise independently, without any learning aids (Check; see Fig. 2), which contributed to mastery of the topic. Again this provided opportunities for the learners to gain insights in the three FA functions of Black and Wiliam (2009).

Fig. 2

Example of an exercise with learning aids in SOWISO

The time window and granularity of mastery observations are determined by the instructional design. There are seven weekly topics, such as functions of one variable, derivatives, functions of two variables, and optimization, which are arranged in a hierarchical order. Each topic consists of approximately ten packages, and each package contains five to ten assignments. Within each topic, three consecutive learning phases are distinguished, which are defined by the tutorial session, quiz session, and final exam. Therefore, the mastery data encompasses 20 measurements for the seven topics: Topic1 … Topic7 and three learning phases: Tutorial, Quiz and Exam (every topic brings mastery data for preparing tutorial session, quiz session and exam, except the last topic, where the module schedule does not allow for a quiz session).

Performance data

Quiz sessions were held at the conclusion of each week during the module, except for the first and last week. The first quiz, which took place at the end of the second week, covered topics from the first two weeks. Performance measures were derived from the quiz results and are labelled as Quiz1Math, Quiz2Stats, …, Quiz5Math, Quiz5Stats. The final exam follows a similar structure and was divided into two sections: ExamMath and ExamStats. The overall Grade for the module was determined by combining the exam scores, which carried a weight of 87%, and the quiz scores, which carried a weight of 13%.

Disposition data

Motivation and Engagement Wheel measures. The Motivation and Engagement Survey (MES), built upon the Motivation and Engagement Wheel framework developed by Martin (2007), categorize learning cognitions and behaviours into four quadrants. These quadrants differentiate between adaptive and maladaptive types, as well as cognitive (motivational) and behavioural (engagement) types of learning. Self-Belief, Learning Focus, and Valuing School shape the adaptive, cognitive factors or positive motivations. Persistence, Task Management, and Planning shape the adaptive, behavioural factors or positive engagement. The maladaptive cognitive factors or negative motivations are Uncertain Control, Failure Avoidance, and Anxiety, while Self-sabotage and Disengagement are the maladaptive behavioural factors or negative engagement.

Mind-set Measures: Self-Theories of Intelligence, Effort-Beliefs and Goal-setting. Measures assessing self-theories of intelligence, including both entity and incremental types, were utilized in this study. These measures were adapted from Dweck's Theories of Intelligence Scale—Self Form for Adults (Dweck, 2006). This scale consisted of eight items: four Entity Theory statements and four Incremental Theory statements. Measures of effort-beliefs were drawn from two sources: Dweck (2006) and Blackwell (2002). Dweck provides several sample statements designed to portray effort as a negative concept, Effort Negative (exerting effort conveys the view that one has low ability), and effort as a positive concept, Effort Positive (exerting effort is regarded as something which activates and increases one’s ability). In addition, Blackwell’s complete sets of Effort beliefs (2002) were used, comprising five positively phrased and five negatively worded items (see also Blackwell et al., 2007; Tempelaar et al., 2015). Goals have been operationalized by the Grant and Dweck (2003) instrument, which distinguishes the two mastery goals Challenge-Mastery and Learning Goals, as well as four types of performance goals. Of the performance goals, two are of appearance nature: Outcome and Ability Goals, and two of normative nature: Normative Outcome and Normative Ability Goals.

Achievement Goal Questionnaire (AGQ). The AGQ is based on the 3X2 achievement goal model (Elliot et al., 2011), which includes six goal constructs. These constructs differentiate between three types of goals: task, self, and other, and two valences: approach and avoidance. They encompass the task-based dimension with task-approach and task-avoidance goals, the intra-personal dimension with self-approach and self-avoidance goals, and the inter-personal dimension with other-approach and other-avoidance goals. The AGQ extension further distinguishes two sub-dimensions within the self-dimension: past and potential (Elliot et al., 2015). As a result, there are eight separate constructs within the extended AGQ: Task-approach, Task-avoidance, Self-approach, Self-avoidance, Other-approach, Other-avoidance, Potential-approach and Potential-avoidance achievement goals.

Achievement and epistemic emotions. The Control-Value Theory of Achievement Emotions (CVTAE, Pekrun, 2006) suggests that achievement emotions vary in terms of valence, focus, and activation. To measure these emotions, we utilized the Achievement Emotions Questionnaire (AEQ, Pekrun et al., 2011), which is based on the CVTAE framework. From the AEQ, we specifically selected four scales related to learning activity emotions that have the strongest correlation with academic performance out of the eight activity scales included in the questionnaire: positive activating Enjoyment, negative activating Anxiety, and negative deactivating Boredom and Hopelessness. Given the primacy of independent, self-regulated learning in the PBL system, the learning activity-related versions of the scales were applied rather than the class- or test-related versions. Where these achievement aspects of emotions focus on doing a learning activity, epistemic emotions relate to the cognitive aspects of the task itself (Pekrun & Linnenbrink-Garcia, 2012). Epistemic emotions were measured with the Epistemic Emotion Scales (EES, Pekrun et al., 2017) and included Surprise, Curiosity, Confusion, Anxiety, Frustration, Enjoyment, and Boredom. Academic Control, measured with the perceived Academic control scale of Perry et al. (2001), was included as the main proximal antecedent of activity emotions.

In this action research, the only control variable available was self-reported prior education.

All surveys were administered using 7-point Likert scales. All participants provided informed consent to use the anonymized student data for educational research.

Statistical analyses

This study employs a combination of person-centred and variable-centred statistical methods. The heterogeneity of the sample itself justifies the need to break down the full sample into homogeneous sub-samples, which is a requirement for variable-centred methods (Howard & Hoffman, 2018). However, besides statistical reasons, there are also substantive arguments to apply a person-centred approach in the analysis. When providing learning feedback to students and designing educational interventions, which are the main objectives of using LA, it is advantageous to identify common characteristics among students rather than addressing each student individually (Tempelaar, 2021).

Drawing on person-centred modelling approaches (Malcom-Piqueux, 2015) and utilizing cluster analysis techniques to identify distinct profiles of learners based on their actual engagement and behaviour, while ensuring homogeneity (Howard & Hoffman, 2018), the analysis employed k-means cluster analysis. The input for the cluster analysis consisted of FA data from the Sowiso e-tutorial, specifically the mastery data of various topics in the first two learning phases. Considering the practical considerations typical in LA applications, where a balance needs to be struck between predictive power and timeliness, three cluster-analysis models were estimated:

• Using all mastery data of all weeks (20 input variables)

• Using mastery data of the first four weeks (9 input variables)

• Using mastery data of the first two weeks (5 input variables)

Despite the differences in data availability among the three analyses, the results of all three analyses showed remarkable similarities. The main distinction was in the number of clusters, as using more data points allowed for additional nuanced clusters while maintaining the basic clusters relatively unchanged. For this research, we selected the cluster solution based on four weeks of mastery data as input. This choice reflected the objective of LA applications, which aim to provide timely learning feedback while allowing sufficient time for intervention.

The decision on the number of clusters was made to ensure maximum profile variability without creating small clusters that consisted of less than 5% of the students. We settled on a five-cluster solution. Higher-dimensional solutions did not significantly alter the characteristics of the clusters and were more challenging to interpret.

Although it was possible to include disposition data in the cluster analysis, we decided to focus solely on trace data. This choice was made to specifically isolate the role of trace data and enable comparisons with mainstream LA research that typically lacks disposition data. By grouping students into clusters based on their use of FA, we were able to examine the relationships between behavioural measures of FA use and self-reported aptitudes.

An alternative approach would have been to combine both behavioural and dispositional measures as the basis for clustering, as done in previous research by the authors (Tempelaar, 2020). In such a case, the resulting student profiles would represent a combination of actual learning activities and self-perceptions of learning dispositions. A third option would have been to create clusters based solely on disposition data and then investigate the differences in learning behaviours among these clusters. An example of this approach can be found in a study by the authors (Tempelaar et al., 2021), which explores characteristic differences in learning behaviours among students with different mind-set profiles, a specific type of learning aptitude.

Considering that dispositional data is often lacking in the majority of LA applications, we chose to create profiles in a way that is feasible for any LA study. Dispositional data were added in a final additional step of the analysis.

To further analyse the differences between profiles, the variable-centred analysis step involved conducting ANOVA tests. The outcomes of these tests will be reported without specifying the significance levels, as all reported results are statistically significant at levels below 0.001. Large sample sizes are instrumental in being able to apply such a strict benchmark for statistical significance. Employing strict benchmarks also helps guard against the accumulation of type 1 errors that may arise from repeated hypothesis testing.

As a preliminary step to the estimation of student profiles, we examined the significance of early assessment of learning data in elucidating subsequent assessments of learning. This was achieved by constructing a path model that explained later assessment of learning through early assessment of learning and assessment for learning data. The path model was estimated using the MPlus statistical package, while all other analyses were conducted using the IBM SPSS statistical package.

Results

We begin our analysis with a traditional LA study: examining trace data representing students’ use of FA during the first weeks of the module to identify patterns in the role of FA in learning and learning outcomes. This analysis is conducted in two steps. First, we derive prediction equations to explain module performance based on data for the entire cohort, as detailed in Sects. “A path model for students’ formative assessment scores, quiz performance and exam performance” and “Prediction equations”. In the second step, we use a person-cantered modelling approach, reported in Sects. “Student profiles by cluster analysis” and “Relevance of profiling”, where we decompose the full cohort into different profiles using cluster analysis. These subsections collectively address the first research question. In Sect. “Learning dispositions and profile differences”, we address the second research question by linking learning profiles to learning dispositions, resulting in a DLA application.

Descriptives of formative assessment use

The use of FA by students, measured as a percentage of mastery achieved in subsequent topics, was initially high but gradually decreasing. Figure 3 visually demonstrates this trend over time. During the first phase of learning, which involves tutorial sessions, the utilization of FA was relatively modest, ranging between 10 and 20% mastery level. However, it became evident that students primarily focussed on "where the learner is going", or obtain feedback on where they are now and where they are going, during the second phase, specifically in preparation for quizzes. Mastery levels after completing quiz preparation fluctuated between 60 and 70% (mastery percentages in Fig. 3 are cumulative). In contrast, the third phase, which involved preparing for the final exam, did not significantly contribute to mastery percentages. In other words, it is during the second phase where students actively employed FA. Both trends can be attributed to the module's design choices. Although students are expected to come to tutorial sessions fully prepared, their preparation is not assessed. However, they are assessed in the quiz preparation phase. As the cluster analysis results will later show, only a minority of students complete their learning in the first phase. The majority begin their initial learning in the first phase and delay checking their mastery until the second phase, quiz preparation. Another design choice involves student-directed learning, where students select their own learning materials. At the beginning of the module, when everything is still unfamiliar, students tend to rely heavily on structured e-tutorials that compensate for their lack of learning regulation. Gradually, they shift to other learning resources that require more self-regulation. Some students dropping out of the module also contribute to this downward trend, which is similarly observed in other research on the long-term use of digital learning resources (Tempelaar et al., 2023). Beneath these general trends, there is significant interindividual variation, primarily due to the existence of different learning behaviour profiles. In Sect. “Student profiles by cluster analysis”, the discussion will focus on these profile differences in e-tutorial use.

Fig. 3

Average formative assessment scores (0–100%) per topic and per learning phase. The black markers represent the data used in the cluster analysis, generated in the first half of the module. These markers include the preparation for tutorials on the first five topics and the preparation for quizzes on the first four topics. However, any preparation for the final exam is not included in these markers

A path model for students’ formative assessment scores, quiz performance and exam performance

The model integrating all facets of student module performance is a path model that described the antecedent-consequence relationships between performance in FA and quiz performance, together with antecedent-consequence relationships quiz performance and exam performance, all controlled for the two measures of prior proficiency: educational track in high school, and entry test score. Figure 4 presents the diagram depicting the path model. In analogy to Fig. 3, the dashed line in Fig. 4 splits variables in early measures, observed in the first four weeks of the module, and late measures, observed in the second half of the same module. The combined scores of quizzes administered in the first half of the module are referred to as QuizMathH1, while the combined scores of quizzes in the second half are referred to as QuizMathH2.

Fig. 4

Path model of module performance for formative assessment, quizzes and final examination; *p < 0.05; **p < 0.01; ***p < 0.001

The fit indices indicated an acceptable fit for the model (RMSEA = 0.018, CFI = 0.995, and TLI = 0.992). The path model diagram is divided into two panels. The left panel represents the first four weeks of the module, and the right panel represents the second half. In the left panel, we see that the aggregated score of the three quizzes from the first half of the module is predicted by two prior education constructs, MathMajor and EntryTest, along with four FA-type variables: the mastery scores students achieve while preparing for the quizzes (that is, covering the first two phases of learning) for math topics in the initial four weeks. In the right panel, we observe that the students' combined quiz scores in the second half of the module are predicted by their previous quiz scores and mastery scores for preparing the quizzes in the second half. Finally, the MathExam score is predicted by both quiz score constructs, QuizMathH1 and QuizMathH2, as well as the prior education indicator MathMajor. The effect sizes (%R2) were 19.6% for MathExam, 39.7% for QuizMathH2, and 26.8% for QuizMathH1. By excluding data from Topic5 and Topic6 quiz scores when predicting QuizMathH2 and excluding QuizMathH2 when predicting MathExam, the predictive power of early measurements was reflected in effect sizes of 14.6% for MathExam and 35.2% for QuizMathH2.

Prediction equations

To further explore the potential of early feedback and intervention, we conducted regression analyses using five different sets of predictors to examine module performance. These predictor sets were as follows: (1) students’ use of FA in preparing tutorial and quiz sessions in the first half of the module, (2) performance in quizzes in the first half of the module, (3) a reduced set of learning dispositions, (4) the combination of the first two sets of predictors (i.e., FA and early quiz scores), and (5) the combination of all predictors (FA data, early quiz scores and dispositions). FA data used in these prediction equations is represented by the black dots in Fig. 3 connected by the blue line. To simplify the analysis, the initial 45 dispositional variables were reduced to nine factors using factor analysis, focusing on factors with eigenvalues greater than one. The prediction equations for quiz performance, utilizing early quiz scores as predictors, were specifically designed to predict quiz scores in the second half of the module. Table 1 presents the predictive capabilities of these five predictor sets for five performance measures: ExamMath, ExamStats, QuizMath, QuizStats, and Grade.

Table 1 Predictive power, as %R², of linear prediction equations for module performance measures, with different predictor sets

The initial two rows in Table 1 revealed that students' utilization of FA for preparing early tutorials and quizzes and the scores of these early quizzes exhibited similar trends across multiple performance measures, with quiz scores showing slightly stronger predictive power than FAs. Both sets of predictors did better in explaining quiz scores than exam scores. In the third row of Table 1, it was evident that student dispositions exhibited an opposite pattern: they had higher predictive power for exam scores compared to quiz scores. When comparing the predictive power of the combined set of FA data and early quiz scores (fourth row of Table 1) with the first two rows, it became apparent that there was considerable overlap in predictive power between FA data and early quiz scores. The last row of Table 1 describes the predictive power of all three predictor sets combined. The inclusion of student dispositions, which complement the predictive power of FA use and quiz scores, resulted in overall predictive power that remained remarkably consistent, explaining approximately one-third of the variation in various module outcome variables.

Student profiles by cluster analysis

By applying cluster analysis to the available FA data from the first half of the module, we were able to identify five distinct clusters that contained a sufficient number of students. These clusters reflected different learning behaviours and were easily interpretable. Although higher-dimensional solutions did not significantly alter the cluster characteristics, they tended to divide smaller clusters into even smaller ones. Figure 5 offers a comprehensive view of these five profiles. Unlike Fig. 3, it is divided into three panels that represent the three learning phases: tutorial preparation on the left, quiz preparation in the middle, and exam preparation on the right. Each of the five profiles is represented by 20 mastery data points, of which nine were used as input for the cluster analysis.

Fig. 5

Average formative assessment scores per topic (0–100%), per learning phase, for five student profiles

We used k-means clustering with Euclidean distance to group the observations into five distinct clusters. Data standardization was achieved by the common response scale of all variable in the cluster analysis: all mastery scores are expressed as percentage points. The choice for the number of clusters is based on substantial arguments: cluster solutions with more clusters create small clusters, below 5% of the sample, whereas cluster solutions with less than five clusters miss patterns of student behaviour that lecturers regard as an important profile of learning behaviour. For example, the four-cluster solutions basically misses the Cluster4 pattern of students who shift their learning from the first phase to the second phase in the first three weeks of the module.

The learning profiles observed in the cluster solution closely resemble those identified in previous studies (Tempelaar et al., 2018, 2021), with profiles differing mainly in intensity and timing of learning. Cluster5 consists of ‘role model’ students who reached high mastery levels in the first learning phase, with FA scores ranging from 60 to 90%. They continued to excel in the second phase, achieving full mastery. Unfortunately, this profile was the smallest, comprising only 90 students. On the other hand, Cluster3 and Cluster4, consisting of 458 and 143 students respectively, reached similar levels of mastery. However, they achieved these levels only in the second learning phase, and did not utilize FA during the first phase. The main distinction between them lies in their behaviours during the initial weeks of tutorial preparation: Cluster4 students began with enthusiasm similar to Cluster5 students but eventually switched to postponing their learning. Cluster2, comprising 300 students, represents the profile of students who made minimal use of FA. Cluster1, consisting of 287 students, falls between the profiles of Cluster1 (inactive students) and Cluster3 (the large group of students achieving full preparation in the second phase). Notably, Cluster1 was the only profile where there was a noticeable decrease in the utilization of the e-tutorial as the weeks progressed.

Relevance of profiling

The extent to which student profiling using FA data in this LA application can predict their module performance is a crucial question. Figure 6 provides a clear answer to this question. In this figure, all module performance measures were rescaled on a 1 to 10 scale, which aligns with the grading system used in the Netherlands. We observed statistically significant and substantial differences amongst the profiles. The effect sizes, measured by Eta squared, were 31.1%, 29.3%, 13.4%, 9.3%, and 17.9% for quiz scores in mathematics and statistics, exam scores in mathematics and statistics, and the final module grade, respectively. Another noteworthy observation pertains to the students' ability to pass the module. The passing benchmark required a final grade of at least 5.5, but on average, Cluster2 students achieve only a 5.3.

Fig. 6

Average performance differences in five student profiles for Quizzes (1–10), final Exam and module Grade

The ordering of student profiles in Fig. 6 aligned with the ordering observed in Fig. 5. This means that students who utilized the e-tutorial more intensively, developed mastery, and received feedback through FA tended to perform better. Additionally, students who initiated these practices earlier also demonstrated better performance. One particularly notable finding was that this improved performance extended beyond the specific topic students practiced in the Sowiso e-tutorial, namely mathematics. The enhanced performance also applied to the other topic, statistics. While cognitive factors could account for why students who engaged in more intensive FA performed better in mathematics quizzes and exams (as they possess a broader knowledge base), cognitive factors alone cannot explain their superior performance in statistics quizzes and exams. This is noteworthy because the statistics topic focused on social sciences rather than purely mathematical knowledge. It appears that factors beyond cognitive abilities, such as affective and behavioural learning aptitudes, were indirectly incorporated in the profiling process based solely on Sowiso mastery data.

Learning dispositions and profile differences

To gain insights into the factors beyond cognitive abilities that contribute to the average performance differences between profiles, we leverage the dispositional facet of our LA application in this research. At the beginning of the module, students provided self-reports on affective, behavioural, and cognitive learning dispositions. ANOVA tests examining profile differences revealed statistically significant variations in several of these learning dispositions, at a strict significance level of 0.001. It was worth noting that significant differences were observed across all disposition instruments, which was expected considering the interrelated nature of these dispositions. In this subsection, we explored each disposition instrument individually to identify the dispositions associated with profile differences that may also be linked to educational interventions.

The motivation and engagement wheel instrument, which has consistently shown the strongest association with module performance in previous research conducted by the authors, exhibited a notable pattern. The engagement factor, rather than the motivation factor, demonstrated the most substantial profile differences. Both adaptive engagement factors (Persistence, Task Management, Planning) and maladaptive engagement factors (Self-sabotage and Disengagement) exhibited statistical significance, with effect sizes ranging from 3 to 7%. In contrast, profile differences in motivational factors had smaller effect sizes, typically below 2.5%, and in some cases lacked statistical significance. Refer to Fig. 7 for a visual representation of these findings.

Fig. 7

Average performance differences indidated by effect size in five student profiles for motivation and engagement

Full ANOVA statistics: Self-Belief: F(4, 1170) = 1.03, p = 0.390, η² = 0.004; Learning Focus: F(4, 1170) = 7.22, p < 0.001, η² = 0.024; Valuing School: F(4, 1170) = 5.62, p < 0.001, η² = 0.019; Persistence: F(4, 1170) = 12.74, p < 0.001, η² = 0.042; Task Management: F(4, 1170) = 8.00, p < 0.001, η² = 0.027; Planning: F(4, 1170) = 20.62, p < 0.001, η² = 0.066; Uncertain Control: F(4, 1170) = 2.50, p = 0.041, η² = 0.008; Failure Avoidance: F(4, 1170) = 0.67, p = 0.610, η² = 0.002; Anxiety: F(4, 1170) = 2.75, p = 0.027, η² = 0.009; Self-sabotage: F(4, 1170) = 13.67, p < 0.001, η² = 0.045; Disengagement: F(4, 1170) = 8.16, p < 0.001, η² = 0.027.

All engagement measures showed a consistent pattern in profile differences: Cluster5 students scored highest on adaptive measures, lowest on maladaptive measures, and Cluster2 students exhibited the opposite pattern, while the other clusters fell in between these extremes.

Regarding epistemic beliefs, represented by Entity Theory and Incremental Theory as the two implicit theories of intelligence, and Effort Negative and Effort Positive as beliefs about effort, there were no substantial profile differences. Only Effort Positive demonstrated statistical significance, but with an effect size below 2%. Please refer to the left panel of Fig. 8 for a visual representation.

Fig. 8

Average performance differences indicated by effect size in five student profiles for epistemic beliefs (left panel), and goal setting (right panel)

Full ANOVA statistics: Entity Theory: F(4, 1132) = 2.38, p = 0.050, η² = 0.008; Incremental Theory: F(4, 1132) = 2.56, p = 0.036, η² = 0.009; Effort Negative: F(4, 1132) = 2.34, p = 0.036, η² = 0.008; Effort Positive: F(4, 1132) = 5.40, p < 0.001, η² = 0.042; Outcome Goal: F(4, 1132) = 11.44, p < 0.001, η² = 0.039; Ability Goal: F(4, 1132) = 9.82, p < 0.001, η² = 0.034; Normative Outcome Goal: F(4, 1132) = 6.72, p < 0.001, η² = 0.023; Normative Ability Goal: F(4, 1132) = 2.60, p = 0.035, η² = 0.009; Learning Goal: F(4, 1132) = 9.59, p < 0.001, η² = 0.033; Challenge Mastery Goal: F(4, 1132) = 2.99, p = 0.018, η² = 0.010.

Goal setting behaviour aligned with mind-set theory exhibits more pronounced profile differences. The two performance goals, Outcome and Ability Goals, which were oriented towards appearance and external validation, and the mastery goal Learning Goals, demonstrated statistically significant differences with effect sizes ranging from 3 to 4%. These findings can be observed in the right panel of Fig. 8.

The patterns observed in motivation and engagement were consistent with the findings in goal setting behaviour. Cluster5 students achieved the highest scores on all types of goal setting, while Cluster2 students scored the lowest, with the other clusters falling in intermediate positions. These patterns also extended to the domain of epistemic beliefs, taking into account that these beliefs encompassed both adaptive and maladaptive facets.

Another aspect related to epistemic dispositions pertained to epistemic emotions. Epistemic emotions, assessed at the beginning of the module, are presented in the left panel of Fig. 9. On the other hand, activity emotions, along with academic control, which were measured at the end of the fourth week, are displayed in the right panel of Fig. 9.

Fig. 9

Average performance differences indicated by effect size in five student profiles for epistemic emotions (left panel), and activity emotions (right panel)

Full ANOVA statistics: Curious: F(4, 1153) = 7.28, p < 0.001, η² = 0.025; Surprised: F(4, 1153) = 6.87, p < 0.001, η² = 0.023; Confused: F(4, 1153) = 3.60, p = 0.006, η² = 0.012; Anxious: F(4, 1153) = 3.61, p = 0.006, η² = 0.012; Frustrated: F(4, 1153) = 4.22, p = 0.002, η² = 0.014; Excited: F(4, 1153) = 9.86, p < 0.001, η² = 0.033; Bored: F(4, 1153) = 6.89, p < 0.001, η² = 0.023; Academic Control: F(4, 1141) = 0.4.13, p = 0.003, η² = 0.014; Anxiety: F(4, 1141) = 5.58, p < 0.001, η² = 0.019; Boredom: F(4, 1141) = 18.74, p < 0.001, η² = 0.062; Hopelessness: F(4, 1141) = 8.72, p < 0.001, η² = 0.030; Enjoyment: F(4, 1141) = 15.64, p < 0.001, η² = 0.052.

Profile patterns in learning emotions were less unambiguous than profile patterns described earlier. While all profile differences were statistically significant, the effect sizes varied. Cluster5 students consistently exhibited the most adaptive positions, scoring highest on positive emotions and lowest on negative emotions. However, Cluster2 students no longer occupied the extreme maladaptive positions. Instead, Cluster1 students tended to score highest on measures of epistemic anxiety, frustration, and activity emotions like anxiety and hopelessness. When comparing levels of epistemic and activity emotions, we observed a positive development, particularly in reducing negative emotions. Negative emotion scores decreased from neutral levels at the start of the module, to below neutral levels halfway the module, accompanied by increased profile differences. The largest profile differences were found for activity emotions like Boredom and Enjoyment, with eta squared effect sizes of 6.2% and 5.2% respectively. Profile differences in Academic Control, with an effect size of 1.4%, were relatively small.

Regarding achievement goals based on approach versus avoidance valences, statistically significant differences were observed, except for the Self-Avoidance goal. Approach goals generally had larger effect sizes compared to avoidance goals, with the highest effect sizes observed for the Other-Approach goal (3.0% effect size) and Potential-Approach goal (4.3% effect size). Please refer to the left panel of Fig. 10 for visual representation. However, the largest effect sizes were found in the right panel of Fig. 10, which represents the economic attitude tendency to Postpone. It indicated an effect size of 10.4% at the start of the module, which increases to 15.8% in the fifth week.

Fig. 10

Average performance differences indicated by effect size in five student profiles for achievement goals (left panel), and economic attitudes (right panel)

Full ANOVA statistics: Task-approach Goal: F(4, 1153) = 7.28, p < 0.001, η² = 0.025; Task-avoid Goal: F(4, 1153) = 6.87, p < 0.001, η² = 0.023; Self-approach Goal: F(4, 1153) = 3.60, p = 0.006, η² = 0.012; Self-avoid Goal: F(4, 1153) = 3/60, p = 0.006, η² = 0.012; Other-approach Goal: F(4, 1153) = 3.61, p = 0.006, η² = 0.012; Other-avoid Goal: F(4, 1153) = 4.22, p = 0.002, η² = 0.014; Potential-approach Goal: F(4, 1153) = 9.86, p < 0.001, η² = 0.033; Potential-avoid Goal: F(4, 1153) = 6.89, p < 0.001, η² = 0.023; RiskTaking: F(4, 1141) = 0.4.13, p = 0.003, η² = 0.014; Postpone: F(4, 1141) = 5.58, p < 0.001, η² = 0.019; WillingGiveUp: F(4, 1141) = 18.74, p < 0.001, η² = 0.062; RiskTakingW5: F(4, 1141) = 8.72, p < 0.001, η² = 0.030; PostponeW5: F(4, 1141) = 15.64, p < 0.001, η² = 0.052; WillingGiveUpW5: F(4, 1141) = 15.64, p < 0.001, η² = 0.052.

Discussion

The benefits of incorporating FA and disposition data in LA applications revolve around two key concepts: timeliness and actionability. While summative assessments, such as quizzes, have the highest predictive power for performance prediction models, they often occur later in the module, delaying the opportunity for feedback and intervention. Figure 4 illustrates this timing issue in our context: by the midpoint of the module, we have data from only one quiz but four weeks of FA data. Given that FA data closely mirrors quiz performance, it is justified to use FA data early in the module as a proxy for quiz data to drive early feedback and interventions. Additionally, disposition data, collected through surveys administered at the start or even before the module, also increase predictive power to facilitate timely feedback.

To illustrate the added value of DLA to any LA application, this study performed profiling based solely on FA data—data that is readily available to any LA practitioner using FA data along with other trace data to provide learning feedback. By decomposing the full cohort into clusters based on different approaches to using FA, we identified profiles that meaningfully differ in the timing and intensity of FA use. In a traditional LA application, the next step might involve a learning intervention, such as informing a student that their learning efforts are lagging behind their peers or occur later than typical. Beyond the circumstance that such feedback may not be relevant—consider a student who excelled in the highest mathematics track in high school and rationally chooses to focus on other subjects rather than math-, the incentive to learn more or earlier is primarily focused on "where the learner is now," without addressing "where the learner is going" or even "how to get there." DLA can significantly enhance this process. It not only improves the accuracy of performance predictions early in the module but also enriches learning feedback beyond mere diagnostic information, providing more comprehensive guidance for students.

Such guidance will combine trace data with disposition information. For example, consider a student who shows low intensity in using FA. The next step might involve checking the student's scores on the Incremental Theory, Entity Theory, and Effort beliefs. According to mindset theory (Dweck, 2006), low effort in learning could stem from a maladaptive belief in the futility of effort, rather than an adaptive belief in its utility. High scores on Effort Negative and Entity Theory, coupled with low scores on Effort Positive and Incremental Theory, would suggest that addressing the student's mindset could be a more effective intervention than simply advising them to work harder. This is because the student may believe that additional practice won't help them, viewing proficiency in math as a fixed trait.

Another example is a student who starts preparing late, primarily during the second or even third learning phase. If this behavior results in disappointing performance in early quizzes, a natural next step is to examine their disposition scores. Specifically, what do the motivation and engagement scores from the Motivation and Engagement Wheel (Martin, 2007) indicate? If the scores are low on adaptive cognitions like Planning and Task Management, it is more effective to address these learning skills rather than simply urging the student to take action. Similarly, if the maladaptive behavior of Failure Avoidance scores high, the intervention should target this specific disposition instead of merely addressing the symptom of starting too late.

The ultimate goal of applying LA is to provide 'actionable learning feedback' (Gašević et al., 2015). While all LA undoubtedly offers learning feedback, the question remains as to how actionable that feedback truly is. Integrating learning dispositions and including FA data appears to be a powerful step toward achieving this goal.

Conclusions

This empirical study focussed on the role that formative assessment can play to enhance learning support via learning analytics (Gašević et al., 2022). We provided a practical application of how we engaged both educators and students in assessment as, for, and of learning in a challenging quantitative methods module at the beginning of a business bachelor program. In line with Black and Wiliam (2009), a unique feature of our contribution is that we argue that educators and students together can substantially benefit from formative assessment and feedback data, in particular in a situation where students are working with mastery learning based e-tutorial programmes that provide continuous formative feedback like Sowiso.

Using principles of dispositional learning analytics (DLA) application in line with recommendations by Buckingham Shum and Deakin Crick (2012) and Buckingham Shum and Ferguson (2012), we explicitly decided to include learning dispositions survey data (e.g., motivation, engagement, learning emotions), assessment data, trace data and performance data. In particular, as students from day 1 onwards worked on their own learning dispositional data gathered from common psychometric instruments like the Motivation and Engagement Survey by Martin (2007) and the Achievement Goal Questionnaire of Elliot et al. (2011), and also continuously received FA data about their learning progress, both students and educators were given plenty of opportunities to gage their assessment as, for, and of learning.

In terms of the first research question of the role that data from FA can play in signaling potential learning support for students who might need this, our path model suggested that we could accurately identify which students might need more additional support in the first half of the module. Follow-up analysis showed that the combined set of FA data and early quiz scores provided considerable overlap in predictive power between FA data and early quiz scores. The inclusion of student dispositions, which complemented the predictive power of FA use and quiz scores, resulted in overall predictive power that remained remarkably consistent, explaining approximately one-third of the variation in various module outcome variables. Subsequently, we applied cluster analysis for the available FA data from the first half of the module, whereby we were able to identify five distinct and easily interpretable clusters of students.

In terms of the second research question of the connection between predictions of academic performance and the current state of students' learning dispositions, we found several consistent patterns. For example, the motivation and engagement wheel instrument of Martin (2007) showed the strongest association with module performance. All engagement measures showed a consistent pattern in profile differences. For example, Cluster5 students (labelled as “role model students”) scored highest on adaptive measures, lowest on maladaptive measures, while Cluster2 students (labelled as “lowest FA users”) exhibited the opposite pattern, while the other clusters fell in between these extremes. As these patterns were fairly consistent after four weeks, this would provide substantial opportunities for actionable learning support specifically for each respective cluster of students.

An evident limitation within our study lies in observing only one of the two parallel learning modes. Where we capture every detail of learning in the blended mode, all learning in the face-to-face mode taking place in the tutorial groups, the exchange between tutors and students, and student self-study outside the e-tutorial SOWISO, remains unmonitored. This also implies that a vital part of the act component of LA remains concealed from our observations. Particularly in our Problem-based learning context, the role of tutors is pivotal, as they engage extensively with students in deliberating how to overcome learning challenges. While we can observe (part of) the information that underpins these discussions, the discussions themselves and the subsequent educational interventions evade our observation. The inherent subtlety of these interventions might elucidate the somewhat modest effect sizes—approximately one-third of the variability—that the predictive equations in Sect. “Prediction equations” display, despite their notable statistical significance. It is conceivable that the undisclosed interventions counterbalance a significant portion of the unaccounted variability. If that is the scenario, these interventions could indeed be quite effective, albeit their mechanisms remain concealed from our scrutiny.