Interactions with educational chatbots: the impact of induced emotions and students’ learning motivation

Abstract

Educational chatbots (EC) have shown their promise in providing instructional support. However, limited studies directly explored the impact of EC on learners’ emotional responses. This study investigated the induced emotions from interacting with micro-learning EC and how they impact learning motivation. In this context, the EC interactions encompassed greetings, biology learning content delivery, self-evaluation, and feedback. This study employed a between-subject experimental design involving 62 college students. Participants were randomly assigned to either the Metacognitive EC group, receiving metacognitive feedback, or the Neutral EC group, receiving neutral feedback. The results of T-tests demonstrated significant differences in specific induced emotions between the two groups while some similarities exist. Importantly, it unveiled that both Metacognitive EC and Neutral EC interactions evoked a spectrum of positive, negative, and ambivalent emotions, in which positive emotions surpassed the induced negative emotions. In general, metacognitive feedback induced fewer negative emotions than neutral feedback. PLS analysis supported the relationships between induced emotions and intrinsic motivation, with positive emotion, ambivalent emotions, and negative emotions influencing interest motivation, which, in turn, shaped other motivational components, including perceived competence, perceived value, and perceived pressure. However, the influence of positive emotion on interest was weaker in the Metacognitive than in the Neutral EC. In conclusion, the study revealed how induced emotions impact motivations and showed that the presence of metacognitive feedback reduced negative emotions and promoted motivation. These findings highlight the need for positive emotion element design and appropriate feedback that will impact learning motivations during educational chatbot interactions.

Introduction

In recent years, advancements in artificial intelligence (AI) and natural language processing (NLP) have led to the development of educational chatbots as promising tools in the field of education. These chatbots are designed to interact with learners conversationally, providing instant feedback, personalized assistance, and instructional support (Kuhail et al., 2023). By leveraging AI technologies, educational chatbots can complement traditional educational settings, offering individualized learning experiences and addressing the diverse needs of students. While the potential benefits of educational chatbots are widely recognized (Okonkwo & Ade-Ibijola, 2021), their effectiveness is contingent upon the ability to evoke appropriate emotional responses from users that can positively influence learning motivation and outcomes (Kim & Pekrun, 2014).

Emotions play a pivotal role in the learning process, influencing students’ cognitive processes, attention, and information processing (Pekrun, 1992). Positive emotions, such as interest, enjoyment, and curiosity, have been linked to increased motivation, improved information retention, and enhanced problem-solving abilities. On the other hand, negative emotions, including frustration, boredom, and anxiety, can impede the learning process and lead to reduced motivation and disengagement (Graesser & D’Mello, 2012). Recognizing the impact of emotions on learning, chatbots are now being designed with emotional intelligence, aiming to recognize and appropriately respond to users’ emotional states (Bilquise et al., 2022; Ehtesham-Ul-Haque et al., 2024; Rajwal, 2022).

While research on educational chatbots and their impact on learning motivation is growing, there remains a significant gap in understanding the effect of ambivalent emotions on learning motivation and learning dynamics. Ambivalent emotions also known as mixed emotions refer to the coexistence of positive and negative emotions experienced simultaneously during an interaction (Lomas, 2017; Naomi et al., 2021). In the context of educational chatbot interactions, learners may experience a mix of positive and negative emotions, such as excitement about receiving instant feedback and frustration when facing challenges in understanding a concept.

Moreover, feedback serves diverse purposes such as self-evaluation, support learning, grading, clarifying expectations, reducing discrepancies, error detection, and increasing motivation (Hattie & Timperley, 2007). Recent developments in Natural Language Processing (NLP) have empowered educational chatbots with adaptive feedback functions. Students can now seek timely and comprehensive guidance on their performance against set criteria, receiving tailored strategies for improvement. However, their emotional and motivational relationships have yet to be studied comprehensively. Furthermore, prior studies on emotions in human-computer interactions often focus on either positive or negative emotions and often overlook the role of ambivalent emotions. Therefore, investigating the effect of overall emotions induced by educational chatbots while interacting with learning content and receiving feedback is essential, as it reflects a more realistic and nuanced portrayal of users’ overall chatbot engagement experiences. By examining the impact of feedback and induced emotions, we can better comprehend how emotions influence learners’ motivation and engagement, offering effective emotion-aware teaching strategies.

In this regard, this study aims to investigate how the overall emotional experiences of learners during their interactions with educational chatbots can affect their motivation to learn. We adopt a quantitative and experimental approach to gather data regarding emotional experiences and learning motivation in educational chatbot interactions. Consequently, there were four research questions:

RQ1

What are the specific positive and negative induced emotions experienced by learners when interacting with educational chatbots?

RQ2

Do learners experience different induced emotions based on whether chatbots provide metacognitive feedback or neutral feedback?

RQ3

Do learners experience different learning motivations based on whether chatbots provide metacognitive feedback or neutral feedback?

RQ4

To what extent do induced emotions influence learning motivation, and does this influence vary depending on whether chatbots provide metacognitive feedback or neutral feedback?

Directing attention to induced overall emotions and their impact on motivation bestows substantial value upon both research and practical application. To begin with, this study extended previous studies on emotional response induced by chatbots to include the connection between ambivalent emotions and their influence on learning motivation.

Additionally, the results of this study provide valuable insights for educators and designers, indicating that facilitating positive emotional design in educational chatbots can effectively support and enhance student’s learning experiences and motivation. By thoroughly understanding the diverse emotional experiences associated with educational chatbots and their impact on motivation, educators and designers can potentially focus on designing emotional elements more effectively. Moreover, this study underscores the importance of instructional feedback in human-chatbot interactions. By exploring the differences in emotions and motivations between metacognitive feedback and neutral feedback, valuable guidance can potentially be provided for educators and designers on how to design and integrate appropriate feedback to enhance learning motivation.

Related work

Emotional responses in human-educational chatbot interactions

While chatbots have been gaining popularity, limited research has explored the role of emotions in educational chatbot interactions (Deng & Yu, 2023). The majority of emotional chatbot studies were in the health and well-being domains (Stević et al., 2023; Tudor Car et al., 2020; Vaidyam et al., 2019). Chatbots in education were reviewed by Kuhail et al. (2023). The analysis encompassed various aspects including the educational domain, platform, design principles, chatbot functions, interaction styles, evidence, and constraints. The study found that from a pool of 36 studies, the majority of chatbots were web-oriented, covering areas like computer science, language, education, engineering, and mathematics. They typically acted as instructional agents, following set paths, with some using personalized learning. However, none directly explored emotional responses in interactions.

A similar review by Okonkwo and Ade-Ibijola (2021) revealed various benefits of incorporating Chatbots in education from 53 studies. These advantages include seamless content integration, quick information access, improved motivation and engagement, multi-user support, and immediate assistance. However, these benefits do not address the emotional and motivational aspects of interactions with educational chatbots. This emphasizes the need for continued research into this gap, given the close relationship between emotions and motivation, which significantly influence effective learning support.

Some studies touched on chatbot emotion as a secondary focus. For instance, Liu et al. (2022) investigated how an AI chatbot can enhance students’ reading engagement. Liu designed an AI chatbot to facilitate discussions and provide emotional cues. In a 6-week experiment with 68 students, their active chatbot engagement was correlated with sustained reading interest, echoing Krapp’s engagement framework (Krapp, 1999). The chatbot’s social interactions and emotional cues elicited positive emotions during book discussions, aligning with Krapp’s emphasis on emotional satisfaction in engagement.

Recently, Guo et al. (2023) explored a novel classroom debate approach, using chatbot interactions to foster argumentative dialogues. This approach involved three stages: students interacting with the chatbot to stimulate idea generation, deliberation of generated ideas within student groups, and active participation in debates with other groups, influenced by insights and emotions from chatbot interactions. The study assessed student engagement, encompassing behavioural, cognitive, and affective dimensions. The study collected data from various sources including chatbot interactions, chat logs, audio recordings, and reflections from a group of 24 students divided into four debate groups. An interesting finding of the study was the generally positive attitude displayed by students toward incorporating chatbot interactions into their debate preparation. This positive sentiment stemmed from the chatbot’s ability to inspire innovative ideas, its facilitation of a unique approach to debates, and the emotions induced through these interactions that contributed to a relaxed and productive learning atmosphere.

Likewise, Jasin et al. (2023) developed an automated Question-Answering chatbot for aiding online Chemistry students. The chatbot employed synchronous communication and instructor immediacy techniques. Designed with an affective approach, it aimed to enhance student learning by establishing a humanlike connection. During a pilot study, 12 online Chemistry students at a Singapore university provided qualitative interviews and self-report data. Thematic analysis revealed diverse emotional and behavioural outcomes. Positive impacts included increased confidence and proficiency when the chatbot addressed queries, especially for complex topics or resource direction. However, negative emotional experiences emerged when the chatbot gave incorrect hints or misunderstood questions, exposing limitations. This study highlighted the potential of an affective-focused chatbot to improve online learning that emphasizes accurate comprehension and responses.

Despite some positive emotions have been indirectly found, this leaves a gap in exploring how induced emotions affect learning motivation, particularly in the context of human-educational chatbot interactions. Furthermore, these studies have only touched on positive and negative emotions, neglecting the influence of ambivalent emotions. The potential positive or negative disruption of motivation and students’ behavioural engagement by ambivalent emotions remains uncertain within the context of educational chatbot interactions (Lai et al., 2021; Tze et al., 2022).

Emotions and learning motivation

The influence of emotions and motivation on learning and performance has been a topic of interest among researchers (Ainley, 2006; Kim & Pekrun, 2014; Kiuru et al., 2020; Linnenbrink-Garcia & Pekrun, 2011). Extensive research has shown that motivation and emotions significantly impact learners’ engagement, persistence, and success in academic settings, particularly concerning the impact of classroom activities and learning arrangements (Tulis & Fulmer, 2013). Classroom activities that are intellectually stimulating and emotionally meaningful can result in a profound and impactful manner for students (Williams et al., 2013). Students are motivated when they feel competent, see stable links between actions and achievement, value the subject, and experience positive emotions (Boekaerts, 2010; Meyer & Turner, 2006). Conversely, negative emotions lead to disengagement. A favourable learning environment and the ability to influence emotions enhance persistence and resource management in learning (Boekaerts, 2010). Emotional sensitive instructions can create a more favourable learning environment by suppressing negative emotions like fear and anger and promoting positive emotions such as pleasure and mastery (Astleitner, 2000). However, despite emotions having a crucial role, research has often looked at these factors as the outcome of learning. For instance, the control-value theory of achievement emotions (CVTAE) (Pekrun, 2006, 2017) offers a framework to explore the links between cognition, motivation, and emotion, and has been studied in various learning contexts. CVTAE primarily examines achievement emotions, which are emotions tied to reaching academic goals, such as studying for an exam. The induced emotions in the present study on the other hand refer to the emotions experienced while interacting with an educational chatbot. Induced emotions are influenced by the chatbot’s responses, user individual behaviour, and the overall experience of the chatbot-human interaction (Jin & Youn, 2023). The chatbot’s tone, responsiveness, and effectiveness in providing information or assistance can evoke emotions such as frustration, satisfaction, engagement, or even curiosity. Currently there are limited research investigating the impact of induced emotions on motivation from interacting with chatbots.

Motivation encompasses energy, focus, perseverance, and goal-oriented (Ryan & Deci, 2000b). Among the various factors influencing student motivation, the role of perceived competence is an established link between efforts and academic achievements. When students feel competent and can see the results of their hard work, they are more likely to experience positive emotions and, consequently, exhibit heightened motivation and active participation in their learning journey. Furthermore, the value students place on a subject and the sense of purpose they derive from their studies contribute significantly to their motivation to learn. When students perceive the subject as relevant and meaningful in their lives, they become emotionally invested, leading to increased motivation and a more positive learning experience. According to Ramirez-Arellano et al. (2018), the relationship between motivation and emotions is causal and reciprocal. Consequently, low motivation can result in a lack of initiation or discontinuation of learning tasks, while high learning anxiety may affect intrinsic and extrinsic learning motivations (Wang et al., 2022). Other studies have demonstrated this relationship in different contexts. For instance, Tze et al. (2016) in their meta-analysis found a significant impact of boredom on motivation for secondary and post-secondary students, while Pekrun et al. (2011) revealed that negative emotions like anger, anxiety, and shame can decrease intrinsic motivation when attending class, studying, or taking test and exams in university courses. Emotions have been shown to lead to different actions and action tendencies driven by specific emotional states and their motivational intentions (Scarantino, 2014). In the context of educational chatbot interactions, It is crucial to take into account the impact of negative emotions on working memory, the memory system utilized for retaining and manipulating information during the execution of various mental tasks (Fried, 2011). Additionally, these emotions could potentially interrupt and reorganize learning goals, distracting the learning flow of learners from on-task to off-task goals (Losenno et al., 2020; Zhao, 2011). This highlights the significance of understanding and managing induced emotions during educational chatbot interactions to create a supportive and effective learning experience.

Kim and Pekrun (2014) emphasized the significance of optimizing academic emotions to enhance motivation and improve learning and performance. This involves creating a positive emotional climate in the learning environment that fosters motivation. Educational chatbot designers can draw insights from research findings and address students’ emotional requirements when developing learning experiences.

In sum, emotions and motivation significantly impact learning and performance. Low motivation and high anxiety can impede learners’ progress while optimizing academic emotions can enhance motivation and improve learning outcomes. Despite their importance, emotions and motivation factors remain underexplored in educational chatbot interaction settings. To address this gap, a basic understanding that target learners induced emotional experiences needs further exploration.

Theoretical model

To identify the relationship between induced emotions and motivation, the reciprocal motivation, emotion, metacognition, cognition, and achievement model (MEMCA) was used (see Fig. 1). The MEMCA model (Ramirez-Arellano et al., 2018) derived from the social-cognitive model of academic motivation, emotion, and self-regulation of Artino (2009) and the control-value theory of achievement emotions of Pekrun (2006). Motivation can deactivate negative emotions, also unpleasant emotions can have negative effects on motivation (Pekrun & Linnenbrink-Garcia, 2012). Although existing empirical research has explored the reciprocal relationship between motivation and emotion (Linnenbrink & Pintrich, 2002), there is a limited understanding of how emotions serve as antecedents to intrinsic motivation. Therefore, as shown in Fig. 2, a proposed Induced Emotion–Motivation model was used in this study to examine the potential emotional responses elicited during chatbot-human interactions and their subsequent impact on learning motivation. In this study, the self-determination theory is adopted as a framework to conceptualize students’ intrinsic motivation. Intrinsic motivation is defined as “the doing of an activity for its inherent satisfactions rather than for some separable consequence” (Ryan & Deci, 2000b). Intrinsically motivated students have a high interest in learning tasks, perceive tasks as having high value or importance, and have a greater sense of competence (Ryan & Deci, 2000a; Xie et al., 2006). Therefore, interest, perceived competence, perceived value and perceived pressure are regarded as motivational components in our model.

In the context of this reciprocal relationship, recently Printer (2023) investigated the influence of positive emotions, including enjoyment, interest, and excitement, on learning motivation. The findings from the study demonstrated a substantial connection between positive emotions and intrinsic motivation within the foreign language classroom. This further supports the Induced Emotion–Motivation model, highlighting the role of positive, negative, and ambivalent emotions in influencing intrinsic motivation during chatbot-human interactions.

Fig. 1

Conceptual model of MEMCA (Ramirez-Arellano et al., 2018)

Fig. 2

The proposed Induced Emotion - Motivation Structural model

Method

Measures

The Positive and Negative Affect Schedule (PANAS) Emotional Scale (Watson & Clark, 1994) has been used widely to comprehend human emotions. The PANAS Emotional Scale is known for its comprehensive approach, covering a spectrum of emotions. It embraces positive sentiments like joy, enthusiasm, contentment, and determination, along with negative emotions such as anxiety, sadness, anger, and guilt. This scale has been used to capture the emotional experiences that individuals navigate in diverse situations, proving valuable in understanding emotions across various domains, including interactions with conversation agents (Yang et al., 2019). The PANAS’s simplicity facilitates practical emotional assessments, and its comprehensive coverage enables the computation of ambivalent emotions, enhancing its analytical capacity (Watson et al., 1988).

The Intrinsic Motivation Scale utilized in this study was a modified version of the original Intrinsic Motivation Inventory (IMI) developed by McAuley et al. (1989). The scale consisted of 12 items, rated on a seven-point Likert scale ranging from 1 (strongly disagree) to 7 (strongly agree). As suggested by the IMI description (CSDT, 2023), the interest subscale is considered the self-report measure of intrinsic motivation. Additionally, intrinsic motivation is positively related to the perceived competence and perceived value and intrinsic motivation is negatively related to pressure. Perceived competence pertains to an individual’s perception of effectively accomplishing a task with confidence (Ryan & Deci, 2000b), while perceived value signifies the internalization and self-regulation experienced when engaging in meaningful activities. Each subscale comprised three items. Some sample items from the scale are as follows: “I thought this learning activity was quite enjoyable.“, “I think I am pretty good at this learning activity.“, “I believe doing this activity could be beneficial to me.“, “I did not feel nervous at all while doing this.” .These four subscales were individually examined in this study, employing the scoring criteria outlined in the IMI (CSDT, 2023).

Reliability analysis was conducted for each of the four subscales, yielding satisfactory values (> 0.70) as follows: interest-enjoyment (Cronbach’s α = 0.88), perceived competence (Cronbach’s α = 0.88), perceived value (Cronbach’s α = 0.83), and tension-pressure (Cronbach’s α = 0.77).

The concept of ambivalent emotion is operationalised through the utilization of Kaplan’s (1972) attitudinal ambivalence metric (Leunissen et al., 2020). This metric defines mixed emotions as a combination of positive affect (PA) and negative affect (NA), with the formula ME = PA + NA − |PA − NA|. Nevertheless, scholars like Priester and Petty (1996) have pointed out that this equation can be simplified to twice the lesser value between PA and NA. Thus, mixed emotions can be better understood as the minimum between PA and NA. In our study, we adopt a mixed emotions measure defined as ME = MIN[PA, NA], where ME stands for mixed emotions and MIN represents the minimum (Ersner-Hershfield et al., 2009). Consequently, our measure involves considering the lower-rated emotion, whether it is positive affect or negative affect.

Context and participants

In this study, we employed a between-subject experiment design to assess the impact of induced emotions and learning motivation. Participants were enrolled from a normal university participant pool in Shanghai, China. The sampling method was convenience sampling, which selects participants based on their accessibility and willingness to participate (Creswell & Creswell, 2017). Thus, the study comprised 62 college students, with 42 identifying as female and 20 as male, which was quite representative of the normal university student population. These participants were randomly divided into two groups: the Metacognitive EC group, with 21 females and 10 males, and the Neutral EC group, with 21 females and 10 males. The participants’ majors were diverse, including psychology, education, philosophy, computer science and physics. Because the learning contents were related to biology, students with a biology major were excluded.

The content covered basic concepts and principles from the topic of the human cardiovascular system, which was adapted from a previous study (Lin et al., 2020). Both groups learned identical learning contents and were in the same learning environment. The only variable was the feedback delivered by the educational chatbots. The study received ethical approval for human subject research. Following the experiment, participants received compensation for their involvement.

Procedure

The study used a chatbot development platform Flow.ai to develop two types of EC. Flow.ai is a user-friendly platform that enables custom automation of tasks with less coding. The human-educational interaction took the form of text, buttons, and static images. Students interacted with the EC via a web browser on the computer device.

The experiment for the EC-student interaction and learning task explanation comprised the following steps. Firstly, students arrived at the laboratory, where the researcher provided a comprehensive explanation of the learning task and the process of interacting with EC. Secondly, students took a pre-interaction objective knowledge evaluation test and PANAS survey. The subsequent EC-student interaction sessions consist of three distinct phases: greeting, learning, and self-reflection. In the greeting phase, the EC introduced themselves to the students and requested their names, which the chatbot will use to address them, creating a personalised touch (see Fig. 3). Moving to the learning phase, the EC utilized dialogue boxes to present various learning modules, allowing students to engage in self-paced learning (see Fig. 4). The self-reflection phase encompassed both self-evaluation and EC feedback.

Fig. 3

Greeting phase

Fig. 4

Learning phase

EC feedback during the self-reflection phase varied based on the groups to which students were assigned. In the Metacognitive EC group, students received different metacognitive feedback depending on their self-evaluation. The metacognitive feedback aimed to guide students in assessing their understanding, which was adapted from previous studies (Yılmaz et al., 2018; Cabales., 2019; Zheng et al., 2019). If students selected the “understand” option, the system posed questions to help them evaluate, reflect, monitor, and plan their learning (see Fig. 5). Conversely, choosing the “don’t understand” option prompted questions to monitor knowledge comprehension, effort, and concentration, such as “how much attention do you pay while learning”. In contrast, the Neutral EC group was instructed to take a 10-second break without receiving any feedback, regardless of the outcomes of their self-evaluation (see Fig. 6). Following this, the Metacognitive EC group participated in a self-reporting process, whereby they rated their level of agreement with the metacognitive descriptions using a Likert-type scale, ranging from 1 to 5 whereas the neutral EC group omitted this interaction.

Subsequently, the ending step, lasting for approximately 15 min, focused on students’ completion of an objective knowledge evaluation, the assessment of PANAS (Positive and Negative Affect Schedule) and the intrinsic motivation inventory (IMI) scale from a survey website. Figure 7 depicts the experiment procedure.

Fig. 5

Self-reflection phase in the Metacognitive EC group

Fig. 6

Self-reflection phase in the Neutral EC group

Fig. 7

EC interactions experiment procedure

Results

Comparing specific induced emotions between educational chatbots (RQ1)

Table 1 depicts the induced positive emotion between Metacognitive EC and Neutral EC. The Metacognitive EC prompts a significantly higher sense of interest (M = 3.77, SD = 0.560) compared to the Neutral EC (M = 3.16, SD = 0.934; p = 0.001). Similarly, feelings of pride are significantly more pronounced with the Metacognitive EC (M = 3.10, SD = 1.076) as opposed to the Neutral EC (M = 2.52, SD = 1.029; p = 0.017). The Metacognitive EC also leads to a greater sense of inspiration (M = 3.16, SD = 0.969) than the Neutral EC (M = 2.65, SD = 1.050; p = 0.024). While not statistically significant, there is a trend suggesting that the Metacognitive EC might induce a stronger sense of excitement (M = 3.32, SD = 0.909) compared to the Neutral EC (M = 3.00, SD = 0.730; p = 0.064). Similarly, the Metacognitive EC evokes a slightly stronger feeling of determination (M = 3.06, SD = 0.998) in comparison to the Neutral EC (M = 2.77, SD = 0.990; p = 0.127). No statistically significant differences are observed between the Metacognitive EC and the Neutral EC for emotions such as “strong,” “enthusiastic,” “alert,” “attentive,” and “active.”

In summary, the analysis highlights the varying positive emotional impacts of the Metacognitive EC and the Neutral EC. The Metacognitive EC stands out for fostering greater interest, pride, and inspiration, whereas trends suggest it might also elicit heightened excitement and determination. However, emotions such as strength, enthusiasm, alertness, attentiveness, and being active remain comparable between the two chatbots.

Table 1 Comparing positive emotion between Metacognitive EC and Neutral EC

Table 2 depicts the induced negative emotion between Metacognitive EC and Neutral EC. The emotions assessed included distress, upset, guilt, fear, hostility, irritability, shame, nervousness, jitteriness, and general fear, with emotions measured on a numerical scale indicative of emotional intensity. Analysis of the data revealed both significant differences and similarities between the emotional impacts of the two chatbots. The Metacognitive EC exhibited a statistically significant capacity to evoke lower levels of distress (M = 1.71, SD = 0.643) compared to the Neutral EC (M = 2.29, SD = 1.071; p = 0.006). Similarly, the Metacognitive EC elicited diminished levels of nervousness (M = 1.71, SD = 0.693) in contrast to the Neutral EC (M = 2.13, SD = 0.991; p = 0.029). Additionally, the Metacognitive EC demonstrated decreased jitteriness (M = 1.42, SD = 0.672) relative to the Neutral EC (M = 1.84, SD = 1.036; p = 0.032). Notably, a trend towards significance was observed in the case of hostility, with the Metacognitive EC (M = 1.06, SD = 0.250) tending to evoke less hostility compared to the Neutral EC (M = 1.26, SD = 0.631; p = 0.059). For negative emotions including guilt, fear, shame, upset, and general fear, there were no statistically significant differences observed between the Metacognitive EC and the Neutral EC.

Table 2 Comparing negative emotion between Metacognitive EC and Neutral EC

In sum, the examination of the negative emotional responses evoked by the Metacognitive EC and the Neutral EC underscores the pivotal role of chatbot design in influencing user emotions. While the Metacognitive EC was associated with lower levels of distress, nervousness, and potentially hostility, emotional reactions to guilt, fear, and shame were similar between both chatbots.

Comparing overall induced emotions between educational chatbots (RQ2)

Table 3 depicts a comparison between educational chatbots on overall induced emotions during interactions. In the case of the Neutral EC, interactions led to a significantly higher level of overall positive emotion (PE) compared to overall negative emotion (NE), with a mean difference of 1.21 and a Cohen’s d effect size of 0.84. Similarly, the Metacognitive EC also showed a significant difference in overall positive emotion, with a mean difference of 1.70 and a Cohen’s d effect size of 0.73. These results suggest that both types of educational chatbots, whether neutral or Metacognitive, effectively elicited positive emotions and surpassed negative emotions, indicating their potential to enhance user experiences and engagement within educational contexts.

Table 3 Comparing overall positive minus overall negative emotion within EC

The data presented in Table 4 offers a comparative analysis of overall ambivalent, positive, and negative emotions triggered by two educational chatbots. The Metacognitive EC elicits a mean positive emotion score of 3.12 (SD = 0.63), which is somewhat higher than the Neutral EC’s mean score of 2.88 (SD = 0.53). Despite the p-value (p = 0.056) not attaining traditional significance thresholds, a discernible trend suggests a potential differentiation in positive emotional impact between the two chatbots. The Metacognitive EC yields a mean negative emotion score of 1.41 (SD = 0.40), in contrast to the Neutral EC’s mean score of 1.67 (SD = 0.68). With a p-value of 0.038, indicating that the Metacognitive EC appears to engender fewer negative emotions than the Neutral EC. In terms of ambivalent emotions, the Metacognitive EC’s mean score stands at 2.83 (SD = 0.80), while the Neutral EC exhibits a higher mean score of 3.23 (SD = 1.12). Although the p-value of 0.053 falls slightly short of traditional significance levels, a discernible trend is evident, suggesting the Metacognitive EC’s potential to evoke fewer ambivalent emotions compared to the Neutral EC.

Table 4 Comparing overall ambivalent, positive and negative emotions between EC

In summary, the data analysis underscores the potential disparities in emotional responses elicited by the Metacognitive EC and the Neutral EC. While trends are discernible for differences in positive and ambivalent emotions, the Metacognitive EC demonstrates statistical significance in evoking lower levels of negative emotions.

Comparing learning motivations between educational chatbots (RQ3)

Table 5 provides a comparison of learning motivation between the two educational chatbots. The “Metacognitive EC” yields a significantly higher mean interest score (M = 5.52, SD = 1.14) than the “Neutral EC” (M = 4.32, SD = 1.46; p = 0.001), indicating that the “Metacognitive EC” effectively cultivates a greater sense of interest in learners. Similarly, the “Metacognitive EC” demonstrates a significantly higher mean competence motivation score (M = 4.87, SD = 1.26) compared to the “Neutral EC” (M = 3.94, SD = 1.47; p = 0.009), suggesting that the “Metacognitive EC” enhances learners’ perception of their competence. In terms of value motivation, the “Metacognitive EC” obtains a significantly higher mean score (M = 6.12, SD = 0.79) than the “Neutral EC” (M = 5.46, SD = 1.03; p = 0.006), indicating that the “Metacognitive EC” effectively emphasizes the value of learning. However, there is no statistically significant difference in pressure motivation between the “Metacognitive EC” (M = 3.06, SD = 1.23) and the “Neutral EC” (M = 3.48, SD = 1.33; p = 0.204), suggesting comparable effects of the two chatbots in this aspect.

In summary, Table 5 shows clear differences in how the “Metacognitive EC” and the “Neutral EC” impact learning motivation. The “Metacognitive EC” is better at promoting interest, making learners feel competent, and helping them see the value in learning. This suggests that the “Metacognitive EC” could help students get more engaged and motivated in their learning.

Table 5 Comparing learning motivation between EC

Influence of induced emotions on learning motivation (RQ4)

To verify the proposed model, we use the structural equation modeling (SEM) approach, employing SmartPLS4.0 SEM analysis to allow the simultaneous computation of relationships across the entire model (Goh & Kinshuk, 2006). Table 6 depicts the items’ loading and construct reliability. The loading and reliability values for the motivational factors all exceeded the threshold of 0.7 whereas 3 items in the emotional dimension exceeded only 0.6 but were retained. Cronbach’s alpha and composite reliability all exceeded the threshold of 0.7. All the average variance extracted (AVE) exceeded the threshold of > 0.5 indicating good convergence validity (HairJr et al., 2021).

Table 6 Items loading, construct reliability and convergence validity

Table 7 shows the Heterotrait–Monotrait ratio (HTMT) of correlations where values are below 1 with one value exceeding 0.9, indicating a satisfactory level of discriminant validity among the constructs. However, considering Ambivalent Emotion’s (AE) derivation from positive and negative emotions, a higher ratio involving AE and negative emotion is expected.

Table 7 Heterotrait–Monotrait ratio (HTMT)

Table 8 shows the paths in the emotion-motivation model. The path from “Interest” to “Competence” is strong and positive (b = 0.669, p < 0.001), meaning that when people are interested, they feel more competent. Similarly, the path from “Interest” to “Pressure” is important (b = -0.478, p < 0.001), showing that more interest can mean less pressure. The path from “Interest” to “Value” is also important (b = 0.665, p < 0.001), indicating that interest connects with seeing value. Moreover, the path from “Ambivalent” to “Interest” matters (b = 0.385, p = 0.010), showing that mixed feelings can influence interest. Additionally, both paths from “Negative” to “Interest” (b = -0.53, p = 0.018) and from “Positive” to “Interest” (b = 0.363, p = 0.022) are meaningful, explaining how these emotions affect “Interest.” These findings help us understand how these emotions work together in the model.

Table 8 Structural paths of both educational chatbots

A multi-group SEM comparison was employed to explore potential path differences between the two educational chatbots. The objective was to investigate whether these chatbot types elicited distinct responses within the examined paths. Table 9 offers a comparison of the emotion-motivation structural model between the “Metacognitive EC” and the “Neutral EC”. The paths “Interest -> Competence” (d = 0.044, p = 0.756), “Interest -> Pressure” (d = -0.135, p = 0.478), “Interest -> Value” (d = 0.038, p = 0.734), “Ambivalent Emotion -> Interest” (d = -0.563, p = 0.166), and “Negative Emotion -> Interest” (d = 1.105, p = 0.169) all exhibit insignificant differences between the chatbots. These findings highlight the similarity in the emotional-motivational relationship by both “Metacognitive EC” and “Neutral EC” in these aspects. In contrast, the “Positive Emotion -> Interest” path reveals a substantial difference (d = -0.594, p = 0.044), suggesting a reduced impact of the “Metacognitive EC” compared to the “Neutral EC” on the relationship between positive emotions and interest.

Table 9 Comparing emotion–motivation structural model between educational chatbots

Discussion

This study was an initial effort to investigate the connection between emotional experiences and their influence on learning motivation in the context of human-educational chatbot interactions. The research explores overall positive, negative, and ambivalent emotions induced during chatbot interactions, as well as the impact of metacognitive feedback on emotions and learning motivation. Below we discuss our findings for each research question.

The study revealed that interacting with both types of Educational Chatbots (EC) can simultaneously evoke a spectrum of emotions, including positive, negative, and ambient emotions, demonstrating the dynamics of user engagement with educational chatbots. This finding demonstrates the diversity in induced emotional responses is important as it mirrors the complexity of human emotions in real-life learning scenarios (Efklides & Volet, 2005). It suggests that the educational chatbots in this study can add depth and realism to the interaction. A diverse positive emotional experience such as excitement, enthusiasm and inspiration can be interpreted as users being engaged, interested, satisfied, and motivated to continue interaction (Gkinko & Elbanna, 2022). On the other hand, negative emotions, like frustration or confusion, may indicate areas of difficulty or areas for improvement in the educational content (Maldonado et al., 2022). Ambivalent emotions, which are more subtle and pervasive, contribute to the overall emotional tone of the learning experience. This may suggest that the design of the educational chatbots, while successful in engaging learners, still contains a small level of confusion and ambiguity situations among them (Gkinko & Elbanna, 2022).

The finding that both neutral and metacognitive educational chatbots can effectively elicit positive emotions while surpassing negative emotions is similar to the findings of Yang et al. (2019) and Qu et al. (2022). Yang et al. (2019) employed a critical incident method with social media conversation agents such as Siri, Google Assistant, Alexa and Cortana, while Qu et al. (2022) utilised an experimental design with a Chatbot teaching assistant to study users’ emotional response with chatbot interactions. Their findings revealed that users of conversational agents tend to experience more positive emotions than negative emotion. This outcome suggests that the design elements contributing to neutral and metacognitive chatbots are effective in creating more emotionally positive interactions. For learners, this implies a potentially more engaging and enjoyable learning process. However, the novelty effect may have contributed to the response and should not be ignored. The novelty effect suggests that individuals may initially respond more positively to new and novel experiences, and over time, this initial excitement may diminish (Huang et al., 2022). Therefore, while the positive interactions observed in the study are promising, designers and educators must consider the potential durability of these effects and how they might evolve as users become more familiar with chatbot technology.

The study revealed differences in the impact on learning motivation between the “Metacognitive EC” and the “Neutral EC.” Specifically, the “Metacognitive EC” rated high in fostering interest, instilling a sense of competence, and highlighting the value of learning. This suggests that the inclusion of metacognitive feedback elements significantly enhances learner engagement and motivation. Our finding aligns with the study by Karaoglan Yilmaz and Yilmaz (2022), where delivering personalized metacognitive feedback to students in online learning improved their engagement. A potential explanation for the observed outcome could be that the metacognitive feedback process encourages students to actively regulate their learning, leading to improved recall and comprehension. As a result, students may become more motivated in their studies (Lee et al., 2010).

From our findings, the absence of feedback in a chatbot learning environment can significantly heighten learner nervousness and distress. Feedback serves as a compass, providing direction, validation, and encouragement. Without this guidance, according to Butler and Nisan (1986), learners may be confused as to the source of motivation, unsure of the accuracy of their responses and lacking the motivational reinforcement that positive feedback can offer. The void of feedback also denies learners valuable insights into their progress, hindering their ability to self-assess and adjust strategies (Haddara & Rahnev, 2022).

The findings from the emotion-motivation PLS analysis revealed that heightened interest positively correlates with increased competence, potentially improving learning outcomes. Conversely, increased interest may alleviate feelings of pressure, fostering a more positive learning environment. Emotional engagement, specifically interest, plays a crucial role in recognizing the value of learning content (Krapp, 1999). The study also highlights the impact of ambivalent emotions on interest and its role in shaping motivation. Our findings are consistent with Mega et al. (2014) study, which suggests that students’ emotions impact their self-regulated learning and motivation, subsequently influencing academic achievement. These results align with the Self-Determination Theory (SDT) (Ryan & Deci, 2000b) which posits that intrinsic motivation, driven by interest, leads to more sustainable and meaningful learning experiences.

The findings of the Structural Equation Modeling (SEM) multi-group comparison between the “Metacognitive EC” and the “Neutral EC” offer valuable insights. First, the analysis reveals that, in various paths such as “Interest -> Competence,” “Interest -> Pressure,” “Interest -> Value,” “Ambivalent Emotion -> Interest,” and “Negative Emotion -> Interest,” there are no significant differences between the two chatbot types. This suggests that the interactions with both “Metacognitive EC” and “Neutral EC” lead to similar experiences, emphasizing the consistent design of the chatbots in cultivating comparable emotional-motivational relationships. However, the negative difference in the “Positive Emotion -> Interest” path (d = -0.594, p = 0.044) suggests that the impact of “Metacognitive EC” on the relationship between positive emotions and interest is reduced compared to the “Neutral EC.” Metacognitive awareness may play a role in this outcome. Through metacognitive feedback, students can enhance their awareness of cognitive processes. This involves recognizing their strengths and weaknesses, gathering pertinent information about the given task, and formulating effective strategies for task completion (Urban et al., 2021). Elevated metacognitive awareness may have affected the dependency on emotions for regulation in learning (Lee & Son, 2023). Further research could investigate other factors and different feedback types such as affective feedback, corrective feedback or delayed feedback to better understand the dynamics of emotion and learning in the educational chatbot environment (Wu & Yu, 2023).

Implications

Considering the research findings, this study has some practical implications for educational chatbot designers and educators in higher education. To begin with, designers and educators ought to recognize and comprehend the diverse emotions induced by human-educational chatbot interaction and how they impact learning motivation. Therefore, designers and educators should utilize richer design elements to create more emotionally positive interactions, subsequently promoting learning and performance. For instance, designers and educators could enhance the learning experience with anthropomorphic emotional cues, speech, dynamic pictures, videos and so on. Utilizing multimedia attributes and personalized characteristics of chatbots can enhance interest motivation (Ryan & Deci, 2000b; Yin et al., 2024).

In addition, designers and educators should realize the importance of feedback. While EC can provide social interaction and emotional cues to induce positive emotions, it can also lead to negative emotions due to the absence of feedback. Integrating EC with appropriate feedback can promote not only effective responses but also the support of targeted cognitive and metacognitive strategies (Saks & Leijen, 2019). Furthermore, the findings reveal that metacognitive feedback reduces negative emotions, and fosters learning interest and motivation. Thus, designers and educators can also provide different types of feedback according to different learning situations and facilitate learning support.

Limitations and future directions

This study has several limitations that should be addressed in the future. Firstly, it has limited statistical power to detect significant differences because of the participants’ nature and small sample size within a normal university in China. Therefore, further studies need larger and more diverse samples to validate the results. Secondly, some external factors that may moderate the potential effects of the chatbot type were not considered, such as users’ prior experiences, personal characteristics, or the context of interaction. These moderating variables could have influenced users’ responses in ways that obscured any distinctions between the chatbot types that were not considered. Lastly, the study only evaluated short-term interactions, any potential differences between the chatbot types might become more pronounced over an extended period. Users might need more time to differentiate and respond differently to the distinct characteristics of the chatbots. Thus, it emphasizes the need for longitudinal investigations to gauge the potential durability of these effects.

Conclusion

In conclusion, this study revealed the diverse emotional experiences learners have with educational chatbots and their impact on motivation. Both neutral and metacognitive chatbots were effective in evoking positive emotions, aligning with the principles of the Self-Determination Theory. Metacognitive chatbots, however, showed a stronger ability to foster interest, competence, and the value of learning. The absence of feedback heightened learner nervousness, emphasizing its crucial role in providing direction and motivation. The emotion-motivation model highlighted the correlation between interest, competence, and its potential to improve learning outcomes. While the study found consistent emotional-motivational relationships across chatbot types, the reduced influence on the relationship between positive emotions and interest in the metacognitive feedback chatbot highlights avenues for future research to identify specific factors or mechanisms influencing this unique pattern.

In addition, this study contributes to and adds value to existing but rapidly growing literature in two distinct ways. From a research perspective, this study addresses a significant gap in the current literature by responding to the lack of research on emotional responses arising from human-educational chatbot interactions. The study bridges this gap by investigating how learners’ emotional experiences with educational chatbots influence their motivation to learn. From a practical perspective of chatbot design, the study’s findings demonstrate the need for a better understanding of the induced emotional dynamics and metacognitive features to enhance learner motivation.