An architectural design and evaluation of an affective tutoring system for novice programmers

Emotions and cognition are intertwined as evidenced by studies (Dolan & Vuilleumier, 2003; Kort, Reilly, & Picard, 2001; Linnenbrink, 2006; Phelps, 2004; Vuilleumier, 2005) investigating the interactions between emotion and attention, learning and memory. Anatomically, Gray, Braver, and Raichle (2002) also points to the integration of emotional and cognitive processes in prefrontal cortex of the human brain. This anatomic link was played out in real life in the unfortunate events experienced by Elliot, a patient of Damasio (1994). As a result of a tumor cut from his cortex near the brain’s frontal lobe, Elliot, despite having a high level intelligence, became incapable of making simple decisions in his daily life and this eventually led to failures in both his career and marriage.

The link between emotions and cognition prompted studies investigating into the relationship between emotions and the learning achievement of students. Pekrun, Goetz, Titz, and Perry (2002) opined that academic emotions (which are linked to academic learning, classroom instruction and achievements) with the exception of test anxiety are largely neglected by educational psychology research. They embarked on a series of qualitative and quantitative studies to uncover the academic emotions experienced by students and the effects of these emotions on the students’ learning and achievements. Their findings revealed that students experience a wide repertoire of emotions in academic settings and both positive emotions e.g. hope, pride and relief and negative emotions such as anxiety and boredom are prevalent in learning. The positive emotions e.g. enjoyment of learning affect achievement positively by strengthening motivation and enhancing flexible learning whereas the negative emotions e.g. anxiety erodes motivation and draws attention away from the task, resulting in shallow learning. The effects that both positive and negative emotions have on learning were also corroborated by other researchers (Ashcraft & Kirk, 2001; Greene & Noice, 1988; Isen, 2000).

Bloom (1984) highlighted the 2 sigma problem which states that students who are tutored on a one to one basis achieve learning gains of 2 sigma over and above those students who are group tutored. He attributed this difference to the process of constant corrective feedback and high engagement of the tutee, maintained through close monitoring and constant encouragement by the tutor. To achieve enhanced learning outcomes, it is crucial that the active engagement and motivation of the student be sustained through this continuous affective feedback loop between the tutor and the student.

In a study by Efklides (2006) on metacognition and affect, she postulated student perceived task difficulty as a factor influencing metacognition in learning. She reasoned that if a very difficult task is assigned to students and they perceived it as beyond their capability, they would develop negative emotions such as frustration and give up on the task. However, if teachers are able to scaffold the students’ learning or employ the use of alternative pedagogical intervention techniques e.g. provision of hints to alleviate the negative emotions, they would be able to sustain the engagement of the students.

Intelligent Tutoring Systems are built with the objective of providing learners with the benefit of one to one tutoring automatically and cost effectively. It acts like a personal training assistant that continually assesses one’s knowledge through interactions with the system and builds a personalized model of one’s acquired knowledge for the provision of tailored instructions or assistance in the form of hints or demonstrations when one seems to require help to move on.

A common criticism of ITS is that they are devoid of emotional awareness and empathy and that limits their tutoring effectiveness (Lepper, Woolverton, Mumme, & Gurtner, 1993). Pivotal to the effectiveness of human tutors in one to one tutoring is the fact that human tutors are able to constantly monitor both the affect and cognitive ability of their tutees and respond with adequate supportive tutoring measures (Goleman, 1995). Analogous to the way human tutors detect and respond to the affect of the their tutees to sustain the engagement of the latter, Affective Tutoring Systems (Picard, 1997) adapt to the tutees’ affect autonomously to bring about enhanced learning outcomes. Most Affective Tutoring Systems (ATSs) incorporate affect sensing, tutoring strategies and learning progress tracking into a single environment. With the ability to sense the affect of students, ATSs can for instance, sense that students are frustrated and offer hints to resolve the impasse.

Although researchers acknowledge the role that affect plays in learning, ATSs on the contrary are rarely implemented (Thompson & McGill, 2012). This is often attributed to the fact that the implementation of ATSs encompasses cross-disciplinary knowledge spanning domains of education, psychology and computer science. Often, the implemented ATS is highly specialized and purpose-built for a particular tutoring subject area, making replication or customization for a different subject area difficult, if not impossible.

Augmenting ITS with the ability to infer the affect of students is the first step in the construction of an ATS. The ensuing challenge is then how do we design the ATS such that it can respond appropriately to the detected affect of students to increase engagement, task persistence and sustain their motivation to learn. The goal is to regulate or ease the negative emotions such as frustration and disengagement occurring in learning so that it does not degenerate into hopelessness. It would be ideal if a recipe that lists the pedagogic strategies to be applied for every tutoring situation that may arise in an ATS exists. This gap in the literature however, has yet to be addressed.

In addition, most implemented ATSs were also not evaluated for their effectiveness and acceptance by the students. Thus, this study adds to the body of knowledge by proposing an architecture for an ATS that tutors in the domain of computer programming and evaluating for its effectiveness and acceptance.

This paper describes the design, implementation and evaluation of an ATS that tutors students in the subject of computer programming. To cater for the generalizability of the ATS for subject areas other than computer programming, the components within the architecture are designed to be loosely coupled so that they can easily be replaced or extended. Equipped with the ability to infer affect, appropriate tutoring response strategies based on psychological principles are also built into the ATS for the regulation of negative affect. A pedagogical based approach is also adopted in this study for the design of the programming exercises and the hints. To close the loop, we conducted qualitative evaluation using focus group discussions to gauge the ATS’s effectiveness and it’s acceptance by the students.

Proposed architecture

Figure 1 shows the proposed architecture of the ATS. It comprises of the Sensor Subsystem, the Student Subsystem and the Tutoring Subsystem. The functions of each subsystem are detailed in the sections below.

Sensor subsystem

The Sensor Subsystem acquires and processes the raw data from the various sensors e.g. web camera for facial cues, input devices such as keyboards and mouse for keystrokes and mouse clicks and eye trackers for eye tracking patterns. All the sensor logs will be consolidated and processed to derive a list of relevant features. The processed features are then passed to the Affect Inference module for inference of the student’s affective states.

For this study, our considerations for the choice of sensors are their unobtrusiveness and ease of deployment. Previous studies have tapped on input modalities ranging from facial expressions (Kapoor, Burleson, & Picard, 2007; Kapoor & Picard, 2005; Teeters, El Kaliouby, & Picard, 2006; Yeasin, Bullot, & Sharma, 2006), body postures (Coulson, 2004; Mota & Picard, 2003), physiological (AlZoubi, Calvo, & Stevens, 2009), text (Shaikh, Prendinger, & Ishizuka, 2008) and input devices (Bixler & D'Mello, 2013) for affect sensing. Some of these sensors e.g. physiological sensors require probes to be placed on parts of the subject’s body which subjects may find to be obtrusive. Being overly conscious of the presence of these sensors will in turn affect the reliability and validity of the measurements. Most of the physiological sensors also require an adequate level of technical expertise to setup which further constraints their deployment in the classrooms. In contrast, in this study, our proposed array of sensors comprising of web cameras, keyboards and mouse are common commodity devices that are unobtrusive and readily available in most learning environment.

Student subsystem

The Student Subsystem consists of both the Affect Inference and the Student’s Action modules. The Affect Inference module will be in charge of inferring the affect of the student. This inference can be achieved through the use of machine learning algorithms that mine through the sensor logs to identify common patterns. The inferred affect will be passed to the Tutorial Strategy module for formulation of an appropriate tutorial strategy that regulates the inferred affect.

The Student’s Action module logs the history of student’s interactions with the ATS. The interactions that are logged can include the duration of time spent on an exercise, the topic and exercise the student is working on and has completed, date and time of compilations, the number of errors encountered for each compilation attempt, the date and time a hint is provided and the number of hints provided to each student. The interaction logs are in turn passed as input to the Affect Inference module for inference of the student’s affect.

Tutoring subsystem

The inferred affect of the student obtained from the Affect Inference module and the history of interactions of the student obtained from the Student’s Action module are passed on to the Tutorial Strategy module within the Tutoring Subsystem. A Rete algorithm based rule engine is embedded within the Tutorial Strategy module. The proposed design of the tutoring response strategy rules based upon the coping strategies elaborated in the related work section is shown in Fig. 2. The tutoring strategy rules are encoded within the rule engine. Using a rule engine offers the ease of modification of the strategy rules by non-programmers as the rules are stored in files externalized from the ATS.

One principle of design of the help system provided by the tutor is that it must be contingent on the students’ needs (Wood & Wood, 1999). When students make a mistake, it is difficult for the system to ascertain whether it is due to slips, misconceptions or a lack of knowledge (Aleven, Stahl, Schworm, Fischer, & Wallace, 2003). If we offer only hint that points at a generic concept or area to look into, weaker students would most likely be still stuck. On the other hand, if we provide hint that is so detailed that it almost gives the answer away, the more proficient students who slip occasionally would not be sufficiently challenged and may also be bored or frustrated by the need to go through an elaborate explanation text when they already know the concept. This justifies putting the request for help under the student’s control and devising hints that progressively increase in level of details.

In addition, by opting for the provision of multiple levels of help, students who slip would not have to go through the more detailed explanations that are meant for students with a genuine lack of knowledge. For the students who are genuinely stuck due to knowledge gaps, the help provided would have to be progressively more explicit until the student is able to resolve the problem.

The first instance when students request for hint, a generic hint relating to the topic would be displayed. For subsequent help requests, detailed hints would be provided to the students. The detailed hints are bottom out hints that not only direct the students on the steps to be taken to rectify the error but also provide an explanation on the error.

To illustrate, if the student uses an assignment operator (“=”) instead of relational equals to operator (“==”), instead of just outputting “replace = with == in line 12”, the full explaination text “The assignment operator (=) is used to assign a value to a variable and a relational equals to operator (==) in a==b is used to check whether the variables a and b equal in values. == should be used in place of = in line 12.” will be shown instead. This will help address the knowledge gap of the student as the detailed explaination enhances the student’s understanding on the rationale for the rectification steps.

The flow of the student’s problem solving process is shown in Fig. 3. Students will first key in codes into the missing lines within exercises in the ATS. The students will then compile the codes within the ATS and when errors are encountered, students will try to debug and rectify the errors on their own. Some students may experience frustration after repeated futile attempts to rectify the errors. The regulation of negative affect such as frustration will be handled by the tutoring response strategy rules within the Tutoring Subsystem.

To illustrate, for the currently configured tutoring response strategy rules, the ATS upon detecting that the students are frustrated, will initiate the display of prompts on screen to remind the students that they can request for hints to resolve the errors. For the version of ATS that was deployed for the trial, these prompts are faded in from the side of the screen and automatically faded out after 8 s. Our design consideration is that the prompts should not be disruptive to the students. In a study by Robison, McQuiggan, and Lester (2009), they concluded that the risk of students reacting negatively to incorrect affective interventions may be high for some affective states and caution should be exercised to avoid unintended effects of inadequate interventions. In consideration of these, we adopted a less disruptive and non-interruptive way of displaying the reminder prompts where students do not have to click on a button to close the prompt. At any instance, the students can also request for hints on their own without the reminder message by clicking on the request for hint button. If the students did not request for hints for 10 s after the system detected that they are frustrated, the system will fade in an empathetic message encouraging the students to continue resolving the errors on their own. This would be an appraisal-focused response (Weiten et al., 2011) that alleviates the frustration by diverting their attention away from the negative affect itself.

Other emotional-focused and attribution based tutoring responses include the display of congratulatory message when the student successfully complete an exercise, attributing the success to their positive effort. Conversely, when the ATS detects that the student is disengaged and has worked on the designated programming exercise for less than 15 min, prompts reminding the student to devote more attention to the task and the need to invest sufficient effort would be displayed. If the disengaged student has already worked on the designated programming exercise for more than 15 min, the ATS would then recommend that the student work on an exercise with a lower difficulty level. This is an example of a problem-focused response strategy.

Design of Programming Exercises

The Cognitive Load Theory (Sweller, 1988) provides the framework for presentation of instructional information to optimize the cognitive load which will in turn lead to enhanced intellectual performance. It theorizes that if the total cognitive load exceeds the available working memory capacity, learning would not be able to occur. An effective instructional design reduces the cognitive load which puts less strain on the working memory requirements. The freed working memory in turn allows the learner to acquire other more advanced schemas which leads to improved learning when repeated over many cycles.

Computer programming is notoriously difficult to learn and novice learners face a range of difficulties ranging from misconceptions on the programming constructs to failure to translate their intention into logical programming codes. The construction of a computer program requires the programmer to retrieve pre-existing knowledge on the program syntax and semantic in their working memory and then devise a plan for combining these together to fulfil a function.Working memory has in fact been established to be a good predictor of programming skill acquisition in a previous study by Shute (1991).

For a novice learning programming, the cognitive load of having to memorize the syntax and semantics of the various programming constructs may overwhelm their working memory capacity. A technique to reduce the cognitive load employed in this study is the fill in the gap programming model. The fill in the gap programming model is employed for the exercises within this ATS where students are required to fill in missing lines of codes within a provided code template for each exercise. A screenshot of a sample exercise within the ATS is shown in Fig. 4. Using a fill in the gap model not only affords the reduction in the cognitive load of the novice students but also helps the students to overcome the initial inertia of not knowing how to start writing the program – a common issue that is encountered by students learning how to program.

In this study, both qualitative and quantitative evaluations are used for evaluation of the proposed ATS. I conducted both an experimental study and a series of focus group discussions to evaluate the effectiveness and acceptance of the proposed ATS.

For the experimental study, the effectiveness of the ATS for the students’ learning was evaluated. As the proposed ATS is to be used as a Java programming tutor, to evaluate its effectiveness, I seek to measure the learning achievement of the students during the trial. Specifically, the learning achievement is defined as the number of exercises completed, the duration of time taken to complete each exercise, the number of exercises attempted and the number of compilation required to complete each exercise. These can be derived from the students’ action logs that are captured within the Student’s Action Module elaborated in the section on Proposed Architecture. The intuition is that if the ATS is effective, the students would be able to attempt and complete more programming exercises and complete them in a shorter amount of time with lesser compilations. Our hypothesis for this study is thus the full affective version of the ATS results in more effective tutoring as compared to the version with the affective function disabled.

This measurement of learning achievement through tracking the number of programming exercises completed and the time taken to complete them was also adopted in the study by Corbett and Anderson (2001). In his study, he further concluded that test achievement was strongly related to the set of problems the students successfully solved and the solution paths that the students follow had little or no impact (Corbett & Anderson, 2001), thus validating our approach to focus on the outcomes and not the process of problem resolution for measurement of learning achievement.

Focus group discussion

Focus group discussion is a group interviewing technique relying on interaction within the group based on topics that are selected by the researcher (Morgan, 1996). The participants within the focus group are usually selected by the researcher. In a focus group, the interaction among the members of the group allows the researcher to extract the attitudes, feelings, beliefs and experiences of the members which may not be feasible with the use of other research methods.

In this research, the developed ATS will likely be deployed in a classroom to complement a human tutor in the teaching of the designated subject area. The use of focus group discussions facilitates the retrieval of a range of students’ perceptions on the usability and acceptance of the system by the students. In addition, it allows for the extraction of genuine feelings and opinions of the students, though on the condition that the environment is comfortable and conducive for them to voice their views. In a focus group discussion, it is envisaged that a group’s capability would be more than the sum of its parts, thus generating more opinions than what can be obtained from the sum of opinions from each individual interview.

In our context, a total of 4 focus group discussions were conducted with the objective of evaluating the effectiveness of the ATS. Twenty-eight students (different from the students involved in the experimental trial) in total were recruited for the focus groups with 6, 7, 8 and 7 students for each of the 4 focus groups.

The participants were asked to trial the ATS for an average of 30 min before the start of the focus group discussion to familiarize them with the functions of the ATS. The focus group discussions were next held in classrooms with the participants seated in a circle. Each focus group discussion lasted for an average of about 20 min. The participants were asked to self-introduce themselves at the start of the session. All the focus group sessions were conducted by a moderator and were tape-recorded. After each session, the tape recordings were then transcribed. The analysis and consolidation of the focus group discussions transcripts using the content analysis methodology is summarized in Fig. 5. Two moderators with an average teaching experience of 7 years each conducted the focus group discussion sessions.

The set of guiding questions for the focus group discussions are

1. 1.

   How was your learning experience with the ATS?

    
2. 2.

   Why do you feel that way about the ATS?

    
3. 3.

   What do you like about the ATS?

    
4. 4.

   What do you think can be improved for the ATS?

    
5. 5.

   What do you think of the feature where the system senses your frustration and engagement and respond accordingly e.g. by providing you with hints if it senses that you are frustrated?

    
6. 6.

   What else would you like to say about the tutoring experience or about the system?

    

These questions serve as a guide to the moderator and the moderator is free to rephrase and re-sequence the questions when he or she conducts the discussion session.

During the conduct of the session, the moderator tries to ensure that every participant has an equal opportunity of expressing his or her view on the topic and that the session is not dominated by any individual participant. After the session, the moderator thanks each participant for their participation and assures the participants that their views are valued and will contribute towards the enhancement of the ATS which will in turn benefit future users of this ATS.

The results of the experimental study show that the time taken to complete each exercise and the number of exercises attempted by the students differs significantly between the non-affective and affective groups. The number of exercises completed and the number of compilations were however not significantly different between the two groups. This implies that the use of the full affective version of the ATS enhances the efficiency (solving exercises in a shorter time) and persistence (expending effort in attempting more exercises) of the students.

It is surprising though that the number of exercises completed was not significantly different. Closer examination of the data further revealed that the students using the full affective version were able to solve the easier exercises in a shorter time but they were unable to resolve the more difficult exercises covering more advanced concepts such as arrays and functions. One possible explaination for this is that these difficult exercises were scheduled towards the end of the 1 h session and students were encouraged to solve the exercises in sequence from the fundamental to the more advanced level. It is plausible the students ran out of time for the resolution of the more difficult exercises which were attempted only towards the end of the session.

The number of compilations was not significantly different for the affective and non-affective groups. It is possible that more sophisticated measures that examines the compilation errors and the compilation behaviour of the students in resolving the errors between compilation attempts may be required (Jadud, 2006). As this version of ATS did not capture the details of the compilation errors encountered by the students, this was not further investigated.

It is thus reasonable to support the hypothesis that the full affective version of the ATS results in more effective tutoring as compared to the version with the affective function disabled as the affective ATS does enhance the efficiency and persistence of the students in solving the programming exercises.

From the focus group discussions, the students were generally positive on their learning experience with the ATS. They were willing to use the ATS as a complement to face-to-face lessons conducted by their tutors and also as a form of e-learning. The students compared the proposed tutoring system with existing alternatives such as Code Academy and IDE and prefer the interactivity and tutoring aid provided by the proposed tutoring system. This also suggests that our affect based interventions in the form of empathetic messages and reminder prompts are rated positively by the students due to their interactivity effects. Thus, usability of the system is generally not an area of concern for the students.

The top rated functions that the students found most useful for their learning were the fill in the gap programming exercises and the hint function. The fill in the gap programming exercises lowers the cognitive load as students only have to fill in the missing lines of codes while the hint helps the students to resolve compilation errors that they are unable to solve on their own. However, the students felt that the explaination provided in the hint messages could be further enhanced to be more comprehensible and possibly shorter to entice them to read it (Anderson, Corbett, Koedinger, & Pelletier, 1995). The students further suggested that audio messages might help. Other than audio, animations and videos may also help in encouraging the students to spend more effort on comprehending the provided hints. Further research may be initiated in the future to investigate whether this would lengthen the reading time of the hints by the students and whether this would subsequently result in deep enhanced learning.

Some of the students expressed their discomfort with their actions being monitored with sensors such as web cameras even though they understood that the recorded logs are essential for the ATS to infer their emotions for the optimization of their learning. We are of the opinion that other than disclosing the purpose of recording the actions of the students within the ATS, more may have to be done to address the privacy concerns of some of these students. One possible solution will be to assure these students that the captured logs would not be used for purposes other than to enhance their learning. The data access and retention policy will need to be made known to the students as well so that they know who is using the data, the purpose it is used for and how the data will be stored and for how long. Another way to circumvent the privacy concerns of students would be to build in the option of allowing the students to opt out from being monitored by sensors for affect inference.

The students also raised concerns on the accuracy of affect sensing and the ramifications should the affect sensing be incorrect. To address this, we would suggest disclosing to the students that the tutoring response strategies formulated in response to the sensed affective states of the students as elaborated in the Tutoring subsystem section are mainly fail-safe. Thus, even if the students’ affect is wrongly diagnosed, there would not be dire consequences on their learning.

S. K. D’Mello, Lehman, and Graesser (2011) proposed the use of affective messages, feedback with accompanying affective expression and dialogue moves for tutorial interventions. They evaluated the efficacy of Affective AutoTutor with an experimental study to evaluate its efficacy in promoting learning and engagement. The affective version of MetaTutor (VanLehn et al., 2014) also used motivational messages that addresses specific affective states of students. The evaluation study that was conducted however, did not use the affect sensors and detectors to determine the optimal moment for displaying the motivational messages and this possibly constraints the effectiveness of the affective interventions. In comparison, our study proposes pedagogical driven intervention measures which are activated at the moment when the relevant affective states of frustration and boredom are exhibited by the students. In addition, both qualitative and quantitative evaluation instruments are used in our study for evaluating the effectiveness and acceptance of the ATS by the students.

Incorporating empathy into an intelligent tutoring system enhances its tutoring effectiveness. In this study, we detailed the design, implementation and evaluation of an ATS. From the literature, we surmise that ATSs are seldom implemented and in addition, for the implemented ATSs, most tutored in the domain of mathematics and science and not for the domain of computer programming.

To cater for the generalizability and subsequent extension of the ATS for use in other tutoring domains, the various components of the ATS are designed to be loosely coupled so that individual component can be extended or modified without affecting other components. This study also addresses the gap in the literature where appropriate tutoring responses must be formulated in response to the inferred affect of the students to optimize their learning.

Santos (2016) reviewed 26 systems that leverages affect in an educational context and concluded that not many of the systems provide affective interventions. In addition, the few works (Grawemeyer et al., 2015; Santos, Saneiro, Salmeron-Majadas, & Boticario, 2014) that evaluated the effects of affective interventions in an educational context adopted a Wizard of Oz approach in contrast to our study which integrated affect sensing and intervention and evaluated the effects of the affective interventions in a tutoring system. Santos (2016) further acknowledged that devising autonomously the appropriate tutoring response to specific students’ affective states (including when to intervene and what affective support to provide) is a challenging task and is an open research issue. This underlines the importance of our study in contributing to the body of knowledge for autonomous affect driven tutorial intervention.

The proposed ATS is evaluated for its effectiveness, usability and acceptance by the students using both the quantitative technique of experimental study and the qualitative technique of focus group discussions. The results for the experimental study support the hypothesis that the full affective version of the ATS results in more effective tutoring as compared to the version with the affective function disabled. The analysis of the transcripts for the focus group discussion further revealed that the students are generally positive on their learning experience with the ATS. The top rated functions from the students are the fill in the gap programming exercise and the hint function which offers explaination on top of the rectification steps.

This research also highlights privacy concerns with regards to the monitoring of students’ behaviors and actions via sensors in an ATS. One suggestion is for options to be provided for students in an ATS to opt in or out of the monitoring but opting out will also mean that the empathetic tutoring response will be disabled as well. This privacy and usability conflict is an issue that will have to be resolved for widespread deployment of ATSs in classrooms and would be a potential area for future research.