As we mentioned, the methodology and tools discussed in this paper have been developed throughout the years within programming courses from the bachelor degree in Computer Science at University. More specifically, the adoption of animation programs has been systematically adopted in two courses.
-
i.
The Procedural Programming course (PP), i.e., Coding 101, the introductory course to programming (1st year of the Computer Science curriculum). The PP course deals with introductory procedural-programming techniques in several languages, primarily C/C++, and secondarily MatLab, covering the basics, control structures, functions and modular programming, arrays and records, pointers, and an introduction to data structures, with a focus on the list data structure. The final exam is composed of two parts: a written test with 30 multiple-choice questions to be completed in 40 min; a practical programming test requiring to implement a simple application with a console-based user interface that implements one or more algorithms on a collection—i.e., a list—of records. This is a 1st-year course, and we assume that most of the students do not have any programming background.
-
ii.
The Object-Oriented Programming course (OOP), i.e., the introductory course to object-orientation (2nd year of the curriculum). The OOP course introduces classes, objects, references, methods and messages, application layers, exceptions and regression tests in several languages, primarily Java, and secondarily C#. The final exam is similar in structure to the one of the PP course: a written test with 30 multiple-choice questions to be completed in 40 min; a practical programming test requiring to implement a medium-complexity application logic and the relative regression tests. Students need to choose the most appropriate data structure, i.e., list, set or map.
In this section we report some experimental results gathered in this framework. We discuss two experiments: (i) first, an analysis of course-completion stats, in order to assess the actual benefits of our approach in terms of learning outcomes; and (ii) second, a user study conducted among a group of students to gain insights about their perception of the strengths and limitations of the method.
Effectiveness on course-completion rates
As a first set of experimental results, we report course-completion rates for different groups of students. We compared our test group, students that have been taught coding in PP and OOP with the help of the Diogene-CT method and tools, to a control group of students that didn’t. In order to do this, we measured the percentage of students that passed the final exam for the PP and OOP course from academic year 2013–2014 to 2018–2019. While instructors and course contents remained largely unchanged, the teaching methodology has evolved significantly through these years:
-
In 2013–2014 and 2014–2015 no Diogene-CT techniques were employed within the course yet. We shall make reference to these cohorts as NO.
-
The cohort of 2015–2016 was a transitory one, in the sense that the robotic-arm metaphor was adopted to support the introduction of programming concepts, but the Diogene-CT toolset was not production-ready at the time, and therefore more rudimental tools were used. We shall make reference to these cohorts as P, for “partial”.
-
Finally, the methodology and the toolset were fully adopted in 2016–2017 and subsequent years. We shall make reference to these cohorts as A, since all of the toolset were used.
Course-completion ratios are shown in Fig. 8 with the column PP and OOP. More specifically, we report the percentage of students enrolled in the course that successfully passed the final exam at the end of the academic year. Since these data were gathered within real everyday academic experience, we couldn’t organize a canonical control group. Our intuition, however, is to use the results of the NO cohorts as controls for the A (and P) cohorts.
This poses several challenges, as it is known that knowledge, attitude and maturity of students may significantly vary throughout the years, and this unavoidably affect performance. In fact, our empirical and rather coarse observation was that the overall quality of students has somehow decreased during these years. To make this intuition more precise, we selected two other courses from our Computer Science curriculum that also had a stable set of contents and team of instructors from 2013 to date, namely Physics (PHY), 1st year, and Foundations of Electrical Engineering (FEE), 2nd year. Both these courses adopt a rather traditional teaching methodology, mostly based on theory-based frontal lessons with some additional exercises, and written tests.
To elaborate on the results in Fig. 8, in addition to PP and OOP, we also report the percentage of students that passed the final test for PHY and FEE. We also report trending lines for each data series.
It is easy to see from the chart that our intuition is confirmed, i.e., both in the case of the PHY course and of the FEE course the percentage of enrolled students that passed the final exam has almost consistently decreased in the years (see the PHY and FEE trending lines). The 2013–2014 represents and exception with respect to this trend, in the sense that the overall performance of the students was particularly poor.
Interestingly, this negative trend is not observable in both PP and OOP courses (PP and OOP trending lines are quite horizontal), again with the exception of 2013–2014. There were small decreases, but overall the percentage of students that passed the exams remained fairly stable, as confirmed by the trending line.
In a framework of generally decreasing success rates, as shown by the PHY and FEE results, we believe this is a clear indication of the fact that the introduction of innovative tools as the ones described in this paper positively impacted learning outcomes. This confirms that the Diogene-CT method and toolset can be an effective support in teaching coding.
Effectiveness on learning outcomes
In this section we report a second set of experimental data about the effectiveness of the Diogene-CT platform with respect to learning outcomes. In order to do this, it is important to note that the tool has been used in two courses of the undergraduate computer-science curriculum at University—PP and OOP—for which teaching over the last decade has also based on other experimental methodologies.
More specifically, we adopt a teaching methodology that is loosely inspired on SOLO—Structure of Observed Learning Outcome (Biggs and Collis 2014). The most relevant aspect of the teaching method with respect to this paper is the organization of the final exam.
Exams are based on a written, multiple-choice test, and on a practical, coding test. During the practical tests students are required to implement a fragment of application logic on a personal computer in our teaching lab. They use the software-development kit used throughout the course—e.g., gcc suite for C/C++ programs, or the JDK for Java programs—and need to deliver a piece of software that is both syntactically and logically correct.
The final exam offers a choice between two different practical tests:
-
the first, called the base test, is based on a simpler specification, and more standard algorithms and data structure; here the focus is on verifying that the student is able to write logically-correct code with a correct methodological approach in order to implement a simple functional specification; loosely speaking, this test allows students to get a C or D grade.
-
the second, called the advanced test, has a more complex specification, less standard algorithms and data structure, it may require to code in more than one programming language, and has a strong emphasis on test-driven development and regression testing; the goal is to verify if the student is able to handle a more complex specification with advanced coding techniques; this test allows students to get an A or B grade.
The organization of the final exams is presented to students very early during the course (typically during the first lesson), and students are completely free to choose the test to solve during the exam. Grades vary based on their choice, as discussed above. More important, it is made very clear, during the lessons, whether arguments are relevant to one level or the other.
Figure 9 shows the percentage of students that chose the advanced test and passed it with respect to the total number of students taking the test in the PP course for each academic year. Notice that grades for the advanced test reflect the increased complexity of the exam and are therefore consistently higher than those for the base test.
It can be clearly seen that introducing the metaphor has brought to a significant increase in the number of advanced tests, and therefore an advancement in grades and learning outcomes. We believe that, on the one side the adoption of the Diogene-CT methodology and tools made more clear the concepts introduced during the lessons, thus generally improving learning, and on the other side they fostered a deeper involvement of students in course activities, with a strong impact on learning outcomes.
To prove our insights, we also conducted a t-test (Welch t-test) on the group where our approach was not adopted (2013–2014 and 2014–2015 years) and the one where our approach was completely adopted (2016–2017 and 2017–2018 years). For each group, we counted the number of chosen base tests and advanced tests. The null hypothesis we want to reject is that using our approach does not change the number of chosen advanced tests. Using a 0.05 confidence interval we obtained a two-tailed P value equals 0.0226. By conventional criteria, this difference is considered to be statistically significant, and we can reject the null hypothesis.
User study
To further validate our approach, we conducted a user study among students. We involved two groups of students:
-
Group 1 was composed of 3rd-year students that had attended both courses, i.e., PP in 2016–2017, and OOP in 2017/2018.
-
Group 2 was composed of 1st-year students attending PP in 2018–2019.
We conducted a survey with each group. The goal of the survey was to gather feedback from the students about the ACME methodology. We made two different questionnaires: the former on the use and effectiveness of Diogene-CT and ACME in PP; the latter on OOP. Each questionnaire contained five questions, the first four of which were multiple-choice (with options Very Poor; Poor; Good; Very good) and the last one open-answer.
The PP questions were as follows:
-
Q1. Do you think that the use of metaphors and animations facilitates learning the basic concepts of procedural programming (variables, assignments, inputs and outputs)?
-
Q2. Do you think that the use of metaphors and animations facilitates learning control structures?
-
Q3. Do you think that the use of metaphors and animations facilitates learning the concepts of modular programming (use of sub-programs and passing parameters)?
-
Q4. Do you think that the Diogene-CT metaphor helps learn several programming languages, since it is not linked to a specific programming language?
-
Q5. What aspect would you like to improve on the mechanical arm metaphor and its animations?
The OOP questions were as follows:
-
Q1. Do you think that the use of metaphors and animations facilitates learning the basic concepts of object-oriented programming (components, classes, objects, constructors, packages)?
-
Q2. Do you think that the use of metaphors and animations facilitates learning the notions of reference and message?
-
Q3. Do you think that the use of metaphors and animations facilitates learning concepts related to inheritance and polymorphism?
-
Q4. Do you think that the Diogene-CT metaphor helps learn several programming languages, since it is not linked to a specific programming language?
-
Q5. What aspect would you like to improve on the robot-community metaphor and its animations?
Twenty-four questionnaires were collected Group 1 about PP (Fig. 10). All of the students who participated in the survey had successfully completed their attendance of the PP course and passed the final exam. As it can be seen in Fig. 10, feedback were largely positive. More specifically:
-
Answers to questions Q1 (impact on learning basic concepts) and Q3 (impact on learning modular programming and functions) were largely positive. This is an important result since modular programming and the use of functions are typically perceived by students as a difficult topic, yet a fundamental one.
-
Also, all answers to question Q5 (language independence) were positive, i.e., all of the participants acknowledged that language-independence is a crucial feature of the approach.
-
The question that gave results more mixed was Q2 (impact on control structures). Here, while the overall judgment was largely positive, students perceived a smaller benefit associated with the methodology. This is understandable since the only scene element directly related to control structures in the procedural-programming metaphor is the lollipop.
-
Answers to Q5, the open-answer question show that students consider very important the availability of the attached mode to play with code and exercise at home.
Twenty-five questionnaires were collected from Group 2 about OOP (Fig. 11). Here all answers were largely positive. It is interesting to note that students showed great appreciation for the impact of Diogene-CT and ACME while learning inheritance and polymorphism, two difficult OOP topics. Also, in this case, open answers to Q5 show that the students consider the availability of the attached mode to be very important. Some of the students also suggested to evolve the metaphor to further ease the learning of these subjects: (a) relationship between a class and its objects; (b) packages and constructors; and (c) the Model-View-Controller (MVC) pattern.
Finally, 46 questionnaires were collected from Group 2 about PP (Fig. 12). In this case, answers were less enthusiastic than those expressed by Group 1, although they were still largely positive (no “Very Poor” options are chosen, and only an overall 14.28% of “Poor” options). This is most likely because these students were still attending the PP course at the time of the survey, i.e., they were still struggling with learning PP concepts. Open answers to Q5 were mainly concerned with requests for improvements and extensions, i.e.: (a) improve string readability; (b) smoothen animations; and (c) speed-up transitions, especially for the most obvious animations. Some of the students commented that ACME can be considered a useful tool for newbies, but it is less effective with more experienced students. Also, in this case, answers confirmed the importance of the attached mode to practice in autonomy.
Based on these ideas, we believe that the results of the survey confirm the effectiveness of the approach when tested on the field. We will consider all comments and suggestions to refine and improve the method and the tools.
Summary of experiments
Diogene-CT can be seen as an holistic approach to the problem of supporting instructors and students in the task of teaching and learning coding. At the roots of this approach are three main components:
-
a set of consistent metaphors, that serve the purpose of simplifying the approach to computer-programming concepts;
-
a collection of executable tools, that make the attempt of introducing the metaphors in actual coding curricula more concrete;
-
a pedagogical approach centered around the metaphor and toolset.
We believe that our experiments show that the combination of these features has brought clear advantages to our computer-science students. Results in section “Effectiveness on course-completion rates” show that course-completion rates have kept stable, in a general context of student cohorts of decreasing quality. Even more important, data in section “Effectiveness on learning outcomes” show how participation in courses was more qualified, and this was reflected by results in final tests. Finally, our user study show that students perceive this approach as a valuable addition to teaching.