We adopted the DBR approach, which is designed for researchers and educators who intend to increase the effects of educational research and improve real-world classroom practices (Anderson & Shattuck, 2012). There are three main phases of DBR: (a) preparing the experiment and placing it in a theoretical context, (b) applying the interpretive framework and conducting the experiment to support learning, and (c) performing retrospective analyses (Reimann, 2011). DBR enables practitioners to focus on designing, testing, and refining a theory-based intervention by addressing practical educational problems in several iterations (Collins et al., 2004; Jan et al., 2010).
We carried out the three phases of DBR over two studies. First, we implemented the initial GAFCC gamification design model in semester one of the 2020–2021 academic year (Study 1, N = 26). Second, we refined the GAFCC model based on the challenges revealed in the data, addressing them by introducing a new element, fantasy. Our design used fantasy to amplify the effects of the original five elements (i.e., goal, access, feedback, challenge, and collaboration) and address decreased affective and behavioral engagement in students. Third, in semester two of 2020–2021, we tested the effectiveness of the new GAFCC-F gamification design model on students’ learning performance, interaction in online discussion forums, and perceptions by comparing these outcomes to those of the GAFCC design model (Study 2, N = 23) (see Sect. 3.2 for the design rationale of the GAFCC-F model). Studies 1 and 2 were conducted in the same online course, “E-learning Strategies and Management” (course length: one semester, or 10 weeks) offered in two successive semesters. These two studies were carried out during the COVID-19 pandemic; therefore, all the courses were delivered via a fully online synchronous teaching mode via Zoom, a web-based videoconferencing platform.
We aimed to compare the effects of the GAFCC and GAFCC-F gamification design models on students’ learning performance, interaction in online discussion forums, and perceptions. The two courses were taught by the same instructors using the same courseware, assessment rubrics, and final assignments. All the learning materials and tasks were distributed by the same learning management system, Moodle. The participants in the two studies shared an interest in technology-enhanced learning, as they were admitted to the same program. “E-learning Strategies and Management” is a postgraduate-level elective course with no prerequisites for students’ enrolment. The course introduced six specific learning outcomes (i.e., factual learning, conceptual learning, problem-solving, procedural learning, principle learning, and attitude learning) as well as the relevant instructional strategies to facilitate mastery of the six learning outcomes in the e-learning context.
Ethics approval for this study was obtained from the Institutional Review Board of the authors’ university. The participants were informed of the research objectives, and they all signed the consent forms before the intervention.
Study 1: Implementation of the GAFCC gamification design model
Participants
Twenty-six postgraduate students (18 females, eight males) participated in Study 1. Their ages ranged from 22 to 48 years (M = 28.5, SD = 6.19). Among them, 92.3% (24 out of 26) were from East Asia (i.e., Hong Kong and Mainland China) and two participants were from the United Kingdom.
Settings of the GAFCC gamified class
Study 1 was designed using the GAFCC gamification design model. We managed all the learning materials and activities on Moodle. A plugin, “Level Up” (Sinnott & Xia, 2020) was installed in Moodle to set up the gamified environment. “Level Up” logged students’ real-time activity results according to pre-specified rules in the system. According to these rules, students were automatically granted a certain number of points when they accomplished specific actions and reached certain performance measures. An individual leaderboard displayed individual students’ accumulated points and their respective ranking determined by those points, whereas a team leaderboard displayed the accumulated points earned by all members in one study group and the study group’s ranking. We announced the points-adding rules at the first session of the course (see Fig. 1).
The implementation of the course using the GAFCC model was conducted as follows. First, in the first session of the course, we assigned a goal for the students, which was to reach the 12th level by completing the various learning tasks in Moodle by the end of the semester. Second, we used the “unlock function” to activate the access element, meaning that the learning tasks on Moodle were not designed to be completed at one go. Instead, students were required to complete the available individual easy quizzes, create a post in the first discussion forum, and submit the first group assignment before moving on to subsequent tasks (i.e., individual difficult quizzes, second discussion forum, and second group assignment).
Third, we gave students’ instant feedback on their performance on quizzes, posts on discussion forums, and submissions of group projects by automating the points-adding process for each task (see Fig. 1 for points values). Points were given to recognize students’ efforts in completing the learning tasks. Two types of incentive schemes were used. The first was a performance-contingent point system, wherein points would only be given once a student had reached a passing grade on a given quiz. The second was a completion-contingent point system, in which points would be given when a student had completed a task (e.g., posts, group assignments).
Fourth, to introduce collaboration, we assigned four group assignments during the semester. Each group assignment required the members to propose relevant instructional strategies to teach a certain topic. For example, students were asked to work in groups and propose appropriate instructional strategies to teach the concept of fruits. Fifth, we assigned several sets of difficult quizzes and difficult group assignments, so students could challenge themselves by attempting these tasks after they had accomplished the easier ones.
Feedback on the GAFCC gamified learning experience
We collected the 24 students’ reflections on gamified learning at the end of the semester. Most students (97%) preferred gamified learning to their previous, conventional, lecture-based courses. However, many students (67%) stated that although they were very engaged in this gamified learning experience in the first two sessions, they gradually lost interest when they became familiar with the gamification scheme.
Additionally, we examined changes in students’ participation, such as posting on forums and attempting quizzes, over the 10 weeks. The results showed that the number of posts and attempts decreased sharply after Week 5 (halfway through the intervention). The number of posts in the first two forums, released before Week 5, were 116 and 183; however, the number of posts in the last two forums, released after Week 5, were 34 and 22. The average number of attempts on the first two sets of easy and difficult quizzes was 68.5 and 51.5, respectively. The average number of attempts on the last two sets of easy and difficult quizzes was 39.5 and 31, respectively (see Fig. 2). Some of the students (45%) who were frequent video game players stated that this gamification would have provided a more exciting learning experience if it included a virtual play environment, for example, virtual characters and a back narrative for each gate/level, thus simulating the thrilling experience of playing a video game.
In addition, instructors noted that the students became less interested in completing new tasks after the novelty period of the first three weeks. One instructor suggested setting up a narrative to link all the tasks, permitting students to follow this narrative to unlock tasks and thus making passing each level more meaningful. It was proposed that the students may obtain a sense of achievement in completing all the tasks and accomplishing the final mission at the end of the semester.
Study 2: Implementation of the GAFCC-F gamification design model
The main problems we encountered in using the GAFCC model were students’ decreased affective engagement (i.e., students became less interested in this gamified learning) and behavioral engagement (i.e., students displayed less interaction with peers on discussion forums and fewer attempts on quizzes) over time. In response, we introduced a new element, fantasy, originating from the literature on computer games, into the GAFCC model. We created a fantasy world by combining certain elements from reality (e.g., tourist attractions in City K, COVID-19 background) and certain imaginative elements (e.g., a talking dragon in the “Save Princess Joanne” story, see details in Sect. 3.4.2).
Participants
Study 2 involved 23 postgraduate students (15 females and 8 males). The ages of the participants ranged from 22 to 47 years (M = 26.35, SD = 5.93) (Fig. 3).
Settings of the GAFCC-F gamified class
This section describes how we implemented the fantasy element in the GAFCC-F model. Figure 4 lists the points-adding rules and game elements of this model. We introduced the game rules and the backstory of “Save Princess Joanne” in the first session of the course. The goal of this story (i.e., to rescue Princess Joanne from the bad dragon) was the core concept of this round of gamification design. We also set up a lead-in backstory to explain why the game characters (i.e., the course students) were required to travel to City K (an authentic city) to save the princess. We created an elderly character, who was embodied in the instructors, to announce the points-adding rules and main tasks in the narrative. Figure 5 presents images of the “Save Princess Joanne” narrative-based fantasy in the imaginary pixel world.
The students’ final mission was to collect points and unlock 12 levels to bring Princess Joanne safely back at the end of the semester. Each level required a certain number of points, and each level corresponded to a part of the game map (see Fig. 3). First, students were required to choose a character out of four pre-set characters to activate the adventure. Second, to meet students’ need for autonomy, we offered two routes (i.e., the City or Island Route) to choose from. The plot of each level could only be unlocked when students had satisfied the previous task. Among the 12 levels, we assigned two very challenging questions for those who were interested in gaining a large number of points in one go. We also set up a rewards section, where students could collect 400 points by completing all the tasks of one route. The remaining rules of play were identical to those of Study 1 (see Fig. 4 for details).
Measures in study 1 and 2
We measured the students’ learning performance, interaction with peers in online discussion forums, and perceptions by using the same instruments in both studies.
Students’ learning performance
To measure the students’ learning performance, a pre-intervention test, a mid-term test, and a post-intervention test were administered. The pre-intervention test was a voluntary performance test, students could choose not to take it. The pre-intervention test contained seven short essay questions examining students’ prior ability to name an appropriate instructional strategy to obtain a given learning outcome. This test was conducted in the first session of the course. The following are sample questions from the test: What are some important elements of a good lesson objective? What are the instructional strategies to teach “concepts”? What are the instructional strategies to teach “facts”?
We measured the students’ learning performance with a mid-term exam (i.e., at Week 5). The mid-term test was also a voluntary performance test. This test examined both the students’ factual knowledge (i.e., explanations of concepts and differentiation of several e-learning design models) and problem-solving ability (i.e., application of appropriate instructional strategies to address a real-world instructional scenario).
The following are sample questions testing factual knowledge: What are the differences between learning outcomes and learning objectives? What are the differences between the ADDIE and 5E models? A problem-solving, scenario-based activity required students to design a fully online course. First, students were required to identify specific learning outcomes and write corresponding learning objectives for a given scenario. Second, they were required to list key instructional strategies to achieve the learning objectives. Finally, they were asked to list all the content and technologies needed to complete the course design under the guidance of the 5E model (i.e., engage, explore, explain, elaborate, and evaluate). The scenario description was as follows: You are working in an instructional design company, and you have been assigned to design a course. The client asked you to design a fully online, five-day, asynchronous course to train 20 newly recruited, inexperienced insurance agents. The course will help the new employees master information about three insurance packages and two sale techniques.
The post-intervention test referred to the following three final assignments: a) design a storyboard to train customer-facing representatives to sell one beauty product, b) analyze one e-learning hack, and c) design and build content for one online Moodle course.
Students’ interaction with peers in online discussion forums
We downloaded the logs of students’ posts in the four online discussion forums generated in the two studies. The first online discussion forum (DF 1) was about the students’ writing their own learning objectives. The students were asked to come up with two scenarios in which their written learning objectives could be applied. The second online discussion forum (DF 2) invited students to share their opinions on how the Fourth Industrial Revolution could influence the education sector. They were asked to select at least two of eight essential technologies listed on the designated website and discuss how they could integrate these technologies into teaching and learning. The third discussion forum (DF 3) was about how the students, as instruction designers, would address some of the issues brought about by COVID-19. The students were required to share a link to an article about how schools, universities, or companies are tackling the issue of teaching and learning or corporate training under these circumstances. From the article they posted, they were asked to identify the key strategies being implemented in these institutions. The fourth discussion forum (DF 4) invited students to share one e-learning application and suggest a scenario in which it could be used. The students’ performance on discussion forums did not contribute to their final course grades, as participation on the forums was not compulsory.
We examined the quality of students’ online posts (both their own posts and their comments on peers’ posts), using the two rating scales for assessing collaborative online notes proposed by van Aalst (2009), knowledge quality and significance of findings. Knowledge quality is an assessment of the epistemic position of the knowledge, scored as simple conjecture (with a score of 1), a factual claim (2), a partly integrated explanation invoking at least one concept (3), or a comprehensive explanation invoking multiple concepts (4). Significance of finding is an assessment of how well students identify the knowledge that they have learnt. Comments were scored as a restatement of the knowledge (with a score of 1), a clear description of the knowledge without limitations (2), a profound description of the knowledge with some limitations (3), or a comprehensive description of the knowledge with limitations and inquiry to others (4) (see van Aalst, 2009 for details).
We further used directed content analysis to code students’ post content, assessing how they interacted with their peers in a fully online social community. Directed content analysis provides a structured process in coding (Hickey & Kipping, 1996), permitting us to use pre-determined codes from prior research to start the coding immediately (Hsieh & Shannon, 2005). A new code was generated if the text did not fit in the initial coding scheme (Hsieh & Shannon, 2005). We adopted van Aalst’s (2009) sub-themes codes for the “community” aspect as the guide for our content analysis, as our study had a similar data source (i.e., students’ posts on online discussion forums) and the same main theme of “community.” There were seven initial sub-codes, namely apologizing, co-authoring, innovating, giving credit, deciding, encouraging, and seeking views. We created new codes or removed codes as needed to fit the content of the texts. Two raters independently marked the two rating scales and coded the content of posts. All the discrepancies were discussed until full agreement was reached.
To understand students’ online interactive patterns, a social network analysis approach was applied to analyze the students’ posts in all four online discussion forums. We used UCINET as the analytical tool (Borgatti et al., 2002). The five indicators we examined were network size, density, degree centralization, out-degree, and in-degree centrality. “Network size” refers to the number of actors in the network (Collins & Clark, 2003). “Network density” refers to the prevalence of direct ties in a network graph (Frey, 2018). This indicator helps to analyze the connectedness within a network (Scott, 2012). Centralization indicates how centered a network is around the most active nodes (Claros et al., 2016). A high value of centralization suggests that interaction is concentrated on a few actors, in other words that not many people contribute to an online discussion (Claros et al., 2016). In contrast, a low value of centralization indicates that many students contributed equally to the online discussion (Huang et al., 2019). Out-degree and in-degree centrality were used to identify how active the students were in posting and commenting. The out-degree value indicates the sum of connections from a given node toward other nodes, and the in-degree value indicates the sum of connections from other nodes directed toward a node (Hansen et al., 2011). For example, in an online discussion forum, “out-degree” refers to the number of posts that a student sent out in the community and “in-degree” refers to the number of replies that a student received in the community.
Students’ and teachers’ perceptions
To understand the students’ and teachers’ overall perceptions of the GAFCC-F model, we conducted two rounds of semi-interviews at the end of the semester, interviewing 15 students and two teachers. Sample questions are as follows: What do you think of “Save Princess Joanne in City K” in terms of motivating or demotivating you in completing individual work? What do you think of “Save Princess Joanne in City K” in terms of motivating or demotivating you in collaborating with group members? and do you have any suggestion for improving this learning experience? When interviewing the teaching team, we asked their findings based on class observation, their perceptions of the fantasy element, and their suggestions for future course design.
We used a thematic analysis approach to process the students’ responses. Several relevant themes developed via initialization, construction, rectification, and finalization (Vaismoradi et al., 2016). Two authors first independently coded the interview transcripts, then discussed the results until a mutual agreement was reached.