Using a gamified mobile app to increase student engagement, retention and academic achievement

Abstract

This study investigated whether the use of a gamified mobile learning app influenced students’ academic performance and boosted their engagement in the subject. Created to better engage students in lecture content, the app was used to deliver multiple-choice content-based quizzes directly to students’ personal mobile devices post-lecture and pre-tutorial. After measuring the relationships between students’ app usage and their engagement, retention and academic achievement in the subject, it is suggested that following the app’s introduction, student retention rates and academic performance increased, and there was a positive correlation between students’ scoring highly on the app and achieving higher academic grades. While the app’s affordances for learning are promising, the causal relationship between the app usage and improved student outcomes requires further investigation. Conclusions made in the context of the wider scholarship of mobile app enhanced learning and applied game principles in HE.

Introduction

Mobile applications (apps) used as learning and teaching tools are not uncommon in Higher Education (HE) (Pechenkina, 2017). However, what makes an educational mobile app effective is a subject of ongoing interest to academics, lecturers, learning designers and other stakeholders invested in education (Hirsh-Pasek et al., 2015). Recent studies report on various HE mobile app offerings which despite the differences in their design and intended uses are united by a common goal of enhancing learning to improve student outcomes. For instance, HE mobile apps can be used to facilitate learners’ knowledge acquisition and transfer (Hannon, 2017), simulate experiential situations to train medical students (Amer, Mur, Amer, & Ilyas, 2017), assess in-class activities to provide immediate feedback (Deb, Fuad, & Kanan, 2017), and engage students in situated learning using augmented reality affordances (Bower, Howe, McCredie, Robinson, & Grover, 2014). Apps with gamified elements integrated into their design, in addition to facilitating learning, can further engage and motivate students (Hamari, Koivisto, & Sarsa, 2014) – where engagement is of particular importance because it may factor into student retention.

Many gaps still remain in the HE mobile app research, with further studies needed to better inform impactful app designs. Contributing to this goal, this article presents the findings of a study which investigated whether the use of a gamified mobile learning app positively affected students’ academic performance and engagement in the subject. The app in question was created specifically to better engage students in lecture content by delivering multiple-choice quizzes directly to students’ personal mobile devices post-lecture and pre-tutorial. The app’s customisation capability meant it could be tailored to suit any discipline-specific content, curriculum or a mode of teaching. Digital leaderboards and badges were embedded into the app’s design to keep students motivated to use the app, while push notifications alerted students each time a new quiz became available, allowing them to do the quiz ‘on the go’ regardless of their physical location. Grounded in the pedagogy of spaced education (Kelley & Whatson, 2013), the app’s primary purpose was to systemically test students on what they learnt in the low-stakes environment, hence consolidating their knowledge of the content. After its initial trial with a first-year accounting unit, the app was then also introduced into several first-year science units. Considering small student numbers in each unit using the app, all cohorts were combined into one sample to ensure meaningful statistical results. Findings are contextualised in the wider scholarship of mobile app enhanced learning and applied game principles in HE.

Mobile app technologies in higher education: An overview

Compared to the fast-growing body of scholarship concerned with educational mobile apps for children (Hirsh-Pasek et al., 2015; Hswen, Murti, Vormawor, Bhattacharjee, & Naslund, 2013; Judge, Floyd, & Jeffs, 2015), more evidence-based research into the impact of HE mobile learning apps is urgently needed. Where a mobile learning app’s effectiveness is concerned, some factors remain constant regardless of a student’s age: such as, learning is most effective when learners are engaged, cognitively active and guided by a goal, and when learning activities are scaffolded and interactive (Hirsh-Pasek et al., 2015). For older learners, however, a set of additional factors may influence their learning patterns and behaviours, such as their motivations, expectations and prior experiences (Salmon, Pechenkina, Chase, & Ross, 2016). All these factors need to be taken into account when designing mobile apps to enhance HE learner experience.

A number of mechanisms were found effective in the task of keeping learners engaged. Question-prompting and automated immediate feedback combined with explanatory strategies (Byun, Lee, & Cerreto, 2014; Sung, Chang, & Liu, 2016) were among them. The ‘push notification’ technology was also deemed useful when tasked with encouraging immediate learning and helping learners stay up-to-date with content (Garbrick & Clariana, 2015; Kudo et al., 2015). A mobile learning intervention’s length is another significant factor attributed to the intervention’s effectiveness, with longer interventions found more effective as they allowed to fully integrate students into the learning process, with the desired effects achieved over time (Sung et al., 2016). While learners’ age did not appear to factor into a mobile app intervention’s success, groups of learners homogenous in age showed higher rates of impact compared to the mixed-age cohorts (Sung et al., 2016). At the same time, mobile learning apps with gamified elements did not appear to achieve significant positive effects on learning (Sung et al., 2016). However, other studies found that when mindfully integrated into an intervention’s design, the game elements worked in accord with other aspects of the intervention to positively impact on student learning (Laine, Nygren, Dirin, & Suk, 2016; Woo, 2014).

Several educational psychology principles attributed to effective learning must also be considered when designing mobile learning app interventions. Based on Ebbinghaus’s (2013) theorising of ‘the forgetting curve’ as a way to explain the finer workings of human cognition and memory, the spacing effect and the testing effect emerged as two pedagogical principles significant when designing mobile apps for learning. With some positive results recoded (Kerfoot, Turchin, Breydo, Gagnon, & Conlin, 2014; Shenoi et al., 2016), when operationalised within an app’s design, these principles could allow students to revisit and consolidate what was learnt by being systematically tested on their knowledge.

The spacing and testing effects and the pedagogy of spaced education (Kelley & Whatson, 2013) behind them are well aligned with a knowledge organisation principle known as ‘chunking’. In the HE context, ‘chunking’ is best understood as a cognitive strategy used to enhance mental performance, where a bulk of information is (re)organised into smaller segments (‘chunks’) for improved comprehension (Afflerbach, Pearson, & Paris, 2008; Cowan, 2014). Enabled by smartphones and similar devices, mobile app technology is well suited to facilitate student engagement with such ‘chunk-sized’ knowledge, leading to better comprehension of lecture material (Lah, Saat, & Hassan, 2014).

Considering student demand for personalised learning options is growing (Shah, Sid Nair, & Bennett, 2013; Wanner & Palmer, 2015) while personal mobile devices become ubiquitous (Mackay, 2014), it is timely to take advantage of mobile app technologies to create new ways for students to personalise their educational experiences. These considerations are particularly salient where the experiences of first-year HE students are concerned as these students are at higher risk of dropping-out compared to their senior peers (O'Keeffe, 2013; Ryan, 2004).

Applying game principles in mobile app design

Gamification in education can be broadly understood as the use of game elements in non-game contexts (Deterding, 2011; Domínguez et al., 2013), purposed with increasing student engagement and motivation. With various studies reporting significant correlations between introducing the gamified elements into the learning process and increased student motivations (Domínguez et al., 2013), some gamification scholars (Hamari et al., 2014; Koivisto & Hamari, 2014) point out that success of any gamification initiative is greatly dependent on the context of its implementation and the attitudes of its intended users. Other recent research found that students generally hold positive perceptions of gamified learning and appreciate social interaction, engagement and immediate feedback it affords (Cheong, Filippou, & Cheong, 2014). Taking all these success factors into account, when strategically embedded into an online learning initiative, game elements have a potential to improve student outcomes (Hirsh-Pasek et al., 2015; Jere-Folotiya et al., 2014; Ke, 2015; Laine et al., 2016; Werbach & Hunter, 2015).

Learner motivation remains a primary concern of game-based mobile learning initiatives, with developers and educators alike trialling different approaches of introducing game elements into the learning process. These include ‘progression trees’, score-generated leaderboards, and digital badges (Abramovich, Schunn, & Higashi, 2013; Ahn, Pellicone, & Butler, 2014; Lokuge Dona, Gregory, & Pechenkina, 2016). Further, considering that low-stakes assessments offered early on in the teaching period build confidence and engagement, and, in turn, have a beneficial effect on retention (Meer & Chapman, 2014), delivered in a gamified format such assessments can make a mobile learning app more effective in its task (Weitze & Söbke, 2016).

Drawing on various examples of impactful mobile learning gamification initiatives, the app at the centre of this study was fitted with such features as push notifications, leaderboards and digital badges, while the pedagogy of spaced education operationalised through multiple-choice quizzes strategically scheduled post-lectures and pre-tutorials was used to boost student engagement.

The app’s development process

After noticing her first-year accounting students growing disengaged whenever a difficult concept was introduced, the unit’s head lecturer turned to mobile technology to improve the situation. Student disengagement is not uncommon in the discipline of accounting (Byrne, Flood, & Griffin, 2014), but the unit in question also ‘suffered’ from being a mandatory breadth (or foundation) unit as it tended to attract students who did not specialise in accounting and who were generally direct-school leavers entering university for the first time. As a result, the unit routinely showed low test performance and high attrition. The lecturer experimented with using such online tools as Kahoot! to boost in-class engagement, but the issue of keeping students engaged in learning after the lecture was over remained an ongoing concern. Because of the proliferation of personal mobile devices among the students, it was decided that a mobile app was the most efficient way to reach out to them and keep them interested in the unit.

Before embarking on an app development process, a number of existing commercial HE learning apps were analysed, but deemed as either too costly and/or failing to combine all of the features and tools identified as desirable during the project’s scoping phase. For instance, it was deemed crucial that the lecturer would have full control over the app’s weekly content and curate quiz questions and their timed release. This would make the app fully customisable and allow its use across various disciplines and classrooms. Therefore, after securing an internal seed grant from the university’s Teaching and Learning Unit, the app was developed ‘in-house’ for both Android and Apple devices. All students in the accounting unit were invited to download the app. Those who elected to use the app would receive regular post-lecture and pre-tutorial push notifications delivered directly to their mobile devices, inviting them to test their knowledge of the concepts introduced in-class by taking a multiple-choice quiz. Based on the students’ app engagement, various data was collected through the app’s analytics centre, including the speed of student responses to quiz prompts and the number of attempts it took a student to get the answer right. Based on these statistics, a leader scoreboard was populated and a winner announced each week. The app’s usage statistics were also ‘fed’ into the university’s learning management system where students could view their current leaderboard placement and digital badges awarded based on quiz performance. Students could also opt out from the leaderboard component and use the app solely for the purpose of revision.

After its initial trial with the accounting unit, the app was also introduced into several first-year science units. While the app’s structure and features remained the same, the quiz questions were tailored to suit the content of each participating unit. Considering small student numbers in each unit using the app, all cohorts were combined into one sample to ensure meaningful statistical results. After the app’s Semester 2, 2015 implementation, its impact on student learning was evaluated. This research was driven by the following questions:

• Did the app’s introduction correlate with improved student retention rates?

• Did the app’s student usage correlate with improved academic performance?

• Did students’ app performance correlate with their academic performance in the unit?

Methods

Rationale

Due to time constraints and limited resources necessary to conduct an in-depth qualitative or mixed methods study, it was decided that as a first step, there would be a quantitative statistically-driven study into the app. The decision was reinforced by the immediate availability of analytical data generated by the students’ app use as well as the relative ease of accessing aggregate student data on retention and academic performance. Finally, the quantitative research undertaken is envisaged as a foundation for the study’s next phases, which would include qualitative investigations.

Sampling

Student participants in this mobile learning app trial were recruited from the wider cohort of students enrolled in the first-year accounting and sciences units in Semester 2, 2015. Of 462 students enrolled in the accounting unit, 265 (57%) opted to use the app; and out of 249 science students, 129 (52%) opted in to use the app. Average grades and retention rates for the pre-app cohorts (enrolled in Semesters 1, 2015 and in both Semesters in 2014 and 2013) were obtained via institutional learning management system. After initially conducting a comparative analysis between the accounting and science cohorts, it was decided to combine all cohorts into one sample to achieve a higher total number and ensure statistically meaningful outcomes. As a result, of the combined sample of 711 students, 394 (55%) signed up to use the app and consented to have their engagement data analytics collected for research purposes.

The study has received clearance from the university’s ethics committee. In regards to student consent mechanism, in the first instance of accessing the app, each new student user was asked to create an account and then proceed directly to the app under the condition that their data would be collected in a de-identified aggregate manner for the purposes of the app improvement and research. Students, however, had a chance to click the ‘opt out’ button if they did not wish to have their analytics data used for research purposes. If the latter option was chosen, student usage analytics would still be recorded but in a separate database, hence would not be used for research purposes. Furthermore, students were informed via email about the research projects, its goals and parameters, and advised to contact the first author (who had no teaching or assessment duties in any of the units involved in this research) with any questions and concerns. None of the students who decided to use the app opted out of the research and no queries or complaints were received.

Both participating lecturers were guided in the process of creating multiple-choice quizzes in alignment with their lecture topics. Commencing on the first day of the semester, one quiz question was set to release daily and expire at midnight of the same day. Once a student commenced the quiz, a timer was set allowing between 20 and 60 s to answer a question. For each correct answer a student was assigned points, which, when accumulated would convert into digital badges based on the following achievement levels: for giving five consequent correct answers, for answering a question in under 15 s, for earning more than 100 points and for reaching the top of the leaderboard.

Statistical analysis was conducted using SPSS with the following data:

• Final student grades for pre and post app cohorts, where the goal was to measure the relationships between the average final grades (percentages) and the app engagement and uptake data; and

• The app usage analytics (such as scores achieved when using the app).

Results

The app’s introduction and student retention rates

In Semester 2, 2015 when the mobile app was first introduced in the trial units, there was an improvement in student retention (calculated at 12.23%), when compared with the pre-app cohorts of Semester 1, 2015. The same pattern of improvement (though not as strong) was also found when compared with Semester 2, 2014 (improvement of 9.22% recorded) and Semester 2, 2013 (improvement of 5.37% recorded) (Fig. 1).

Fig. 1

Student retention rates for accounting and science cohort, before and after the mobile app was introduced in Semester 2, 2015 (N = 6939)

The app’s usage and academic performance

Students who used the app demonstrated an average grade/percentage mark of 65.19% compared to those who did not use the app, the latter averaging a grade/percentage mark of 58.16%. Therefore, the app users on average achieved marks 7.03% higher compared to students who chose not using the app.

Students’ performance within the app and their academic performance in the unit

A significant positive correlation of .40 was found between performing well on the app tasks and achieving higher academic grades (as per Fig. 2).

Fig. 2

Correlation between app score and percentage mark (academic performance) (N = 6939)

Limitations

This study’s focus was on correlations between students’ app usage and their academic performance in the unit, however causal relations between these variables would need to be gauged further through future in-depth studies of student experience. A potential limitation of this study is attributed to the novelty effect which must be taken into account when evaluating educational technology initiatives as it can skew the results (Clark, 2015). A common phenomenon that students can experience during a course of study, particularly so when new technologies such as gamification are introduced (Koivisto & Hamari, 2014), the novelty effect has implications for extended application. For instance, the perceived usefulness of gamification declines with use and with learner’s age (Koivisto & Hamari, 2014). At the same time, because participants in our experimental group self-selected to use the app, it was likely there was a sampling bias which may have resulted in more conscientious students who were open to new experiences participating in the experimental group (Cochran, 1977).

Another limitation of our study lies in the sample composition. While our initial goal was to undertake analysis separately for the accounting and sciences cohorts (see Figs. 3 and 4, for examples of initial findings), a decision was made to combine the cohorts to create a larger sample and produce statistically meaningful results. It is worth noting that when analysed separately, the app’s positive impact for the combined science cohorts was found to be significantly lower than for the accounting cohort, showing only 5–7% improvements in retention rates. These differences can be explained with disciplinary specifics of the accounting and science cohorts.

Fig. 3

Student retention rates for sciences and accounting

Fig. 4

Student retention rates by course

Conclusion

This study evaluated the effectiveness of a gamified mobile app as a learning tool. Relationships between student retention rates, academic achievement and the app usage were studied, with a number of positive correlations emerging. With interdisciplinary comparisons deemed problematic due to small cohort numbers, combining accounting and several sciences cohorts into one larger sample allowed for more statistically significant findings to be generated.

Keeping in mind that correlations do not necessarily entail causations and that other factors outside the study’s purview could be at play, the marked increase of student retention rate by 12.23% may suggest there was indeed some important trigger present for this increase to happen, especially since other variables (teaching staff, curriculum, and assessment types) did not change. This increased retention phenomenon is of interest, considering the expected decrease in first-year university student retention rates (O'Keeffe, 2013). At the same time, a 7.03% increase in students’ average academic performance may also suggest there is a potential to go beyond simply ‘engaging’ students in the learning activities with the app to actually boosting their academic knowledge acquisition and retention, converting it into higher final grades.

Finally, the positive correlation of .40 between students’ scoring highly on the app and achieving higher academic grades is also of importance as it can be used as a measure of internal consistency, suggesting that a subject-integrated app is well-positioned to help guide students towards better learning outcomes. With its design drawing on such pedagogical principles as spacing and testing effects as well as taking advantage of such known motivation-boosting features as digital badges and leaderboards, the app worked in synch with other learning processes, serving as a self-regulated but content-aligned learning tool. Considering evidence that mobile learning interventions tailored to learner groups homogenous in age have generally stronger impact (Sung et al., 2016), the fact that the presented intervention was introduced in the units traditionally populated by direct school leavers likely factored into the overall positive findings. Further trials, including those involving inter-class control groups and comparative studies with different disciplines and modes of learning can help clarify the findings further.

General implications of the findings are concerned with three main stakeholder groups in HE: universities, lecturers and students. In regards to the implications for universities, economic and social consequences of student attrition are substantial and can be understood in terms of lost tuition fees, forfeited funding for student places, reputational damage, bad publicity and various resources used up on students who ultimately do not complete their degrees (Johnson, 2012). The presented analysis of student engagement with the app and the consequent increased student retention suggests it is worth continuing to explore the retention-boosting potential of mobile app technologies.

Another set of implications are of concern to lecturers, and educators more widely. The following affordances of the app emerge as important to be considered when designing impactful mobile app offerings for students: an option to customise content to a specific course, the ease of use and general flexibility, the automated collection of the app usage analytics, and taking advantage of students’ existing patterns of engagement with mobile technologies to integrate the learning app into their daily usage repertoire.

In regards to students as stakeholders of mobile learning initiatives, there are three key implications. First, as students’ demand for personalised education is growing, mobile apps can enable student access to course material at a time of their own choosing by ‘pushing’ the quizzes to their mobile devices in sync with lecture and tutorial cycle. Second, the app’s leaderboard and digital badges features generate feedback for students allowing them to see how they are performing compared to their peers, with points and badges serving as milestones of the learning process. Third, the app can be used as a revision tool students can utilise when preparing for assessments.

The implications for these three groups of stakeholders present some useful considerations for HE institutions, academics and students, who are interested in taking better advantage of mobile technologies for learning.