Challenges and contexts in establishing adaptive learning in higher education: findings from a Delphi study

Abstract

Higher education institutions are increasingly interested in using adaptive learning as an innovative data-driven approach to teaching. The actual use of adaptive learning in courses remains, however, low. This is despite positive attitudes of institutional leaders towards its adoption and promising results of early studies on its effectiveness.

This study examines the challenges that prevent higher education institutions from adopting adaptive learning concepts in teaching. We used a four-stage Delphi design to empirically identify, categorise, and prioritise the challenges of adaptive learning raised and rated by experts from two universities with different organisational and socioeconomic contexts, one from Switzerland and one from South Africa. Considering different contexts allowed us to include various perspectives on the research topic and thus broaden the view on the challenges of adaptive learning. Overall, three main dimensions related to technological, teaching and learning, and organisational challenges with eight corresponding categories were identified. Our findings revealed clear differences between the two universities regarding the emerged challenges and their rankings. These differences are linked to different socioeconomic backgrounds (South Africa and Switzerland) and organisational contexts (e.g., type of the university, teaching model, and implementation phase) of the universities. We conclude by proposing practical recommendations for institutional leaders and project implementers on the factors to be considered when implementing adaptive learning in higher education settings. These recommendations relate to the necessary infrastructure, institutional commitment, support and resources.

Introduction

Higher education faces many challenges today, including quality, high costs, diversity of students, equitable access as well as providing learning opportunities to other non-standard learner groups e.g., older adults (Elletson & Stromeyer, 2019; Weber, 2019). Various innovative teaching approaches and technologies are currently emerging to meet these challenges (Veletsianos, 2016). One of them is adaptive learning that provides a personalised learning experience mainly in blended and online learning environments. Enabled through technological innovations such as learning analytics and machine learning, adaptive learning refers to technologies or systems that monitor learner progress and use data to continuously modify teaching content to the behaviour and needs of individual learners (Becker et al., 2018). Adaptive learning systems can be of various types, ranging from simple systems based on a preconceived set of rules to complex systems with self-learning algorithms. Research on the empirical impact of adaptive learning is still limited, mainly because adaptive systems are still in their infancy (Weber, 2019). However, some early studies could show positive effects on students’ learning outcomes (e.g., Bailey, Vaduganathan, Henry, Laverdiere, & Pugliese, 2018; Holthaus, Pancar, & Bergamin, 2019; Yarnall, Means, & Wetzel, 2016) and a decrease in course dropout rates (Daines, Troka, & Santiago, 2016). In a survey on computing and information technology in higher education, Green (2018) found, for example, that a majority of higher educational leaders show a positive attitude toward adaptive learning and believe that it has great potential to improve student success. However, the same survey has found that just 8% of educational courses use adaptive learning technologies in praxis.

Despite the recognised benefits and growing interest in using adaptive learning in teaching, its broad implementation remains rather limited. The review of previous studies shows that higher education institutions face various barriers and challenges when testing or adopting adaptive learning concepts. The main types of challenges discussed in the literature, we elaborate more in details below, relate to technology, pedagogy, and management-related issues (e.g., Bailey et al., 2018; Johnson & Zone, 2018; Zliobaite et al., 2012). Major technological challenges involve, for example, dealing with real time data (Zliobaite et al., 2012), difficulties in integrating adaptive learning solutions into existing learning management systems (LMS), the complexity of adaptive systems and their ease of use (usability) (Dziuban et al., 2018; Tyton Partners, 2016). Some pedagogical challenges relate to the need of redesigning curriculum (Educause, 2017) and the role of faculty in the adoption process (Oxman & Wong, 2014; Tyton Partners, 2016). One commonly cited challenge is faculty engagement. When first exposed to the adaptive approach, faculties often show resistance towards using technology (C. Johnson & Zone, 2018). In many cases, faculties express concerns about the benefits of adaptive learning, their diminishing role in a course design, loss of control over courses, and additional workload (Hall Giesinger, Adams Becker, Davis, & Shedd, 2016; Izumi, Fathers, & Clemens, 2013; C. Johnson & Zone, 2018; Tyton Partners, 2016). In particular, in the phase of piloting adaptive learning, faculties often struggle with the use of adaptive software for communicating with students and modifying learning content, because they have lack of experience with adaptive technologies or receive inadequate support (O’Sullivan, 2018). C. Johnson and Zone (2018) argue, for example, that a successful adoption of adaptive learning can be achieved through faculty engagement, emphasising a faculty role in the adoption process and creating a faculty culture embracing technology use. Some challenges discussed in the literature relate to managerial issues. They include integration of adaptive initiatives into the overall university strategy to support its larger goals such as reducing drop-outs, for example, (O’Sullivan, 2018). Another important challenge is the institutional commitment in the form of leadership support, involving the promotion of an innovation culture and allocation of necessary financial and personnel resources (Bailey et al., 2018; C. Johnson & Zone, 2018). Reflecting on the slow progress of its adoption, Weber (2019) concluded that the major challenge today is, however, a huge investment in time, money, resources and vision, mainly due to the remaining complexity of adaptive technology, high licensing fees and long-lasting scepticism of faculties towards its potential to disrupt education in general.

The review of previous studies has shown that challenges of adaptive learning often appear fragmented, or they are discussed through particular lenses in the literature. No systematic classification of the challenges related specifically to adaptive learning has been proposed until now. It also remains unclear which challenges deserve higher attention in the implementation process. Finally, the discussed challenges are dominated by the views of just a few countries (e.g., U.S., UK, Australia), in which adaptive learning is the most used in teaching practices. To our knowledge, no comparative perspective has been published to the challenges, particularly in the context of different socioeconomic conditions so far. While conducting comparative studies across different countries is often useful to consider cases with different experiences in the research topic (Johnson, Jacovina, Russell, & Soto, 2016) and with a maximum variation in the properties that the study intends to explore (Creswell & Poth, 2018). This approach could help researchers reconceive important factors that could influence the adoption of technological innovations in educational environments, identify new ones, and gain different perspectives on the adoption problem.

To address the challenges related to adaptive learning more systematically, we use the Delphi method based on the procedure proposed by Skulmoski, Hartman, and Krahn (2007). This study aims to identify, categorise and prioritise challenges of adaptive learning at two universities with different socioeconomic backgrounds (South Africa and Switzerland) and organisational contexts (e.g., type of the university, teaching and learning traditions, and implementation stage). The results of the study have revealed that both universities address within their specific contexts different challenges in different priorities across the three identified dimensions: “technology”, “teaching and learning”, and “organisation”. The technological dimension includes fundamental infrastructural issues such as internet access and internet quality, technological specifications of adaptive systems, and human-related issues referring to teachers’ and learners’ attitudes and perceptions about adaptive learning. The teaching and learning dimension comprises instructional and curriculum elements as well as learners’ and teachers’ characteristics including motivational aspects, required skills, and necessary commitment for teaching and learning in adaptive environments. The organisational dimension includes issues related to institutional strategies, managerial issues that focus on providing support to lecturers and students and assuring necessary resources to adaptive learning projects.

We begin by describing the two universities, one from Switzerland and one from South Africa. Then we outline the research design and methodology. In the results section, we present a comparative analysis of the identified challenges and their rankings across the two universities as well as a category system of the challenges. Then we discuss possible differences in challenges and their rankings by using qualitative data of the respondents. Finally, we point to some limitations of the study, summarise the conclusions that can be drawn from the present research, and give some practical recommendations on the implementation of adaptive learning in higher educational institutions. These recommendations could be valuable for higher institutional leaders and project implementers seeking to introduce adaptive learning concepts in teaching.

Context of the two universities

For this study, the role of the context is of particular interest, and therefore it will be explained more in depth. The importance of context in educational research has been recognised for some time (Gamoran, Secada, & Marrett, 2000; A. M. Johnson et al., 2016). Despite its widely accepted importance, context is difficult to define (Bate, Robert, Fulop, Øvretveit, & Dixon-Woods, 2014). We refer to context as a unique environment of a university with interrelated conditions and factors that affect the adoption of adaptive learning. At the organisational level, the context (“inner context”) includes issues related to the administrative structure of a university, leadership, teaching and learning traditions and practices, experience with adaptive learning, viz. the implementation stage. At the societal level, the context (“outer context”) includes laws and regulations determining universities’ primary roles, mandates, linkages to industry, federal funding regulations, as well as cultural, political and economic events and conditions. According to Bate et al. (2014), the inner context can be managed directly, but the outer context is usually too big and complex. Nevertheless, the outer context can be managed indirectly by taking advantages of opportunities arising from the context, for example by making use of favourable economic conditions for introducing effective educational technologies.

Recognising the importance of the context, we briefly describe the two universities and their motivation to introduce adaptive learning concepts in teaching.

Swiss university

The Swiss university offers nationally recognised Bachelor’s and Master’s degrees as well as a range of further courses in economics, information technology, law and health, based on a blended learning model. For 80% of their study time, the students complete in the form of supervised distance learning via the Moodle learning platform, and the remaining 20% in face-to-face classes at regional centres. The average age of the students is 33, with 32% female students. Most of the students are part-time or full-time employed. The Swiss university maintains a didactic support centre within the Department of E-Didactics. This centre offers instructional design support for lecturers, technical support for the use of the learning platform for lecturers and students, and online training to improve the digital skills required for teaching and learning in online environments. The interest in using adaptive learning concepts at the university resulted from the heterogeneous composition of the student groups (e.g., different lifestyles, full or part-time employment, family responsibilities) and the need to make the Swiss university education system more flexible and efficient.

Since the university provides courses on the Moodle platform, it was agreed to develop and test adaptive rule-based instruction concepts first on this platform. To date, eight courses with adaptive concepts have been tested and implemented on Moodle. The further development of adaptive learning remains one of the central strategic pillars of the university’s digital transformation.

South African university

The South African university officially came into being at the beginning of the 2000s as part of the South African government’s plan to transform higher education. Some previous campuses were merged to establish a university, where South Africans from different backgrounds have come together. The South African university is a traditional teaching and learning university, but also offers most of the degrees with a distance mode of delivery. The university offers ample masters and PhD programs across the eight faculties. The university has adopted a strong strategic priority for blended learning and self-directed learning to integrate in teaching and learning across the board. Their blended learning view is stipulated in the strategic priority document of the university. Most importantly, apart from the alignment with the blended learning aim, personalised and adaptive learning are in line with the teaching and learning strategy – 2016 to 2020 of the university. Personalised and adaptive learning are currently being researched in the research unit of self-directed learning with the aim to be piloted and rolled out into more modules and programmes across the campus.

Implementation stages of adaptive learning

Like many technological innovations, the implementation of adaptive learning occurs in stages (Graham, Woodfield, & Harrison, 2013; Rogers, 2003). The South African university is still at the beginning of the implementation process. It is aware of the advantages adaptive learning brings to students, faculties, and organisation as well as exploring the ways of how adaptive concepts can be integrated into the curriculum. The Swiss university implements adaptive learning at the course and module level, testing the new teaching practices and policies to facilitate the wide adoption across the faculties. Neither of the universities, similar to many other higher education institutions implementing adaptive learning around the globe today (Weber, 2019), has achieved the scaled implementation of adaptive learning across courses and faculties yet. Bringing their unique contexts to the Delphi study, both university panels aimed thus to reflect on the challenges of adaptive learning and the framework conditions required to achieve the scaled implementation of adaptive learning at their respective university in the future.

Research methodology and design

To identify, categorise and prioritise the challenges of adaptive learning in both universities, we used the Delphi method based on the procedure proposed by Skulmoski et al. (2007). Delphi is a research method used to collect anonymous judgements of experts, usually on a poorly defined topic or complex problem without face-to-face interaction. A typical Delphi procedure consists of multiple rounds of data collection and analysis strategies. The analysis of each round contains the summarised information and feedback report on experts’ judgements that researchers send to the participants. The Delphi method is based on the methodological assumption that the experts’ collective judgement is of greater quality than judgements of individuals, because experts are assumed to have the ability to reconstruct incomplete or uncertain knowledge based on probabilistic mental models. When they use additional information provided in the feedback for the further reconstruction of their mental models, such mental models are validated, which leads to higher-quality judgements in the follow-up surveys (Häder, 2014). Delphi has a variety of application areas. It is well suited for generating new ideas, predicting future developments, reaching consensus or obtaining and evaluating experts’ opinions (Häder, 2014; Linstone & Turoff, 1975). Our Delphi study was designed with the purpose of gathering experts’ opinions on the challenges of adaptive learning. The Delphi method allowed us to gain experts’ collective opinion on such a complex phenomenon as adaptive learning in a structured but still flexible manner. Another reason for the Delphi method was its broad acceptance and use as a rigorous research method in higher educational research (Clayton, 1997). The Delphi method has been broadly applied to forecast the emergence of new technologies, to identify technology implementation strategies, to develop curricula, and to deal with challenges in educational settings in general (Nworie, 2011; Osborne, Collins, Ratcliffe, Millar, & Duschl, 2003; VanDykGibson, 2016).

Workshop

Prior to the Delphi survey, a two-day workshop was held with experts and other members of both universities on the topics of technology-based, adaptive learning and modern sensors of learning analytics in August 2016 in South Africa. The aim of the workshop was to exchange experiences in the field of adaptive learning as a preparation of the Delphi study. In addition, the selection criteria for the Delphi participants were defined, the recruitment strategy was identified, and a common understanding of what adaptive learning is, was built during the workshop. The following definition of adaptive learning was integrated into the online Delphi survey. Adaptive Learning are learning applications or systems, which continuously adapt and personalise instruction on different educational parameters like the sequence of tasks, task difficulty, time and type of feedback, pace of learning speed, reinforcement plan, and others depending on the current individual needs of the learner (based on Zimmermann, Specht, and Lorenz (2005)).

Selection of experts

A purposive sampling strategy (Elo et al., 2014) was chosen for the Delphi study to identify participants with necessary expertise. The experts were selected on predetermined criteria to ensure that they bring along various perspectives and the required knowledge about adaptive learning from their fields. The predetermined criteria for being included in the sample were:

• teaching or research experience in the field of technology-based learning or self-regulated learning;

• expertise in providing technological learning applications;

• university management staff involved in a strategic decision-making regarding innovations in learning;

• affiliation to one of the institutions (South African university or Swiss university).

The recommendations on the required sample size are highly variable in the literature (Avella, 2016; Cochran, 1983; Delbecq, Van de Ven, & Gustafson, 1975; Häder, 2014; Skulmoski et al., 2007). For our Delphi study, we aimed to reach a panel size of minimum 20 experts, because this panel size is, on the one hand, large enough to allow different perspectives on the researched topic to emerge; on the other hand, it is small enough to allow a management of participants’ responses to open-ended questions. The final panels consisted of 27 experts of the South African university and 24 experts of the Swiss university. We recruited the experts for both panels following the “gatekeeper” recruitment strategy by using field representatives who chose and suggested appropriate experts within their faculties and personal networks based on the predetermined criteria (Brady, 2015).

Table 1 gives an overview of the participants along the rounds and their current occupations. The South African university panel size of the first and second rounds remained stable (n = 20). However, it should be noted that some new participants joined the second round, and some participants of the first round did not participate in the second round any longer. In contrast, the final Swiss university panel was formed in the first round, and no additional participants joined the panel after the first round.

Table 1 Participants’ Occupations in Each Round of the Study

Procedure

The Swiss university study ran from July 2017 to August 2018, while the South African university study ran from May 2017 to August 2018. For each of the four rounds, the participants received an email with a link to an online web-based survey.

Ethical considerations

For each round of the Delphi study, the participants had to explicitly state their agreement to participation in it. Non-participation had no negative effects on the invited experts. The experts could withdraw from the study at any time without giving a reason. All responses were anonymised for the data analysis procedure and communication of intermediate results in feedback reports to participants after each Delphi round.

Data collection and analysis methods

The Delphi study was conducted in four stages separately for each university, as shown in Fig. 1.

Fig. 1

Description of the Delphi Process and Research Methods. Note. TBL – technology-based learning, AL – adaptive learning

The first round was conducted to identify the challenges of technology-based learning and relevant dimensions that served as a backbone for designing the second round with the focus on adaptive learning specifically. Since adaptive learning is quite a new term and a complex phenomenon in the educational landscape, the first round aimed to identify the challenges of technology-based learning in general. The experts were asked to provide short descriptions of the challenges related to technology-based learning that their universities face. Open-ended questions were used to collect the responses of the experts.

The second round was designed based on the results of the first round. The experts were asked to describe challenges related to adaptive learning along the dimensions and categories identified in the first round. As in the first round, open-ended questions were used in the second round.

The MAXQDA software was used for managing participants’ responses to open-ended questions and coding the data. The data analysis of the participants’ responses was conducted by inductive coding without any pre-given sets of codes (Miles, Michael Huberman, & Saldaña, 2014; Saldaña, 2013). This approach allowed us to stay open to all possible responses produced by the experts and thus to diminish preconceived concepts and opinions of the researchers during the process of data analysis. Figure 2 shows the overview of the data analysis procedure with the corresponding strategies to ensure the quality of data analysis. Following the recommendations on ensuring the quality of the research by Elo et al. (2014), internal validity was assured by the discussions among the members of the research team (researcher triangulation). Reliability was achieved by describing how the single rounds of the Delphi study built on each other, documenting every process step of data analysis, and defining theoretical concepts (consistency rules).

Fig. 2

Data Analysis Procedure with the Corresponding Strategies for Ensuring the Quality of Data Analysis

The third round focused on reducing the identified challenges of adaptive learning to a reasonable number, so that the participants could rank the challenges in the last (fourth) round. The participants of each panel group were asked to choose, from the list of identified challenges, up to ten which in their opinion were the most relevant challenges that needed to be overcome, so that adaptive learning could be implemented sustainably at their institutions. In addition, the participants had to justify their choice. We have chosen only those challenges for the last ranking round that had been selected by at least several participants (by a minimum of three participants of South African university and a minimum of five participants of Swiss university). Several challenges (e.g., affordable private internet access and internet access) were aggregated to a common challenge (e.g., affordable private internet access) to avoid overlapping. The aggregation of the emergent challenges was done in the last round, because we wanted to ensure that all possible issues raised by the participants were included in the final lists of challenges for the ranking round. The final lists of challenges resulted in 16 Swiss university’s challenges and 11 South African university’s challenges.

In the fourth round, we provided the participants with the final lists of challenges as well as with anonymised summaries of their justifications respectively. Before the challenges were ranked in order of their priorities for the broad implementation of adaptive learning at both universities, the participants could read the summaries of the previous round and make comments about the summarised statements. Whenever possible, the summaries included original (close to the quotes) and contradictory statements of the participants, because we aimed to consider the opinions of all the experts. To balance the effect of bias, the order in which the challenges were listed was manipulated randomly for each expert.

Results

In this section, we first compare two universities with regard to the identified challenges. Then we present the challenges that two panels selected for the ranking round. Finally, we present the results on how the experts of both panels ranked those challenges in order of their priorities.

Identified and selected challenges

One of the objectives of our study was to identify and systematically categorise challenges related to adaptive learning using empirical data from two universities. Table 2 presents the comparative analysis of the challenges of adaptive learning resulting from the second round. In total, 47 challenges were identified and grouped by 8 main categories along the 3 empirically identified dimensions. The technological dimension “technology” included two main categories related to infrastructural, hardware and software issues and how lecturers, students and staff use technology in educational contexts, focusing in particular on their perceptions and personal beliefs about the use of technology when teaching and learning. The teaching dimension “teaching and learning” included three categories that referred to learners’ and lecturers’ personal characteristics, as well as to instructional and curriculum aspects that provided learning and teaching environments for lecturers and students. The organisational dimension “organisation” comprised categories related to the required resources and strategic orientation of an institution towards adopting e-learning innovations in general and an adaptive learning approach specifically. Additionally, it comprised general managerial issues including implementation process, personnel policies, trainings, faculty support, and quality assurance in teaching.

Table 2 Full List of Challenges of Adaptive Learning by Dimensions and Categories

The comparative analysis of the challenges identified in the second round revealed that both universities had some common challenges (17 items) and some unique challenges (16 items of Swiss university and 14 items of South African university).

A further comparison of the challenges on the dimensional level showed that the universities had quite similar profiles (Fig. 3), if we consider all challenges shown in Table 2. Most of the challenges of both universities relate to organisational issues, followed by the challenges related to teaching and learning issues. The least of the identified challenges are technological. However, the universities’ profiles are changing (Fig. 4), if we consider only the challenges selected for the ranking round (up to ten most important items). The list of selected challenges is displayed in Table 3. After the selection procedure, organisational challenges became even more salient for both universities in round 3. However, both universities differ regarding the two other dimensions: whereas the South African university panel selected more technological challenges for the ranking round, the Swiss university panel paid greater attention to teaching and learning issues.

Fig. 3

Percentage of challenges of adaptive learning identified in the second round across dimensions for each university

Fig. 4

Percentage of the selected challenges of adaptive learning for the ranking round across dimensions for each university

Table 3 Comparison of Swiss university and South African university Panels’ Rankings of the Selected Challenges

These differences in the universities’ profiles after the selection procedure provide first indications about the relevance of context in the implementation process.

Ranking

In the last round, the two panels were asked to rank the challenges according to their priorities for the broad implementation of adaptive learning at both universities. The results of the ranking are shown in Table 3. Measures of central tendency (means and medians) and standard deviation were used to define agreement of the panels’ participants on the relevance and priorities of the challenges in the implementation process. To understand better the differences in the selected challenges and their priorities, we decided to include all selected challenges for our analysis and not to limit our analysis, for example, to top-five challenges. Otherwise, some challenges relevant to the corresponding panel could easily be overlooked. This is briefly demonstrated on the basis of the three types of challenges displayed in Table 3:

• a) common challenges selected by both panels (in bold);

• b) common challenges selected by one of the two panels (underlined);

• c) unique challenges selected by a corresponding panel (not marked);

Both panels selected only 3 out of 17 initial common challenges for the ranking round. If we restricted our analysis to the top-five challenges, two challenges of the South African university related to personnel and financial resources and usability aspects of the system would not be included in our analysis at all.

However, these challenges do remain relevant for both panels as they were selected for ranking from the initial list. Yet, depending on the university, some challenges deserve more attention and must be addressed in the implementation process with a higher priority. Before providing required resources to adaptive learning projects, for example, the South African university should grapple first with some technological challenges related to general infrastructure and internet access.

The overall assessment of the selected challenges indicates a strong disagreement between the two panels regarding the relevance of the challenges (selection of up to 10 important challenges) and their priorities (ranking position) in the implementation process (Table 3). For example, the South African university panel ranked the common challenge related to the accessibility and availability of necessary infrastructure as one of the top priorities (third ranking position). That infrastructural challenge was not even selected by the Swiss university panel for the ranking round. Similarly, only the Swiss university panel selected and highly ranked (fifth ranking position) the common challenge related to the professional development of the lecturers. Both panels agreed, however, on the importance of institutional commitment to adaptive learning in the implementation process, rating it as one of the top priorities to be addressed when implementing adaptive learning in their institutions. Finally, unique challenges of the universities appeared in the top positions. The Swiss university panel top-ranked the unique challenge related to the need of forming required competences and expertise in staff, whereas the South African university panel considered the unique challenge related to the attitude of lecturers and students towards technology as a second priority in the implementation process.

Overall, both panels disagreed strongly on the importance of the challenges for the successful implementation of adaptive learning in their institutions by selecting predominantly different challenges for the ranking round. This finding points to the relevance of considering a specific context when implementing adaptive learning. In addition, both panels ranked the three common challenges differently, which also supports our claim that the priorities of challenges must be examined attentively in the implementation process.

Evidence from qualitative data

In the following, we use the qualitative data of the panel participants generated through open-ended questions in the second and third rounds with some exemplary quotations to show the differences in the selected challenges and their rankings between the two universities. The three identified dimensions, namely, “technology”, “teaching and learning”, and “organisation” serve as lenses for organising the presentation of the findings.

Technology

The South African university panel selected and high-ranked more technological challenges than the Swiss university panel. A salient difference between the two institutions is visible in the category “infrastructure, hard- & software”. A South African university expert described the educational infrastructure as “a determining factor” when implementing adaptive learning: “The necessary infrastructure needs to be in place to effectively and sustainably implement adaptive teaching; if not, it will be very hard to sustain”. In addition, the South African university panel commented on the current state of the infrastructure, stating that “not all students have access to infrastructure (hard- and software) when at home; not all have smartphones”. In contrast, the Swiss university panel focused primarily on the quality aspects of technology, including the good usability and robustness of an adaptive learning system. One expert has justified the importance of the usability challenge: “Usability is crucial, since the learning process of the students is to be supported by adaptive learning environments. Lack of usability would lead to additional cognitive stress and make learning more difficult”. Some participants also reported that robustness was important for the acceptance of an adaptive learning system and a good image of the university.

Affordable private internet was another infrastructural challenge relevant solely for the South African university. The South African university panel ranked the affordable private internet challenge as one of the top priorities to be addressed in the implementation process. The experts pointed out that many students were not able to afford internet at home because of high costs. One expert suggested that “the university management should drive national initiatives by providing exceptions on existing data pricing structures, or obtain free data for educational use, or negotiate with partners from the private sector for affordable internet packages”. Another expert reported that the internet challenge needed to be addressed on both institutional and national levels: “One of the major challenges in South Africa for online learning, which is part of an adaptive-learning strategy, is the problem of affordable access to internet, which government and university management should address”.

In addition, the two university panels recognised the importance of the issues related to lecturers’ and students’ perceptions and beliefs about the use of technology; focusing, however, on university-specific aspects. The South African university panel selected and second-ranked the unique challenge related to the negative attitude towards technology, including adaptive learning systems. The experts’ comments referred to the lack of buy-in from lecturers and their unwillingness to integrate learning technology in curricula. The South African university lecturers continued to see technology as an add-on and remained focused on traditional instruction. One respondent emphasised the role of lecturers’ openness towards technology for adaptive learning: “For adaptive learning to be sustainable, lecturers will all need to buy into the idea and be more open to the use of new and different technology”. A negative attitude towards technology was not addressed by the Swiss university panel at all. Instead, the Swiss university panel emphasised the importance of recognising advantages of adaptive learning by students and faculty for improving its acceptance and intrinsic motivation to teach and learn in adaptive environments. Notable comments related to that challenge were: “The particular benefit of the adaptive process must be obvious for students and lecturers”; “If benefits and additional value [of adaptive learning] are recognised, the students will also use the adaptive system”.

Teaching and learning

The challenges related to teaching and learning played a more important role at the Swiss university than at South African university. In particular, challenges related to instructional and curriculum elements were paid greater attention by the Swiss university panel. The Swiss university panel stressed possible implications of adaptive learning concepts on the existing blended learning model of the university, addressing the issues related to new quality requirements for online courses, development of suitable instructional materials, redesign of existing courses, rethinking of the combination of classroom and online instruction, and new interaction requirements in adaptive environments. Representative examples of responses and comments were: “Adaptive learning can only take place if the learning materials are designed for adaptive learning”; “By hook or crook, adaptive learning cannot be applied to each module. Realistic assessments protect against unnecessary investments”; “In particular, it should be assessed which contents are suitable for adaptive learning”.

Whereas the Swiss university panel recognised changes in curriculum and instructions, which the implementation of adaptive learning involved, the South African university panel emphasised lecturers’ and learners’ personal characteristics challenges, pointing for example to the insufficient level of students’ digital literacy and media skills. One expert reported: “While students may be used to using advanced technologies, these skills may need to be fine-tuned for the academic setting and focused on the achievement of disciplinary related outcomes”. The challenge related to students’ digital literacy was not selected by the Swiss university panel as a priority to be solved at all. However, the Swiss university panel highlighted the importance of self-regulated learning skills when learning in an adaptive environment, commenting on it that it was also “indispensable for extra-occupational training” and “fundamentally decisive in distance learning” in general. One expert pointed to the “potential problem of adaptive learning” in the self-regulation context, noting that “students might get the impression that an adaptive system is alone responsible for their learning success, because it suggests the appropriate tasks to them”.

Finally, the South African university panel selected a university-specific challenge related to the staff’s workload as one of the priorities. One expert stated: “Staff is too overloaded, and we do not have time to sit and experience problems with technology or to figure out what is needed”. Another expert reported that “most staff members (management, admin and lecturers) are working under a lot of pressure due to the heavy workload. Therefore, most staff members do not have time to explore alternative teaching-learning strategies”.

Organisation

Although both universities faced different challenges in different priorities when implementing adaptive learning in their institutions, organisational issues was the area in which the universities differed least.

Both university panels stressed the crucial role of the institutional commitment to innovative pedagogical approaches such as adaptive learning, rating it as one of the first priorities. They clearly demarcated institutional commitment from an operative didactical and technical support that should be provided to students, lecturers, faculty and staff. Emphasising the role of top management in the implementation success, the South African university panel commented on the need of “the institution to be committed to the financial and strategic (training, teaching-learning strategy) challenges the [adaptive learning] project might entail”. One respondent reported that institutional commitment included “budgetary allocation for technological resources and a dedicated IT support”. In general, most of the experts of both panels mentioned the urgent need for personnel, financial and time resources in adaptive learning projects. Some experts warned “not to underestimate the development expenditure”, because the implementation of adaptive learning “requires a lot of time”. Providing personnel and financial resources to adaptive learning projects, as well as forming required competences and expertise in staff, were ranked as two organisational top challenges by the Swiss university panel. Some Swiss university respondents suggested to “evaluate lecturers’ competence profiles” and noted the difficulty to “have the people [with the right competences] on board”.

Finally, both panels also recognised the need for providing different types of support for students and lecturers to succeed in adaptive learning environments. Both panels commented that lecturers and course developers required support in technical and didactical questions. Like lecturers, students should be able to receive support with technical problems and hands-on training on what adaptive learning was, what skills and competences it required and, finally, how to use adaptive learning systems when learning.

Overall, the qualitative data of the experts and comparative analysis of the challenges between the two universities showed that the universities faced predominately different challenges and in different priorities when implementing adaptive learning at their institutions. At the dimensional level, the South African university panel paid greater attention to technological issues, whereas the Swiss university panel addressed issues related to teaching and learning. The interesting finding was that most of the similarities between the universities were found at the organisational level. In particular, the common challenges related to institutional commitment and required resources were highly ranked by both panels. Overall, the differences in challenges and their rankings support our claim that challenges of adaptive learning are context-specific, and their priorities do matter when implementing adaptive learning in a higher educational context. In the following session, we discuss possible explanations for differences in the selected challenges and their rankings and point to the limitations of the study.

Discussion

The findings of the study revealed clear differences in challenges of adaptive learning between both universities. At the dimensional level, the South African university panel identified and highly ranked many technological challenges. In contrast, the Swiss university panel allocated greater importance to teaching and learning issues. Most similarities between both panels were found on the organisational dimension. Thus, in the following section we will address the variations between the two panels regarding the selected challenges and their rankings according to their importance for the scaled implementation of adaptive learning at their institutions. The discussion section is organised by the dimensions of challenges related to technological, teaching and learning, and organisational issues.

Technology

The Swiss university panel emphasised the importance of the quality aspects, such as good usability and robustness of adaptive systems. This is presumably due to the experience gained by piloting adaptive learning courses. In comparison to the South African university, which plans to introduce adaptive learning in online teaching, the Swiss university is already in the second stage of implementation that is characterised by adopting adaptive learning strategies and experimenting with new policies and practices to support its implementation (Graham et al., 2013). In contrast, the South African university panel emphasised the importance of infrastructural and internet issues. However, the Swiss university panel addressed neither of those challenges in the ranking round. Due to the basic infrastructure already being in place, this challenge might seem negligible for implementation success. In Switzerland, as in the U.S., for example, many infrastructural first-order barriers seem to have been overcome (Ertmer, Ottenbreit-Leftwich, Sadik, Sendurur, & Sendurur, 2012), these remain, however, one of the priorities in South Africa, as qualitative data of this study shows. The limited broadband access and high cost of internet are serious technological challenges that impede online learning in South Africa, as previous research showed (Elletson & Stromeyer, 2019; UNESCO, 2017), and ones that were also captured in our study. Deciding to implement adaptive learning in a South African context might thus have tremendous implications for government and private sectors regarding policy changes. This includes creating better conditions to ensure broadband accessibility of internet and access to infrastructural resources for the higher educational sector. It also raises questions concerning equal access to education. These findings suggest how close infrastructural challenges are linked to the broader socioeconomic context of the university, highlighting the urgent need for solutions on the institutional, local and national levels.

In addition, the findings revealed that both universities have different perceptions and beliefs about the use of technology in teaching and learning. Negative attitude towards technology, known as a second-order barrier (Ertmer, 1999), was selected and highly ranked solely by the South African university panel. The Swiss university panel did not address that challenge in the study at all. The explanation might lie in the different types of the universities. The South African university is a traditional on-campus university with some e-learning offers, whereas the Swiss university is a university that for decades provides studies to a large extend through technology. Lecturers, staff and students are thus well skilled in using technology for teaching and learning purposes. These different structural characteristics might lead to the establishment of different teaching-learning traditions, such as different perceptions about using technology for teaching and learning at both universities (Eaves, 2009).

Teaching and learning

Clear differences between the two panels were found in terms of teaching and learning issues. While the Swiss university panel emphasised the relevance of some challenges linked to fundamental rethinking of the curriculum, the “blended learning” model, and redesign of instructional content caused by adaptive learning, the South African university panel focused on student’s skills. Indeed, redesigning curriculum and courses for adaptive learning requires a different approach to content development (Educause, 2017). Redesigning content at the level of lesson and learning objectives, instructors follow the so-called backward design approach, in which learning activities are developed in accordance with predetermined objectives and outcomes. Implementing adaptive learning thus requires instructors to change their instructional design practices fundamentally, which might result in an additional workload and resistance to implement adaptive learning in their courses (A. M. Johnson et al., 2016). It is not surprising that this curriculum challenge has been recognised as one of the downsides of adaptive learning (Educause, 2017) and as an important challenge by the Swiss university panel. Being in the first implementation stage of adaptive learning (Graham et al., 2013), the South African university might have not recognised all the challenges related to instructional and curriculum elements that adaptive learning involves yet. This finding helps, therefore, to raise awareness of such curriculum challenges for institutions that plan to introduce adaptive learning in teaching. In terms of teaching and learning aspects, only the South African university panel emphasised the relevance of students’ digital literacy and media skills. This difference between the universities can be explained by the specific socioeconomic context. As previous research shows, the access to digital literacy in South Africa is still unevenly distributed and associated with the middle class - an advantaged minority who have access to technology (Kajee & Balfour, 2011). According to Kajee and Balfour (2011), this justifies the constrained mandate of higher education to provide wide access to learning to students from disadvantaged groups regarding class, race and gender.

Organisation

It was in terms of organisational issues that the panels differed least. As with previous research (Bailey et al., 2018), both panels stressed the crucial role of institutional commitment to adaptive learning. The recent study by Bailey et al. (2018), for example, identified inconsistent leadership support as one of the greatest challenges for implementing digital learning in a higher educational context. As leadership shifts its priorities, the funding level varies, having, for example, a negative impact on the resources required for digital learning programmes (Bailey et al., 2018). One of the strategies they suggest to ensure the successful implementation includes embedding digital learning in the strategic plan of the institution. This also explains why the Swiss university panel ranked the challenge related to strategical developing (the “Distance University” strategy), and embedding adaptive learning strategy in it, as one of the top priorities. In fact, aligning any technology implementation strategy to institutional goals remains crucial for their implementation success (Hall, 2013). In addition, both panels agreed on the relevance of support and training issues. However, this finding is not new. Literature on adaptive learning has already recognised the importance of institutional support and training for the successful adoption of adaptive learning (Dziuban, Moskal, Cassisi, & Fawcett, 2016; Pugliese, 2016; Wingo, Ivankova, & Moss, 2017). Finally, as expected and predicted in the literature (Bailey et al., 2018; Chauhan, Taneja, & Goel, 2015; Izumi et al., 2013), both panels recognised the relevance of providing personnel, financial and time resources. This seems to be crucial for the successful adoption independent of organisational and broader social contexts of both universities. As qualitative data of both universities showed, resources-related challenges interplay with other aspects of adapting learning. This includes infrastructure, usability, complexity of adaptive systems, curriculum, course design, instructional models, skills, trainings, support services, and strategic alignments, which all need significant investments in time and money to change them according to the affordances of adaptive learning.

Limitations of the study

One of the purposes of the study was to classify the challenges of adaptive learning. We do not claim that the presented classification is exhaustive, as it relies on two cases, and more cases, for example, from a mature implementation phase (Graham et al., 2013) or other socioeconomic environments would be beneficial. For the same reason, we do not intend to generalize the findings of our research at this point. Instead, analysing first the issues related to adaptive learning within each case separately and then across the cases, we aimed at identifying differences and commonalities between both universities regarding the emerged challenges, which helped us to gain a better understanding of the challenges that could appear in different contexts. Although many qualitative researchers are opposed to the term of generalizability (Lincoln & Guba, 2000), generalization might still be achievable under certain conditions (Creswell & Poth, 2018). To generalize in our case, a careful selection of additional representative cases, for example, universities with different study models, from different implementation phases and socioeconomic environments would be imperative. In addition, we stay aware of the possible experts’ biases. Yet the resulting classification is a first systematic attempt towards categorising challenges of adaptive learning considering different contexts of universities, specifically one of a developing country.

Another limitation concerns the composition of the expert panels. Future Delphi studies should include among researchers, lecturers and management staff, for example, students as an important stakeholder group, one that is directly involved in adaptive learning and thus directly affected by the results of the study. Moreover, the Swiss university adaptive learning strategy was to implement adaptive learning via the existing Learning Management System, which included building adaptive mechanisms by internal resources. Such a strategy to adaptive learning could influence the panel’s responses. Another approach would be to acquire a commercial platform with already built-in content and adaptive mechanisms, which may lead to other challenges compared to an in-house approach. Johanes and Lagerstrom (2017) conclude, for example, that many adaptive learning applications are commercial today, so that additional challenges for higher educational institutions are a proprietary character of many algorithms, results and data that companies are unwilling to release, e.g., via a research paper or press releases.

In addition, our Delphi design involved a single ranking round. The Delphi process of a conventional Delphi normally ends when a consensus with 70% agreement among panel participants is achieved (Vernon, 2009). However, it was impossible for us to continue the ranking process to ensure the stability of the ranking results due to the panels’ reluctance to participate after the first two rounds and the long duration of the study caused by the need of deploying the first round on technology-based learning challenges. Hence, the exact priority order of the challenges should be considered carefully.

Conclusions

The purpose of our Delphi study was to identify, classify and prioritise challenges of adaptive learning at two universities, one from Switzerland and one from South Africa. By taking a comparative approach to inquiry and using a rigorous data collection and analysis method, we identified challenges of adaptive learning based on data from two universities with different socioeconomic and organisational contexts. The challenges were classified along the three empirically identified dimensions related to technological, teaching and learning, and organisational issues with eight related categories, namely: (a) infrastructure, hard- & software, (b) perceptions and beliefs about adaptive technology, (c) instructional & curriculum elements, (d) learner characteristics, (e) lecturer characteristics, (f) institutional strategies, (g) management, (h) resources.

The presented findings revealed clear differences in the selected challenges and their rankings between the two universities, except for one organisational challenge related to the institutional commitment that was highly ranked by both university panels. We argue, therefore, that the challenges of adaptive learning are context-specific, and thus to succeed in the implementation of adaptive learning requires careful consideration of university-specific challenges and their priorities along the three identified dimensions, namely “technology”, “teaching and learning”, and “organisation”. Implementing universities should be ready to address different challenges and in different priorities within their specific organisational and broader socioeconomic contexts.

Conclusions for practice

Based on the findings of our study, we propose the following recommendations to institutional leaders and project implementers who want to adopt (1. awareness/ exploration adoption phase) or have already been piloting adaptive learning (2. adoption/ early implementation adoption phase) for instructional purposes in their institutions.

• a. Committing to adaptive learning and making it part of the university strategy

Start by examining organisational issues. Leadership commitment is one of the crucial factors that affect the successful adoption of adaptive learning. Align the adaptive learning strategy with institutional strategic goals, making it part of the university strategy. Ensure faculties buy-in and constantly strengthen leadership commitment by communicating the evidence-based benefits of adaptive learning such as its impact regarding students’ success, institutional economics and national educational policy. To receive quick pay-outs, make the implementation process agile and successive.

• b. Building the necessary infrastructure

Employ the accessibility and availability of the infrastructure, including needed hard- and software, internet access and internet quality. If a fundamental infrastructure is not in place, the implementation of adaptive learning becomes hardly possible. For institutions struggling to set up a basic infrastructure (e.g., affordable internet for students), a close cooperation of institutional leaders with the private sector or government might be a solution. When choosing or developing adaptive technologies, pay attention to their essential characteristics, including, for example, robustness and usability. They might positively affect motivation and use of adaptive technology by lecturers and students.

• c. Building needed capabilities, providing support and resources

Adaptive learning brings new requirements for students, lecturers and staff. Thus, recognise and build necessary competences and expertise by hiring, for example, instructional designers and providing high-quality training and support services to staff and students to develop needed knowledge and skills for teaching and learning in adaptive learning environments. Understanding and knowing how to use the affordances of adaptive learning might positively affect lecturers’ and students’ attitudes about using adaptive technology in teaching and learning. Implementing adaptive learning is time- and cost-consuming. Allocate necessary resources for developing staff, improving pedagogical strategies, (re) designing courses and learning materials, and conducting research for constant improvement. Recognise that the implementation process is time-consuming. Therefore, calculate a realistic workload for the involved team to support their engagement and commitment. As adaptive learning is a data-driven approach, build learning analytics capabilities, including continuous feedback to lecturers and learners to improve the quality of teaching and learning. Take care of data privacy issues.

Conclusions for research

The challenges that have emerged from this study present reasons why adaptive learning is still not broadly implemented at universities, although in theory there seem to be many advantages of such a didactic approach. To enhance the adoption of adaptive learning in higher education, we call for more research on how to tackle the challenges resulting from this study. For example: How can institutions support the acceptance of adaptive learning by teachers and students? Which skills do staff and students need for teaching and learning in adaptive environments? How can universities gain the required skills for the implementation and use of adaptive learning (e.g., train staff, hire staff, provide time)? Which changes in instructional approaches come with adaptive learning? How can the investment of personnel and technological resources for adaptive learning be justified (i.e., return on investment)? To answer these questions, it would be valuable to investigate additional cases with different socioeconomic and organisational contexts. Possibly research from organisations at later stages of the implementation of adaptive learning could help organisations at earlier stages. It would be fruitful to find general tendencies of what works best under which specific conditions to solve the challenges resulting from this study. Some experts’ statements included, for example, indications for relationships between the identified challenges. Thus, future research should focus on exploring such interdependencies between the challenges to develop a framework suggesting efficient strategies to overcome those challenges. Finally, the categories of challenges for adaptive learning from this study can serve as a framework to organise and sum up existing and future research on adaptive learning.

Adaptive learning is a promising approach to teaching with a great potential that can meet various students’ needs, provide access to education for disadvantaged groups and in disadvantaged regions, and improve quality of education by enabling personalised learning experiences at scale. The implementation of adaptive learning remains challenging. But, considering its potential advantages, the investment in extensive research into how its adoption rate can be increased will pay off, because the successful adoption of adaptive learning will carry a high yield for students, institutions and regional economy.