Redeveloping a global MOOC to be more locally relevant: design-based research
International Journal of Educational Technology in Higher Education volume 17, Article number: 9 (2020)
This study demonstrated a globally and locally relevant MOOC design model. By redeveloping a culturally responsive MOOC based on the course “Learning How to Learn (LHTL)” on the Coursera, the researchers created a sister MOOC to meet the local needs of Chinese learners. In this three-year design-based research study, four major iterations were conducted: (1) to analyze and prototype culturally responsive course contents for Chinese MOOCs learners; (2 and 3) to redevelop, pilot, field-test, and improve the new MOOC; and (4) to develop research-supported instructional practice for continued growth. This research contributed to the study of MOOCs by providing a thorough understanding of the design principles of, and theoretical claims about, the development of a culturally responsive MOOC. We hope this effort will help underpin further studies involving culturally responsive and comparable sister courses to advance MOOC research.
Online education has become more accessible and affordable to students of diverse cultural backgrounds from all around the world (Allen & Seaman, 2013). In this paper, we describe how we collaboratively localized an English language MOOC—Coursera’s “Learning How to Learn” (LHTL)—to meet the local needs of Chinese MOOCs’ learners. We introduce our use of multiple course development iterations with a design-based research approach to continuously improve the quality of the course offering. Presently, our localized (“landing”) LHTL has been offered in four major MOOC platforms in the Chinese-speaking regions with about 10,000 students.
As of September 1st, 2019, 2.4 million learners have visited the Coursera LHTL. (The complete title of the course is “Learning How to Learn: Powerful Mental Tools to Help You Master Tough Subjects”). LHTL has been one of the most popular and successful courses in the MOOC world. The goal of LHTL was to provide everyone an “easy access to the invaluable learning techniques” based on the state-of-art knowledge of neuro- and cognitive sciences. In light of the fact that students all around the world spend roughly 12 to 16 years in schools, not to mention the continuing learning needs of their careers, LHTL advocates students “learn how to learn” as an important lifelong skill. Volunteers have subtitled LHTL lecture videos in languages such as Spanish (launched January 30, 2016), Portuguese (launched April 21, 2015), simplified Chinese (launched January 30, 2016), and many others. Among those translated LHTLs, however, the course dashboards revealed participation differences. Given that the number of Chinese language users is approximately 1.5 billion, which is 2.5 times more than Spanish speakers (about 600 million) and six times more than Portuguese speakers (about 250 million), LHTL did not reach major Chinese-speaking learners. Moreover, Table 1 shows that even though the Chinese site of Coursera LHTL had highest percentage of active learners among all other three languages, those active Chinese learners did not turn into more completers. Being MOOC instructors for years, we looked for solutions to engage global learners beyond borders.
The present study is guided by the assumption: A MOOC can be cultural responsively redesigned and redeveloped for Chinese LHTL learners. We developed five design-based research objectives in this study:
Create a working circle to collaborate on prototyping culturally responsive course contents for the landing LHTL.
Pilot the course using a small “in-class” version in Taiwan.
Implement the course on one of Taiwan’s MOOC platforms, to evaluate learning activities and improve the landing course, based on learner feedback.
Encompass another three large MOOC platforms in Mainland China.
Develop research-supported instructional practice for continuing growth.
In this work, we develop design principles of the development of culturally responsive MOOCs. More specifically, we first review theoretical underpinnings, and resonate our design-based research practice. By means of a set of iterations, we then present our analyses, findings, and iterative improvements. Finally, we generalize four design principles as lessons learned in concluding remarks.
Design framework for a culturally sensitive MOOC
Development and challenge of MOOCs
After the New York Times called 2012 “The Year of the MOOC,” (Pappano, 2012) Massive Open Online Courses have gone on to underpin a new landscape for online learning. The development and characteristics of MOOC platforms vary, but MOOCs support global learners in significant ways toward social inclusion and mutual understanding (Conole, 2015). MOOCs challenge instructors and students in various ways (Firmansyah & Timmis, 2016). Instructors are exhausted with course preparation and management when enrollments scale up quickly; for students, personal responsibilities compete with individual priority of learning. Without proper support that improves motivation and encourages persistence to learn, it is unlikely that MOOC learners would commit to stay and enjoy quality learning experiences. One major criticism of MOOCs is the high attrition rate. However, many MOOCs students may be “grab-and-go” learners who only review knowledge and information they need and have no intention of earning the course certificate (Conole, 2015; Hew, 2016). Interestingly, there are no commonly available statistics related to the percentage of “textbook completers.” One might estimate the attrition rate for textbook completion to be even higher than that of MOOCs, yet there is no discussion of eliminating textbooks as a consequence. With the growth of MOOCs during the past decade, it has provided broad opportunities for researchers across disciplines. Several reviews synthesized the development of MOOC Research. MOOC researchers tended to employ desk research (Olazabalaga, Garrido, & Ruiz, 2016) and quantitative approach (Zhu, Sari, & Lee, 2018), even for case studies (Montes-Rodríguez, Martínez-Rodríguez, & Ocaña-Fernández, 2019); the majority of research focused on learner-student perspective (Deng & Benckendorff, 2017). Moreover, there exist some gaps in the literature that may limit our understanding of MOOCs, such as scarcity of instructors-focused research (Veletsianos & Shepherdson, 2015), and lack of research on cultural tensions among pedagogies, resources, and learning environments (Liyanagunawardena, Adams, & Williams, 2013).
Context-embedded online learning
Concerns have been raised over the cultural differences influencing MOOC students’ motivation and self-regulated learning (e.g., Hood, Littlejohn, & Milligan, 2015). A few research studies have explored how learners’ cultures interact with MOOCs. For example, Western MOOCs expect learners to learn independently online (Jardin & Gaisch, 2014). Zhong, Zhang, Li, and Liu (2016) further affirmed that Western MOOCs research assumes the individual as a unit of analysis. East Asian students who share Confucian traditions tend to behave cautiously in online discussion boards where instructor’s posts are greatly valued and followed. In contrast, American MOOC students are more willing to challenge and question course instructors. Other research has indicated that Western-based MOOCs share interest-based learning as the fundamental value (Jardin & Gaisch, 2014), whereas Chinese students express high interest in credit-earning or even degree-offering MOOCs (de Freitas, Morgan, & Gibson, 2015). Furthermore, synchronous sessions are thought to be necessary for Chinese online students so they feel “taught in person.” (Gunawardena, 2014). In Japan, a first-time, face-to-face meeting between teacher and online students is considered vital in building mutual trust (Lopoulos & Romero, 2010). Learners in the collectivistic society could enjoy the virtual learning environment only when they feel connected and familiar with each other (Henning, 2003). Similarly, if the course contents were highly cultural-dependent, many international online students may experience barriers when they learn by MOOC alone (Tingoy & Gulluoglu, 2012).
Indeed, culture shapes how people feel, value, think, behave, and learn. Nkuyubwatsi (2014) called on MOOC instructors to keep diversity in mind and proactively enable cultural translation in course activities. Efforts on language translations have removed some hindrances to the uptake of MOOCs in foreign settings. To reach learners globally, Coursera partners with worldwide volunteers to translate its courses so that non-English learners also can benefit from its course offerings (Bali, 2014). For example, a team of volunteers named “Borderless Education” has translated many popular MOOCs for Chinese learners. Moreover, cultural translation facilitated by local study groups (e.g., within Indonesian cities) or language-specific online forums make contents easily understandable from other cultural settings. (i.e., Coursera, 2014; Firmansyah & Timmis, 2016; Veletsianos, Collier, & Schneider, 2015). However, activities like moderating language-specific forums or study groups may create too much work and overwhelm MOOC instructors. To explore the missing element of culturally sensitive MOOC design (Andersen, Na-songkhla, Hasse, Nordin, & Norman, 2018), an expanded review in online education and education in general is needed.
Culturally sensitive course design
In face-to-face classrooms, learning difficulties can happen when teachers were insensitive about students’ diverse backgrounds (Gay, 2000). Online learners expect activities to be culturally considerate (e.g., symbol, metaphor, time zone, example, etc.; see Mohamed, Schroeder, & Wosnitza, 2014). Diverse values (e.g., lifestyles, attitude about competition, value of certificate, etc.) affect MOOCs students’ learning engagements, too (Conole, 2015). However, despite the fact that MOOCs have perhaps some of the most diverse learner populations in the world, discussion and research about developing culturally responsive MOOCs seem to be sparse (Sinclair, Boyatt, Rocks, & Joy, 2015). Students who have diverse backgrounds present challenges to online instructors, an issue that has not been properly studied and reported (Hannon & D’Netto, 2007; St. Amant, 2007). For example, Uzuner (2009) reported that while there have been a few research into understanding the online learning experiences of Asian or Asian-American students in the U.S., little research has been conducted about African-American or Hispanic-American online learners.
Gay (2000) suggested teachers incorporate students’ cultural elements into learning designs in order to help students bridge their existing experiences, habits, and communication styles to the new knowledge. For example, in the lesson of procrastination, LHTL uses zombies as the metaphor about unconscious and automated executive functions that human brains create to offload routine tasks. This metaphor is illustrative to Western learners because it borrows concepts both from the undead of European folklore and the defunct zombie process in computer programming. However, Chinese students might not make good sense of zombie because Chinese zombies are depicted as stiff corpses dressed in the official gown of the Qing Dynasty, horizontally outstretching their arms, and hopping around. Furthermore, some memory techniques are language-specific. The mnemonic technique (memorizing lists by creating a sentence in which the first letter of each word is identical to the first letter of each item on the lists) taught in LHTL is useful for learners speaking phonogram languages but not to logogram speakers (e.g. Chinese learners). Therefore, MOOC instructors should be aware of cultural diversities and design culturally sensitive online courses (Bentley, Tinney, & Chia, 2005, p.119–123; see also Edmundson, 2013; Wang & Reeves, 2007). We reasoned that, when it comes to MOOCs, simply providing translated video captions and Web pages were far from meeting non-English speakers’ learning needs. After all, learning in MOOCs is both an intellectual and emotional experience (Conole, 2015). Therefore, we undertook a more fundamental course redevelopment approach. We initiated a cross-national collaboration among four faculty members from three countries for the redevelopment (two of them were Chinese native-speaking instructors). Based in part on the “English Learning How to Learn,” this new syncretic course, which was named the “Tao of Learning (學習之道, literally “the way to learn,” TOL)” in Chinese, has been gradually rolled out on three mainland Chinese platforms and the Taiwan’s platform eWant. In the next section, the approach taken in this DBR is described with a focus on the iterative course designs and development phases.
The DBR in practice
To learn from the state-of-art practice of MOOC development, we retrieved some publicly available instructions and handbooks created by major higher education institutions whose courses mainly targeted global audiences (e.g. Scott, Kendra, & Woodgate, 2015; University of Illinois, 2013; Vanderbilt University, 2014). Partner institutions of large MOOC platforms usually provided design suggestions at the website of instructional supporting units (e.g. Center for Teaching Excellence) to potential MOOC instructors. Unfortunately, design instructions of MOOCs tacitly were based on developers’ or instructors’ lifeworlds (see also Andersen et al., 2018), and we found little information about cultural considerations in their course production phases. Design-based research approach has advantages in tackling open and innovative questions. It focuses on comprehensive solutions to continuously arising challenges via reflective iterations (Plomp & Nieveen, 2007, 2010). In the DBR process, not only was a practical solution reached in our study, but a proof-of-concept based on theoretical underpinnings was also field-tested. The following sections elaborate how our DBR was implemented via consecutive steps of exploring ideas, constructing artifacts, course offering, and evaluating artifacts, and how analysis was carried out through the four iterative development cycles.
Characterizing design-based research
Design-Based Research advocates reveal better educational solutions through continuous improvement (Plomp & Nieveen, 2007, 2010, e.g. Alharbi & Jacobsen, 2018; Feng, Lu, & Yao, 2015; Guloy, Salimi, Cukierman, & McGee Thompson, 2017; Hughes & Morrison, 2018; Stork, Zhang, & Wang, 2018). In the earlier literature of DBR, Collins (1992) pointed out the academic tendency of publishing successful experiments had misled subsequent researchers and practitioners. Collins argued that research reports should include systematic descriptions about every design’s iterations, experiment trials, evaluation of results, revision of designs, summarized design principles, and lessons learned from the study (see also Amiel & Reeves, 2008; Barab & Squire, 2004; Cobb, Confrey, DiSessa, Lehrer, & Schauble, 2003; Pool & Laubscher, 2016). A DBR carries the following characteristics (Anderson & Shattuck, 2012, pp. 16–18): (1) Being situated in a real educational context; (2) focusing on the design and testing of a significant intervention; (3) using mixed methods, involving multiple iterations; (4) involving a collaborative partnership between researchers and practitioners; (5) evolution of design principles; and (6) practical impact on practice. Recent DBR studies showed its advantages in designing and developing unique online learning experiences. For example, Alharbi and Jacobsen (2018) employed DBR at University of Calgary to develop flexible and accessible miniMOOC of graduate student supervision for faculty; Feng et al. (2015) proposed a comprehensive Distance Education Development for Adults model in China and designed a master’s program targeting distance education practitioners using two DBR iterations; Stork et al. (2018) facilitated multicultural awareness of Chinese and American university students by connecting them through Adobe Connect. These studies shared the iterative, contextual, and improvement nature of DBR (Wang & Hannafin, 2005), and we believe DBR fits our course redevelopment project, too.
Identifying tentative products and cycles of design process
By employing DBR (Design-Based Research Collective, 2003; Pool & Laubscher, 2016; Wang & Hannafin, 2005), we organized a working circle of seven people to co- and re-develop LHTL to TOL for Chinese learners. Supported by the campus production team, the working circle was able to focus on the following key development tasks for the landing TOL:
MOOCs and LHTL study group: To review and discuss literature about MOOCs and knowledge of the brain science behind LHTL.
Material collection: To gather culturally appropriate materials to be used.
Course design model: To identify and select a suitable course development model.
Pilot and evaluation: To pilot the redeveloped course modules and gather student feedback.
Associated activities: To design both online and offline learning activities.
Throughout the process of course redevelopment, we conducted progressive and iterative reviews to guide our revisions of the landing TOL. We first employed a needs analysis by outlining students’ and our expectations for the expanded course. After completing the needs analysis, we went on to design activities and write video scripts. The working circle then drafted associated presentations, after which we piloted the lectures in a face-to-face undergraduate course (Iteration 1), began filming lectures and developing course modules, and translated the English Coursera LHTL into Chinese.
After the course modules were officially produced and organized, a beta MOOC was offered in the September of 2017 (Iteration 2), and we conducted external expert reviews to evaluate creatively and critically the quality of the product and the design process (McKenney, Nieveen, & Van den Akker, 2006). In March 2018, the official landing TOL was presented on three major MOOC platforms in China (Iteration 3) and reached three types of learners (interest-based, for-credit, and self-paced). Finally, in the summer of 2018, the TOL was delivered as a general education course similar to a “College 101” in Taiwan’s universities (Iteration 4). Every action in the production of TOL, including analysis, design, development, implementation, evaluation, and revision, was guided by the idea of creating a culturally relevant curriculum (Bentley et al., 2005; Edmundson, 2013). Table 2 shows key information related to the iterations.
Data collection and analysis within iterations
Figure 1 shows the data collection and analysis along with the course iterations. Every iteration followed four development stages. The working circle members and subject matter experts explored ideas for both the content and the delivery of TOL (e.g., Confucius’ teachings). We then constructed artifacts such as quizzes, worksheets, online and offline activities, and translated video subtitles. We offered TOL in various formats and platforms. In the end of each offering, we evaluated TOL and made necessary changes for course improvement. Data generated through every iteration were analyzed, and we reflected on lessons learned in each iteration. Multiple documents, log files, student assignments, and reports were generated during the process of iterations. We conducted weekly mini-cycle analyses and end-of-course retrospective analyses among cycles in order to refine design principles in the authentic MOOC setting. In iteration 1, results from National Chiao Tung University (NCTU)’s standard course evaluation were used to ensure the appropriateness of newly developed materials. In iteration 2, both course evaluation and eWant logfiles were used to probe the outcome of the first large online rollout. In addition to descriptive statistics, cluster and classification were applied to understand potential learner patterns in TOL. In iteration 3, our analyses of logfiles continued to examine if the found learner patterns kept appearing in the following course offerings. Moreover, cross-platform surveys were developed to obtain opinions from both simplified and traditional Chinese learners. In iteration 4, while analysis of logfiles continued, a role-assignment strategy (De Wever, Schellens, Van Keer, & Valcke, 2008) was used to improve the quality of online discussion. Besides, themes of learner participation and learner gains from TOL were qualitatively analyzed via learners’ reflection papers on forum discussions. Procedures of data collection and analysis were approved by the Research Ethic Committee of NCTU (NCTU-REC-107-104). An online research information protocol (see Additional file 1) was provided to learners describing their rights as research participants at the time when they enrolled in TOL (the landing LHTL).
We have redeveloped a new, 7-week Chinese course: The Tao of Learning, based in part on the English “Learning How to Learn” MOOC. Our study reports the redevelopment iterations outlined above.
Iteration 1: small “in-class” version in Taiwan
The first iteration aimed to respond to the first and second research objectives: Create a working circle to collaborate on prototyping culturally-responsive course contents for the TOL, and then pilot the alpha course. The first iteration of TOL recruited 15 Taiwan college students. One of our researchers served as the face-to-face class instructor in teaching the modules, and his students used the designed course sites, activities, and assignments. We designed three world-café (a group dialogue method for knowledge sharing) roundtable discussions and invited two-thirds of the students to share their tips and tricks about learning in the filming studio. Midterm and final course evaluations were conducted.
Exploration of ideas
To convert LHTL from a 4-week, interest-based MOOC to a MOOC that offers college level credits in China, the course needed to be expanded to meet the minimum credit-hour requirement of these China’s online learning platforms. The working circle, which was consisted of six masters and doctoral students at NCTU, participated in the English version of Coursera LHTL as learners, studied key concepts of LHTL, and initiated a study group for the companion book “[A Mind for Numbers: How to Excel at Math and Science —Even If You Flunked Algebra]” (Oakley, 2014). A knowledge-mapping was conducted by the working circle to determine existing key ideas in the LHTL as well as new and cultural-responsive topics and resources. For example, “smart pills” have been widely discussed and can even be bought online in China; there was pseudoscience about brains, which misled parents and teachers; and poorly designed test-prep companions were sold widely in bookstores. We addressed these culturally-relevant topics in the Chinese landing TOL. Moreover, the working circle assessed contents and analogies that may not fit the needs of Chinese learners and proposed suggestions in the revision.
Construction of artifacts
We followed the successful design principles of the Coursera LHTL from the second author (Jung, Kim, Yoon, Park, & Oakley, 2018; Oakley, Poole, & Nestor, 2016): high instructor and TA quality, clear and concise content, and proper course format. Firstly, we invited Dr. Mooming Poo, the Director of the Institute of Neuroscience at the Chinese Academy of Science in Shanghai, China, to become the fourth co-instructor. The first author then organized and coached TAs to manage the TOL online communities among platforms. Secondly, we reviewed the new lecture scripts from both instructors’ and learners’ perspectives to ensure that layman language was used to communicate evidence-based learning insights. Cultural heritage such as Confucius’ teachings and ancient Chinese scholars’ stories were used wherever appropriate. Thirdly, we keep the course layout consistent throughout every week and each item was numbered such that students could locate what they needed easily.
The first author believed that the content of TOL was not only appropriate to general audiences but also was pedagogically essential for pre-service teachers. Therefore, he opened a new teacher education course and used it as the alpha course of TOL. Each new topic was a face-to-face lecture in this alpha course. Using a “Share one, get six” activity, where students used Google Forms to fill in three ideas learned and then copied one of the three ideas from other classmates right after each course session, we evaluated what students learned by those form responses and revised video scripts accordingly. To engage students in reflecting their authentic learning strategies, three world cafes were conducted, too. The three-round world cafes significantly helped prepare students to share during the focus group interviews in the film studio.
Evaluation of artifacts
Students gave average high satisfaction ratings for the alpha course (M = 4.33 out of 5, SD = 0.78) in NCTU’s standard course evaluation. Among all the survey items, students highly agreed that “the instructors had a positive attitude toward students’ learning” (M = 4.45), “the instructors taught this course properly” (M = 4.36), and “the course content was well-organized” (M = 4.27). In the open-ended questions, students frequently expressed a desire to have taken the course in their childhood, found the in-class discussions helped them clarify what they watched in the course videos, and world cafes enabled student-to-student interaction in deeper ways. They suggested a reordering of several lecture snippets. Six student-group interviews about particular learning strategies were recorded as optional videos for future Chinese students. These videos were found to be of great value in the subsequent iterations of TOLs.
Iteration 2: large online rollout on one MOOC platform in Taiwan
The second iteration aimed to respond the third research objective: Implement the beta course on one of Taiwan’s MOOC platforms, evaluate learning activities, and improve the landing course based on real MOOC learners’ feedback. Author1 again served as the lead instructor; this iteration attracted more than 1500 MOOC learners. This large enrollment (for a typical MOOC in Taiwan) provided a good opportunity for “stress testing” to ensure that the platform and course worked smoothly.
Exploration of ideas
Using a MOOC, we wanted to reach learners as varied as possible. eWant, the largest MOOCs provider at Taiwan’s NCTU, devoted staff resources to market the course both virtually and physically. In addition to interest-based online learners, TOL was accredited by the government’s in-service teachers’ professional development program (September 2017) and civil servants’ training program (January 2018). The enrolled teachers and public officials earned credit hours after successfully completing TOL. Additionally, we focused on creating standard operating procedures for instructors, TAs, and eWant course designers to facilitate TOL. Weekly engaging announcements, discussion board prompts, and course evaluation (both internal and external) were laid out in the beta offering.
Construction of artifacts
After the revision of the alpha course, the additional topics were scheduled to be filmed in Hsinchu, Taiwan, and Shanghai, China. eWant supported the filming and production in the summer of 2017. The seven-week TOL was offered in traditional Chinese (See Table 3). Moreover, eWant had a bilingual engine that allowed simplified-Chinese learners to switch and learn in the simplified-Chinese interface.
Evaluation of artifacts
Iteration 2 was productive. The course evaluation was again used and revealed 4.51 out of 5. Students showed appreciation where they could, not only for the ability to take the course with cultural-friendly content, but also for the interactions with instructors and TAs directly in Chinese. In the satisfaction survey, students responded TOL was more (45.8%) and much more (5.9%) difficult than they had expected. However, they were still satisfied (58.5%) and very satisfied (26.7%) about what they had learned in TOL. Students were also satisfied (31.4%) and very satisfied (66.1%) with the four instructors.
We leveraged the system logs to understand how Chinese students learned in TOL. The system logs revealed that 54% of the visitors watched lecture videos and 33% took quizzes. Moreover, even though this beta offering was not marketed in Mainland China, 117 learners indicated they physically located in China, which consisted of 7.5% of the total enrollees (61.3% Taiwanese, 31.2% elsewhere). We then analyzed classifiers that affected learning completion in TOL. We removed enrollees who had no learning activities (i.e., view lecture videos, read forum posts, post/reply posts, take exams, and submit assignments), and a J48 pruned tree (C4.5, Quinlan, 1993) classification (a top-down tree growth algorithm) with 10-fold cross validation was implemented. Using the 963 valid instances (Pass = 213, Fail = 750), J48 correctly classified 96.3% instances with Kappa statistic = 0.8873. TOL learners who passed the course not only attempted homework or quizzes more than 12 times, but also had following characteristics: (1) viewed course videos more than 92 times; or (2) posted more than six articles; or (3) read fewer than 24 posts. A clustering analysis using expectation-maximization (EM) algorithm explored three behavioral types of students (Log likelihood = − 15.74082): Cluster 0 (34%) - halfway give-ups; Cluster 1 (12%) - hardworking students; and Cluster 2 (55%) - windows shoppers (See Table 4) (Fig. 2).
After understanding our students via the above-mentioned analyses, we invited an external reviewer from Taiwan’s National Open University to provide professional feedback in terms of course organization. Moreover, students voluntarily provided more than a hundred errata that significantly improved the quality and accuracy of the course. Consequently, the beta TOL was selected by MOOCs learners as Taiwan’s top MOOC for Fall 2017; this award provided quality evidence for initiating new MOOC offerings at China’s three major MOOC platforms (XueTangX, NetEase, and CNMOOC).
Iteration 3: encompass three large MOOC platforms in Mainland China
The third iteration aimed to respond to the fourth objective: Encompass another three large MOOC platforms in Mainland China. Given the success of the beta offering, the China platforms were confident in adopting our landing LHTL. The primary goal in Iteration 3 was three-fold: (1) To create course sites that met different platform standards; (2) To reach interest-based, self-paced, and for-credit Chinese MOOC students; and (3) To embed data collection, whenever appropriate, for future research.
Exploration of ideas
We studied the main MOOC providers in China and chose three of them, based on their outreach, support, and reputation. XuetangX from Tsinghua University and China’s Ministry of Education, CNMOOC from Shanghai Jiao Tung University, and NetEase from Guangzhou NetEase Computer System Company, were selected to be our TOL partners. The three Chinese MOOC platforms had distinct characteristics and targeted users: XuetangX was government-supported and nicknamed “China’s Coursera,” CNMOOC selected for-credit online courses offered only by top Chinese Universities, and NetEase focused on self-renewal of working professionals. The three platforms covered the majority of Chinese online learners.
In addition to diversifying TOL Chinese learners throughout three platforms in China, we strategically mixed learners with different learning goals in eWant. With the conjunction of the second eWant offering, we tested a “MOOC-inside” model for TOL. A teacher education course “Learning and Reading Strategy” was offered at NCTU, and the first author blended this course with fully online TOL and face-to-face world cafes. As a result, we mixed diverse backgrounds of learning in the eWant offering: Interest-based MOOC learners, training program learners, and for-credit students so that they could learn from each other. After diversifying our learners among various goals and learning environments, we wanted to investigate learning experiences across platforms. However, the software development history of each MOOC platform varies, and this caused barriers for comparing students’ learning trajectories across different learning management systems. Therefore, a universal course survey for all TOL learners was necessary to both capture and differentiate their learning experiences (Table 5).
Construction of artifacts
The working circle and eWant staff explored specifications and requirements and then created course sites accordingly in the three Chinese platforms. Although switching traditional Chinese contents to simplified Chinese can be performed automatically by switch-system encodings, the actual terms and idioms geographically have been used differently. One of our working circle members who was a Chinese national conducted a second proofreading for all the simplified Chinese course sites. Moreover, three-step Qualtrics surveys were designed and conducted to understand students’ learning experiences among all four platforms for TOL. The sets of items included satisfactions, learner gains, learner self-evaluations, and anonymous learner profiles. When students proceeded to the given week, namely, in the beginning (week 0), middle (week 3), and end of the courseFootnote 1 (week 7), we invited students self-evaluating their progress and providing feedback.
Evaluation of artifacts
We conducted the same logfile analyses using the new eWant cohort (N = 462). If classifications and clustering were consistent among iterations of the same course, we would consider the metric and the observed subpopulations stable (i.e., robust to random variance in characteristics of learners and instructor facilitations). The J48 and EM revealed similar results. Students who attempted taking quizzes and submitting assignments more than 12 times were likely to pass TOL. In addition, students could be grouped also as halfway give-ups, hardworking students; and windows shoppers. In the Qualtrics course surveys, students who responded in all four platforms were strongly positive (Table 6), and felt their interest, understanding, skills, and intention to use what they learned in TOL increased compared to their prior experiences (Table 7).
Iteration 4: continued growth for both research and instruction
The fourth iteration was intended to respond to the fifth research objective: Develop research-supported instructional practice for continued growth. Reich (2015) suggested that next generation MOOC research should focus on interventions that facilitate learning. As this is an ongoing study, we continued to implement new practices and conduct evaluations among various platforms following the enactment, review, and revision steps in compliance with the DBR.
Exploration of ideas
In Iteration 4, two opportunities were presented to the TOL course team. First, TOL was invited by NetEase to test a fee-based offering in the summer. We initially wanted to explore whether Chinese students would accept taking TOL for a fee (about $15 USD) and eventually learned from the low enrollment (n = 4) that fee-based TOL may not be a plausible outreach strategy. Second, TOL was planned to be a part of the Taiwan’s Summer College for General Education. National Taiwan University selected college-level courses to be offered nationwide in summer provided students the opportunity to fast-track their college graduation. One hundred forty-seven students enrolled in this two-credit, fully online TOL, and 83% of them passed the course. Interestingly, predicting students’ learning outcomes by learning activities became less meaningful in this small cohort. J48 classification revealed that regardless of how student participated in TOL, as long as the students attempted 13 or more times in submitting quizzes or assignments, then he or she would pass the course (kappa = 0.57). Due to the fact that learner feedback from previous iterations seldom mentioned forum discussions, we therefore planned to explore strategies that might improve quality of online discussion.
Construction of artifacts
TOL at eWant was used for this summer course. One additional learning design was plugged into this entire for-credit iteration. We randomly assigned one of five discussant-roles (icebreaker, source searcher, scholar, summarizer, and moderator) to each student to play in each week. We used this fully online and for-credit offering to see whether learners felt that forum discussions facilitated learning when each student had clearer role expectations in weekly forum discussions. Students were required to submit reflection papers to review their contributions and experiences in forum discussions at the end of course.
Evaluation of artifacts
We turned our focus to analyze students’ forum discussions and reflection papers. To explore what learners experienced and how they experienced in our new learning design, we followed Patton’s (2002) qualitative analytical procedures. Learners’ discussion threads and reflection papers were separately downloaded, de-identified, and coded by every paragraphs. Discussion threads were induced and revealed common themes of learning experiences in TOL. We then seek to explain how TOL learners perceived, described, felt about, remembered, and made sense of the experience in forum participations from reflection papers. Three discussion participation themes stood out. They: (1) knew the meaning of learning more deeply, not only from instructors but, more importantly, from peer classmates; (2) realized online forums were beneficial to one’s understanding about learning; and (3) sensed an online learning community.
In reflection papers, students again stated the following four gains from TOL. First, they recognized their own contributions to peer classmates. For example, one student felt encouraged “When classmates said, ‘Hey, you truly made a point’; ‘Wow …. what you said resonates with me’; and ‘That’s very unique’, I think I am contributing to the whole class.” Moreover, another learner realized that “[My] life is unique so my stories motivated my classmates to think differently. The way I do, and I’m more than happy to do, as a summarizer was to add my own experiences to bring classmates various dimensions about a topic.” Second, learners sensed ownership of responsibility to learn. Through forum participation, they metacognitively reviewed not only their learning in TOL, but also their life experience in learning. One student stated his participation that “I won’t do sloppy canned responses. I reviewed and rethought what I posted. If I felt my posts failed to serve my assigned role for that week, I deleted and replied again. I would like to be responsible to my fellow readers.” Besides, another student shared that “TOL was the first time I overcame my mental obstacle. I found it became my own breakthrough about learning: I learn while I teach. Now I’m more and more willing to share my opinions.” Thirdly, learners experienced peer learning both inside the outside TOL. One student found knowledge correspondences between lecture videos and peer testimonials that “I witness my classmates sharing what they did to enhance learning. Their stories actually prove the mentioned learning theories. I’m more confident [with the theories] than ever.” Besides, another learner proactively “shared the ideas with my real-world friends if I learned anything from the post.” Lastly, students reflectively concluded their lessons learned in TOL. A student confessed that “the reason I enrolled in TOL was because it was an online course, and I would not need to talk in public. However, I gradually got used to talking aloud. I got over what I was afraid to do. Taking TOL was my learning milestone!” Another student turned himself to be active learner because “I witnessed my progress in both learning the course and contributing meaningful posts. These are my major gains in TOL.”
Course redevelopment is never easy; instead, it requires careful alignment among content, activities, and assessments (Chen, Bauer, Anderson, Hannah, & Provant-Robishaw, 2015; Chen, Lowenthal, & Bauer, 2015). This study reports a DBR instance for the redevelopment of LHTL to a landing (“localized”) MOOC that accommodates Chinese learners’ needs. Through DBR, this study addresses previous concerns regarding the lack of research and practice about cultural issues in MOOCs (Liyanagunawardena, Adams, & Williams, 2013; Nkuyubwatsi, 2014), and provides a thorough understanding of the design principles of, and theoretical claims about, developing a culturally responsive MOOC. Our assumption guiding this research turned to be accurate: TOL offered a culturally accessible MOOC that benefits Chinese learners. Seeing the growth of TOL, it is expected that the outcomes from this DBR can inform future MOOC instructors.
Can there be a possible third way that reconciles the giant and Western-centered MOOCs and small, locally fragmented MOOCs? TOL developed a third way: Redeveloping landing MOOCs throughout a close collaboration between the original MOOC instructors and the landing MOOC instructors. In the redevelopment of TOL, we carefully examined Chinese learners’ needs for this course, piloted our prototype in the alpha and beta offerings, and applied research-informed improvement in course iterations. At a practical level, our framework provides a structure that can help guide MOOC course redevelopment in a fashion that facilitates culturally responsive learning. In addition to reaching the five research objectives, we generalized four design principles based on the project:
Collaborating with local MOOC instructors. Designed and led by Chinese-speaking instructors and TAs, TOL is adjusted to meet the experiences of Chinese learners.
Partnering with local platforms. Partnering with locally grown LMS’s helped reach more learners. Learners enjoy cultural scaffolds provided by the design of local LMSs, their supporting technical staff, and instructors.
Developing a set of comparable MOOC sites. The knowledge provided in both LHTL and TOL are identical. We can analyze students’ learning gains across platforms based on this comparable set-up.
Embedding research with MOOC instruction. In addition to system logs, we collect researchable data that is designed to be part of the learning process in TOL.
Reviewers of MOOC research (Williams & Su, 2015) worried about the feasibility of urging MOOCs instructors to conduct rigorous research while operating courses, interacting with thousands of students, and maintaining educational quality. Rather than domain-independent, “plug-in” experiments, Reich (2015) advocated more domain-specific MOOC research to identify best teaching and learning practices that could advance the science of disciplinary learning. Zhu et al. (2018) further called upon cross-cultural comparison research to not only uncover how MOOCs paradigms differ in various regions of the world but also gain localized understanding of educational philosophies of MOOCs. By creating MOOC sister courses among different cultures, domain-specific MOOC research could be conducted among these comparable course sites. Our work is a proof-of-concept, showing that creating a learning environment that enables domain-specific MOOC research is practicable.
Educationally, we developed an ecological circle for MOOC course development. One of the reasons that MOOCs research was fundamental but not widely applied by MOOCs instructors, was due to the gap between research outcomes and educational applications. It is unlikely for MOOCs instructors to adopt a finding that was about MOOCs but has nothing to do with the course feedback or improvement. Our MOOC redevelopment model was practical and research embedded. Future MOOC instructors could use our model, not only to develop and maintain a sustainable MOOC, but also to develop potential research projects that benefit the scholarship of teaching and learning over time.
Availability of data and materials
All data generated or analyzed during this study are not publicly available due to the restriction by the Research Ethic Committee of National Chiao-Tung University (NCTU-REC-107-104).
URLs for the three publicly available surveys: Beginning: https://nctucommunication.qualtrics.com/jfe/form/SV_bD9vRF1Z4SrSYF7?Q_Language=ENMidterm: https://nctucommunication.qualtrics.com/jfe/form/SV_0Vd53g7K36fqd9z?Q_Language=ENEnd-of-course: https://nctucommunication.qualtrics.com/jfe/form/SV_b44jn9vNEMTU3Pf?Q_Language=EN
Alharbi, H., & Jacobsen, M. (2018). Evaluating the design and development of the quality graduate supervision miniMOOC. Quarterly Review of Distance Education, 19(1), 13–26.
Allen, I. E., & Seaman, J. (2013). Changing course: Ten years of tracking online education in the United States. Newburyport: Sloan Consortium.
Amiel, T., & Reeves, T. C. (2008). Design-based research and educational technology: Rethinking technology and the research agenda. Educational Technology & Society, 11(4), 29–40.
Andersen, B. L., Na-songkhla, J., Hasse, C., Nordin, N., & Norman, H. (2018). Perceptions of authority in a massive open online course: An intercultural study. International Review of Education, 64, 221–239.
Anderson, T., & Shattuck, J. (2012). Design-based research: A decade of progress in education research? Educational Researcher, 41(1), 16–25. https://doi.org/10.3102/0013189X11428813.
Bali, M. (2014). MOOC pedagogy: Gleaning good practice from existing MOOCs. MERLOT Journal of Online Learning and Teaching, 10, 44–55.
Barab, S., & Squire, K. (2004). Design-based research: Putting a stake in the ground. Journal of the Learning Sciences, 13(1), 1–14.
Bentley, J. P. H., Tinney, M. V., & Chia, B. H. (2005). Intercultural Internet-based learning: Know your audience and what it values. Educational Technology Research & Development, 53(2), 117–127.
Chen, K.-Z., Bauer, C., Anderson, J., Hannah, E., & Provant-Robishaw, C. (2015). Resolving bottlenecks: Converting three high-enrollment nursing courses to an online format. Journal of Nursing Education, 54, 404–408. https://doi.org/10.3928/01484834-20150617-09.
Chen, K.-Z., Lowenthal, P., & Bauer, C. (2015). Effectiveness and student perception of three high-enrollment health studies online courses. Health Education Journal, 74, 197–208. https://doi.org/10.1177/0017896915581060.
Cobb, P., Confrey, J., DiSessa, A., Lehrer, R., & Schauble, L. (2003). Design experiments in educational research. Educational Researcher, 32(1), 9–13.
Collins, A. (1992). Toward a design science of education. In E. Scanlon, & T. O’Shea (Eds.), New directions in educational technology, (pp. 15–22). New York: Springer-Verlag.
Conole, G. (2015). Designing effective MOOCs. Educational Media International, 52, 239–252.
Coursera (2014). New learning hubs locations hosted by the New York Public Library and seven other international partners. Retrieved from http://coursera.tumblr.com/post/84322385012/new-learning-hubs-locations-hosted-by-the-new-york
de Freitas, S. I., Morgan, J., & Gibson, D. (2015). Will MOOCs transform learning and teaching in higher education? Engagement and course retention in online learning provision. British Journal of Educational Technology, 46, 455–471.
De Wever, B., Schellens, T., Van Keer, H., & Valcke, M. (2008). Structuring asynchronous discussion groups by introducing roles: Do students act in line with assigned roles? Small Group Research, 39(6), 770–794.
Deng, R., & Benckendorff, P. (2017). A contemporary review of research methods adopted to understand students' and instructors' use of Massive Open Online Courses (MOOCs). International Journal of Information and Education Technology, 7, 601–607. https://doi.org/10.18178/ijiet.2017.7.8.939.
Design-Based Research Collective (2003). Design-based research: An emerging paradigm for educational inquiry. Educational Researcher, 32, 5–8.
Edmundson, A. (2013). Cases on cultural implications and considerations in online learning. New York: ISR.
Feng, X., Lu, G., & Yao, Z. (2015). Professional task-based curriculum development for distance education practitioners at master’s level: A design-based research. The International Review of Research in Open and Distributed Learning, 16(2), 288–310.
Firmansyah, M., & Timmis, S. (2016). Making MOOCs meaningful and locally relevant? Investigating IDCourserians—an independent, collaborative, community hub in Indonesia. Research and Practice in Technology Enhanced Learning, 11, 1–23.
Gay, G. (2000). Culturally responsive teaching: Theory, research and practice. New York: Teachers College Press.
Guloy, S., Salimi, F., Cukierman, D., & McGee Thompson, D. (2017). Insights on supporting learning during computing science and engineering students’ transition to university: a design-oriented, mixed methods exploration of instructor and student perspectives. Higher Education, 73(3), 479–497.
Gunawardena, C. N. (2014). Globalization, culture, and online distance learning. In O. Zawacki-Richter, & T. Anderson (Eds.), Online distance education: Towards a research agenda, (pp. 75–108). Edmonton: AU Press.
Hannon, J., & D’Netto, B. (2007). Cultural diversity online: Student engagement with learning technologies. International Journal of Educational Management, 21(5), 418–432.
Henning, E. (2003). I click, therefore I am (not): Is cognition distributed or is it contained in borderless e-learning programmes? International Journal of Training and Development, 7(4), 303–317.
Hew, K. F. (2016). Promoting engagement in online courses: What strategies can we learn from three highly rated MOOCS. British Journal of Educational Technology, 47, 321–341.
Hood, N., Littlejohn, A., & Milligan, C. (2015). Context counts: How learners’ contexts influence learning in a MOOC. Computers & Education, 91, 83–91.
Hughes, J., & Morrison, L. (2018). The use of e-textiles in Ontario education. The Canadian Journal of Education, 41, 356–384.
Jardin, T., & Gaisch, M. (2014). Extending the MOOCversity: A multi-layered and diversified lens for MOOC research. In EMOOCS, (pp. 73–79).
Jung, E., Kim, D., Yoon, M., Park, S., & Oakley, B. (2018). The influence of instructional design on learner control, sense of achievement, and perceived effectiveness in a supersize MOOC course. Computers & Education, 128, 388–388.
Liyanagunawardena, T. R., Adams, A. A. & Williams, S. A. (2013). MOOCs: a systematic study of the published literature 2008–2012. The International Review of Research in Open and Distance Learning, 14, 202–227.
Lopoulos, N., & Romero, M. (Eds.) (2010). Educational social software for context-aware learning: Collaborative methods and human interaction. New York: ISR.
McKenney, S., Nieveen, N., & Van den Akker, J. (2006). Design research from a curriculum perspective. In J. Van den Akker, K. Gravemeijer, S. McKenney, & N. Nieveen (Eds.), Educational design research, (pp. 62–90). London: Routledge.
Mohamed, A. M. F. Y., Schroeder, A. C. U., & Wosnitza, M. (2014). What drives a successful MOOC? An empirical examination of criteria to assure design quality of MOOCs. In The proceedings of 2014 IEEE 14thinternational conference on advanced learning technologies. Athens.
Montes-Rodríguez, R., Martínez-Rodríguez, J. B., & Ocaña-Fernández, A. (2019). Case study as a research method for analyzing moocs: Presence and characteristics of those case studies in the main scientific databases. International Review of Research in Open and Distributed Learning, 20, 59–79.
Nkuyubwatsi, B. (2014). Cultural translation in massive open online courses (MOOCs). eLearning Papers, 38, 1–10 Retrieved from https://lra.le.ac.uk/bitstream/2381/28554/1/Cultural%20translation%20in%20MOOC%20Edited.pdf.
Oakley, B. (2014). A mind for numbers: How to excel at math and science, even if you flunked algebra. New York: Penguin.
Oakley, B., Poole, D., & Nestor, M. (2016). Creating a sticky MOOC. Online Learning, 20, 1–12.
Olazabalaga, M. I., Garrido, C. C., & Ruiz, G. U. (2016). Research on MOOCs: Trends and methodologies. Monográfico, 15(7), 87–98.
Pappano, L. (2012). The year of the MOOC. Retrieved from https://www.nytimes.com/2012/11/04/education/edlife/massive-open-online-courses-are-multiplying-at-a-rapid-pace.html
Patton, M. Q. (2002). Qualitative research & evaluation methods, (3rd ed., ). Thousand Oaks: Sage.
Plomp, T., & Nieveen, N. (2007). An introduction to educational design research. In Proceedings of the Seminar Conducted at the East China Normal University, Shanghai (PR china), November 23–26, 2007. Enschede: Netherlands Institute for Curriculum Development.
Plomp, T., & Nieveen, N. (Eds.) (2010). An introduction to educational design research. Shanghai: SLO- The Netherlands Institute for Curriculum Development.
Pool, J., & Laubscher, D. (2016). Design-based research: Is this a suitable methodology for short-term projects? Educational Media International, 53(1), 42–52.
Quinlan, J. R. (1993). C4.5: Programs for machine learning. Burlington: Morgan Kaufmann.
Reich, J. (2015). Rebooting MOOC research: Improve assessment, data sharing, and experimental design. Science, 347, 34–35.
Scott, I., Kendra, L., & Woodgate, A. (2015). Edinburgh MOOCs handbook: How to grow your own MOOC with Coursera, version 1.2. Edinburgh: The University of Edinburgh https://moocs.unige.ch/files/3414/2503/3908/Growing_an_Edinburgh_MOOC.pdf.
Sinclair, J., Boyatt, R., Rocks, C., & Joy, M. (2015). Massive open online courses (MOOCs): A review of usage and evaluation. International Journal of Learning Technology, 10, 1–23.
St. Amant, K. (2007). Online education in an age of globalization: Foundational perspectives and practices for technical communication instructors and trainers. Technical Communication Quarterly, 16, 13–31.
Stork, M. G., Zhang, J., & Wang, C. X. (2018). Building multicultural awareness in university students using synchronous technology. TechTrends: Linking Research & Practice to Improve Learning, 62(1), 11–14.
Tingoy, O., & Gulluoglu, S. S. (2012). Effects of culture on distance learning in higher education in the light of digital divide and multicultural curriculum and design strategies. In Paper presented at the 2012 IV International Conference “Problems of Cybernetics and Informatics (PCI)”, (pp. 1–3). Baku: https://ieeexplore.ieee.org/document/6486275.
University of Illinois (2013). Guide to MOOCs at Illinois. Retrieved from http://moocs.illinois.edu/docs/moocs-at-illinois-guide.pdf
Uzuner, S. (2009). Questions of culture in distance learning: A research review. International Review of Research in Open and Distance Learning, 10(3), 1–19. https://doi.org/10.19173/irrodl.v10i3.690.
Vanderbilt University (2014). VU Coursera Resource Guide. Nashville: Center for Teaching, Vanderbilt University Retrieved from https://cft.vanderbilt.edu/tag/coursera/.
Veletsianos, G., Collier, A., & Schneider, E. (2015). Digging deeper into learners’ experiences in MOOCs: Participation in social networks outside of MOOCs, notetaking and contexts surrounding content consumption. British Journal of Educational Technology, 46, 570–587.
Veletsianos, G., & Shepherdson, P. (2015). Who studies MOOCs? Interdisciplinarity in MOOC research and its changes over time. International Review of Research in Open and Distributed Learning, 16, 1–17.
Wang, C., & Reeves, T. C. (2007). The meaning of culture in online education: Implications for teaching, learning and design. In A. Edmundson (Ed.), Globalized e-learning cultural challenges, (pp. 1–17). Hershey: IGI Global. https://doi.org/10.4018/978-1-59904-301-2.ch001.
Wang, F., & Hannafin, M. J. (2005). Design-based research and technology-enhanced learning environments. ETR&D, 53, 5–23. https://doi.org/10.1007/BF02504682.
Williams, V. S., & Su, N.-F. (2015). Much a MOOC about nothing: Is real research coming? International Journal on e-Learning, 14, 373–383.
Zhong, S.-H., Zhang, Q.-B., Li, Z.-P., & Liu, Y. (2016). Motivations and challenges in MOOCs with Eastern insights. International Journal of Information and Education Technology, 6, 954–960.
Zhu, M., Sari, A., & Lee, M. M. (2018). A systematic review of research methods and topics of the empirical MOOC literature (2014–2016). The Internet and Higher Education, 37, 31–39.
The authors would like to thank the anonymous reviewers for their thoughtful comments on this work.
This work was supported by Taiwan’s Ministry of Science and Technology [grant numbers MOST 106–2511-S-009-004- and MOST 107–2511-H-009-006-MY2]. The funding agency had no involvement in this research.
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Chen, KZ., Oakley, B. Redeveloping a global MOOC to be more locally relevant: design-based research. Int J Educ Technol High Educ 17, 9 (2020). https://doi.org/10.1186/s41239-020-0178-6
- Mass open online courses (MOOCs)
- Course redevelopment
- Design-based research