Evaluating a blended course for Japanese learners of English: why Quality Matters
International Journal of Educational Technology in Higher Education volume 16, Article number: 6 (2019)
With the increase in the number of blended courses in higher education over the past few years, it is now increasingly important to assure the quality of online education by evaluating blended (or hybrid) courses from various aspects such as consistency of learning objectives with assessment, learning materials and technology used, and pedagogical and technical support among others. This article describes the evaluation phase of a blended course of English for general academic purposes (EGAP) targeting second-year undergraduate Japanese students mostly from the Faculties of Law, Letters, Economics, and Human Sciences at Osaka University. In this study, the researchers adopted an inclusive approach to blended course evaluation. In order to ensure the course quality from the outset, the Fifth Edition of Quality Matters Higher Education Course Design Rubric was utilized as the major reference. As part of the evaluation process, students’ perception on the usefulness of the course was also measured quantitatively and qualitatively through an attitudinal survey instrument and open ended reflection questions. Eventually, to add an outsider positionality, the blended course was peer-reviewed by a certified reviewer from Quality Matters (QM) after having been self-reviewed by the researchers. The findings of the evaluation survey demonstrated students’ overall satisfaction with the course, and their responses to the open-ended questions provided further insight into the educational and technical difficulties they encountered. The QM peer review also yielded a score of 70 out of 99, resulting in failure to meet the essential standards. However, comments from the peer reviewer guided the refinements and improvement of the course design, and the course currently meets all the requirements of the Higher Education Course Design Rubric (Fifth Edition) upon amendment. This study discusses the implications for design, development, and evaluation of English-as-a-foreign-language blended courses and provides practical tips for online/blended course designers. Further refinements to the existing course can be made in the future by implementing it several times with various groups of students.
Blended learning, also known as hybrid learning, has been a buzz word in higher education over the past two decades. However, there is not much consensus over its definition. Graham (2006) observes that there is a vast array of responses to the question of “what is being blended?”, and categorizes the most commonly mentioned definitions into the following three groups.
"Combining instructional modalities (or delivery media)
Combining instructional methods
Combining online and face-to-face instruction" (p. 3).
According to Graham (2006), the third definition, which recognizes blended learning as a combination of face-to-face and computer-mediated instruction, captures the essence of the concept more accurately than the other two. He maintains that in the past, traditional face-to-face learning environments have remained separate from computer-mediated learning environments. Nevertheless, thanks to the emergence of technological innovations, face-to-face and computer-mediated environments have started to merge.
Osguthorpe and Graham (2003) state five benefits for the adoption of blended learning, which are as follows:
Blended learning approaches allow faculty to use class time more effectively in order to improve student learning.
Access to knowledge:
Blended environments increase students’ access to information in ways that are impossible to accomplish with textbooks.
Blended learning makes it possible for students to interact with their peers and teachers both in class and online, whereas such interactions are limited in purely online courses.
Blended delivery systems enhance learner control and give them choices as to what to study and how to study it.
Blended environments reduce time spent in class and increase an institution’s intake of tuition-paying students. They also lower expenses by reducing the costs of employing full-time faculty.
There has been a growing interest in blended learning in higher education (Garrison & Kanuta, 2004; Garrison & Vaughan, 2008). The most commonly recognized reason within the literature as to the popularity of blended learning is that it is believed to combines the best of both worlds by mixing effective elements from face-to-face and online learning, given that it is well designed. The field of language learning and teaching has also been abundant with studies on blended learning, 24 of which Grgurovic (2017) has reviewed in five categories, (1) comparative studies, (2) teacher perceptions, (3) learner perceptions, (4) technology tools studies, and (5) course implementation. She concludes that blended learning will most probably remain the preferred approach in ELT (English Language Teaching) in the years to come.
The number of blended courses in higher education across Japan is also increasing and expected to grow rapidly (Gruba & Hinkelman, 2012). According to McCarty, Sato, and Obari (2017), blended learning, in the Japanese context, can provide a more integrated approach for teaching and learning, prevent learner alienation, improve completion rates, decrease dropout rates, and enhance the quality and quantity of interaction. The authors cite case studies conducted at Japanese universities on the potential effectiveness of blended learning and flipped classroom. Their findings indicate that compared to traditional methods, these approaches are more effective in improving students’ proficiency levels. As the prevalence of blended instruction has expanded exponentially in recent years, so has the necessity to evaluate the quality of blended courses; hence, quality evaluation is of paramount importance in blended course design, development, and delivery (Gruba, Cárdenas-Claros, Suvorov, & Rick, 2016).
There are numerous checklists, guidelines, and rubrics for online and blended course design developed by several organizations (e.g., Quality Matters, United States Distance Learning Association, the Illinois Quality Online Course Initiative Rubric, the North American Council for Online Learning, the Council for Higher Education Accreditation, the Higher Learning Commission, Blackboard Inc.), many of which mainly focus on the instructional design and development of the course, not on the course delivery and instruction (Piña & Bohn, 2014). Among these checklists, guidelines, and rubrics, the researchers have opted for the Quality Matters Higher Education Course Design Rubric as the main frame of reference. Furthermore, they have adopted an inclusive approach to blended course evaluation. The basic Successive Approximation Model proposed by Allen (2012) informed the design and development of a blended course of English for general academic purposes (EGAP) mainly targeting second-year undergraduate Japanese students from the Faculties of Law, Letters, Economics, and Human Sciences at Osaka University. In order to ensure the course quality from the outset, besides the Standards Checklist (Vai & Sosulski, 2011), the Fifth Edition of Quality Matters Higher Education Course Design Rubric (Quality Matters, 2014) was also utilized as the major reference. As part of the evaluation process, students’ perception on the usefulness of the course was measured quantitatively and qualitatively through an attitudinal survey instrument and open-ended reflection questions. Eventually, to add an outsider positionality, the blended course was peer-reviewed by a certified reviewer affiliated with Quality Matters—assigned by QM due to the match between her field of expertise and the course content as well as her familiarity with Japanese learners—after having been self-reviewed by the researchers.
Course design, development, and delivery revisited
As already indicated, this paper is a report on the evaluation process of a blended course of EGAP, titled Osaka University Global English Online (OUGEO). However, to give the readers a brief background on the previous stages of course design and development, the instructional design, checklist, and rubric utilized in those phases will be shortly explained, and reference will be made to other works within the literature which have made use of similar resources for online or blended course design.
Basic successive approximation model
The agile Successive Approximation Model (SAM) that benefits from iterative design processes was introduced by Allen (2012) as a replacement for the traditional Analysis, Design, Development, Implementation, and Evaluation (ADDIE) model. Unlike ADDIE’s five giant sequential steps, SAM consists of repeated, interwoven small steps, technically called iterations.
To create the blended course in this study, the basic Successive Approximation Model, known as SAM 1, was chosen as the guiding instructional design model, which is suitable for small projects that do not require specialized technical skills such as software programming (see Fig. 1).
The iterative nature of SAM 1 allowed for continuous evaluation, and consequently, for corrections, adaptations, mitigations, refinements, and adjustments at the early phases of the blended course design and development (see Mehran, Alizadeh, Koguchi, & Takemura, 2017 for more detailed descriptions of the design and development phases).
The standards checklist
As remarked by online course designers and developers (e.g., Stavredes & Herder, 2014), standards ensure consistency and overall quality throughout the course design and development. Having taken a thoughtfully designed, research-focused, practice-oriented, step-by-step approach to online course design and development, Vai and Sosulski (2011, pp. 189-195) presented a checklist that serves as a standards index and best-practices model for course designers and instructors to consistently use and to reflectively self-evaluate their online courses. This checklist guided the design and development phases of the current study and provided opportunities to iteratively do reflective self-evaluations of the created blended course.
Quality Matters rubric
What is it and why quality Matters?
Quality Matters (abbreviated as QM henceforth) started with this question raised by a small group of colleagues in the MarylandOnline consortium based in the USA: How is the quality of an online course measured and guaranteed? QM is now an international organization that is recognized as a leader in quality assurance for online education in both K-12 and higher education, and aims to promote and improve the quality of online education and student learning nationally and internationally through a variety of ways such as developing research-informed, and practice-based quality rubrics and standards, providing professional development in the use of evaluation tools to improve the quality of online education, and offering peer review and certification of quality in online education. As mentioned by Wise and Im (2015), QM has been adopted by many educational institutions to review and assess the quality of their online and blended courses.
Applying quality Matters
While the QM rubric is not so well-known in the realm of foreign language education, it has been widely used and applied to different programs of disciplines, and some of them are briefly explained below.
In her case study, Harkness (2015) documents the results of five academic years of the strategic application of QM to online learning programs at the University of the District of Columbia leading to the establishment of sustainable online education at this institution; for example, passing course grades of A-D increased 19.7%, failing course grades of F decreased 66.6%, and withdrawals from online courses reduced by 23.5%. Hollowell, Brooks, and Anderson (2017) also describe how QM helped their institution, North Carolina Central University (NCCU), address the increasing rates of Ds, Fs, and withdrawal by students enrolling in online courses.
Martin, Ndoye, and Wilkins (2016) examine how QM standards guide the identification and analysis of learning analytics data, which is “the interpretation of a wide range of data produced by and gathered on behalf of students in order to assess academic progress, predict future performance, and spot potential issues” (Johnson, Smith, Willis, Levine, & Haywood, 2011, p. 28), to monitor and improve learning in a fully online master’s program in Instructional Systems Technology at a university in the USA. The study provides a framework which helps instructors see whether their online courses meet the QM standards requirements and consequently enhance the effectiveness of online teaching and learning.
According to Dietz-Uhler, Fisher, and Han (2007), retention rates are reported to be lower in online classes than in face-to-face ones. They thus investigate whether online course design promotes student retention, using QM to design and review their psychology and statistics online courses. They reported that their retention rate over multiple offerings of both courses is roughly 95%.
Lowenthal and Hodges (2015) use QM to evaluate the quality of six randomly selected MOOCs (Massive Open Online Courses). Three trained QM peer reviewers analyzed each of the MOOCs using the QM 2011–2013 rubric. Some of the MOOCs scored very well and, with some minor revisions, two of the MOOCs could pass a QM review and, therefore, be considered high quality online courses. This suggests that MOOCs have the potential to be high quality online courses, at least in terms of course design.
Kwon, DiSilvestro, and Treff (2017) utilize the QM standards and they identify strengths as well as weaknesses of their graduate online adult education program. The results revealed that the adult online graduate courses fulfilled the key components of QM standards in general. Moreover, students’ evaluations of the courses were quite consistent with the peer instructors’ evaluations, and areas identified as needing improvement were information about accessibility, technical support, and course orientation, and descriptions of instructional materials.
The authors have found one study within the literature which has investigated the use of the QM rubric within an EFL setting. In his study, Al Zumor (2015) scrutinizes the standards of the QM rubric, 2011–2013 Edition. The findings indicated that the rubric has the potential for enhancing online foreign language education in general and can in particular make EFL learning process more humanized by increasing the instructors’ and learners’ sense of online presence. Similarly, in the present study, the QM rubric has been utilized as the major reference to evaluate the blended course of EGAP.
It is worth noting that the Quality Matters Research (QMR) is a term which comprises research that supports the QM rubric and process, discusses its use, and focuses on its impact. Readers are referred to the curated resources on QMR (https://www.qualitymatters.org/research/curated-research-resources) where they can find more theoretical and practical studies on QM.
Participants and case description
A total of 86 undergraduate students from Osaka University were enrolled in a blended course of EGAP designed and developed by the researchers. The majority (N = 83) of the students were from the Faculties of Law, Letters, Economics, and Human Sciences, whereas only three were from science and engineering backgrounds. Fifty-six percent (N = 48) of the enrollees were males and 44% (N = 38) were females. Most of them (N = 75) were in their second year, while there were seven junior and four senior students. There is little data available on the proficiency levels of the participants, but to give the readers a general idea, the results of a TOEFL ITP Test administered in 2012 to all freshmen at Osaka University revealed that on average the students scored 479.73 with 677 as the highest and 330 as the lowest scores (Kimura & Mori, 2013). The participants of the current study, however, took the British Council online placement test (available free of charge at https://learnenglish.britishcouncil.org/content) at the beginning of the semester with an average score of 71.07, a maximum score of 96, and a minimum score of 54 out of a full score of 100. According to the British Council, learners with a score of around 70 fall into A2 or B1 CEFR (Common European Framework of Reference) levels.
The blended course, officially titled “Practical English (e-learning)”, was first offered in 2012 with the aim of helping university students improve their academic English proficiency, getting them prepared for studying in English-speaking countries, and enabling them to gain a score of 490 to 520 on the TOEFL ITP®. The students would typically go through 12 weeks of online self-study using a commercial package called Linc English and an online library of video lessons known as English Central. Although one of the course objectives was to get the students prepared for study-abroad programs, it did not sufficiently include practice on language production in spoken and written forms and mostly focused on receptive skills.
In an attempt to enhance the back-then-existing course, the researchers designed and developed a new blended course to replace the old one, already referred to as Osaka University Global English Online (OUGEO). OUGEO aimed at developing students’ practical English language skills, in particular speaking, in an integrated way so that they could advance to higher levels of conversational and general academic English (up to B2 and C1 levels on the Common European Framework of Reference for Languages), as well as gain skill and confidence when speaking. The course was offered at three levels to accommodate for different proficiency groups. It started with a face-to-face orientation session, during which the students were introduced to the course and were informed about the course schedule, requirements, access to online materials, assignment submission, grading policy, etc. In total, there were five face-to-face and ten online study sessions. The online component of the course was hosted on the Osaka University learning management system, Blackboard Learn, locally known as CLE (Collaboration and Learning Environment). Details about the design, development, implementation, and evaluation of the course have been documented at OUGEO: Behind the Scenes webpage (https://sites.google.com/view/ougeo) since the main course webpage is only accessible to those with an Osaka University personal ID and password.
As recommended by SAM 1, evaluation is an indispensable component in the course design and development cycle. In order to evaluate the quality of the blended course, the following instruments were utilized: (1) QM self and peer review, and (2) a course evaluation questionnaire.
Quality Matters self and peer review
The Fifth Edition of the QM Higher Education Course Design Rubric (Quality Matters, 2014) was accessed and used via a paid institutional subscription due to two main reasons: QM is research-supported (Legon, 2006, 2015) and recommended by online course design experts (e.g., Boettcher & Conrad, 2010; Ko & Rossen, 2010), and the rubric is flexible to be used to evaluate the design and development of both online and blended courses. It consists of a set of eight general standards and 43 specific review standards to gauge the quality of online or blended courses. Annotations explain the applications of the standards and their interconnectedness. The rubric has a weighted scoring system used by the review team to determine whether a course meets the standards. Standards with three-point values are considered essential, and all must be satisfied for a course to meet the QM standards overall. It is worth noting that a minimum score of 84 out of 99 (nearly 85%) is required for a course to be QM-certified. The eight general standards of the rubric are listed below.
Course Overview and Introduction
Learning Objectives (Competencies)
Assessment and Measurement
Learning Activities and Learner Interaction
Accessibility and Usability
The non-annotated version of the rubric is available for free on the QA (Quality Assurance) Resources section of the website. It is worth mentioning that the fifth edition of the rubric had been available until July 1, 2018 before the sixth edition was released. The current link, thus, takes users to the most recent version of the Higher Ed Rubric, i.e. the sixth edition, instead of the fifth (available in Additional file 1: Appendix B) which was the most recent version available during the course review and was thus utilized in this study. The new edition features the same general standards, yet there are some modifications made to sub-standards, with the total score changed from 99 to 100.
There are several QM review types ranging from self-review to official course review. In the present study, the self-review tool was used to informally evaluate the quality of the designed blended course. Self-reviews are confidential, and the reports are not available to anybody except for the individual conducting the review. A preparatory review was then selected to benchmark the course. This paid review is an informal review process carried out by a master reviewer who is also a content expert to determine if a course has met QM standards, which results in a report that provides insight on where to focus course improvements—specific areas not meeting QM standards, for example—and can help highlight professional development needs. Figure 2 adapted from Adair (2014) summarizes the QM quality assurance process.
Course evaluation questionnaire
Despite being a comprehensive rubric for online or blended course design, the QM rubric is in fact not capable of detecting problems that are likely to occur during the course implementation such as potential technical glitches. An evaluation questionnaire was, therefore, administered to the students over the last week of the course so as to quantitatively and qualitatively measure their satisfaction with blended instruction and to identify areas in need of improvement. The questionnaire was adapted from Harker and Koutsantoni (2005) who evaluated the effectiveness of a web-based program for learning English for academic purposes. The adapted version of the questionnaire includes 81 Likert-type items followed by several open-ended questions all translated into Japanese (refer to Additional file 2: Appendix A for a bilingual version). The course evaluation questionnaire was responded by 71 students, 37 males and 34 females, out of a total of 86 enrollees on a voluntary basis. In this paper, responses to items 1–10 and 41–47 will be analyzed and reported since these items reflect the respondents’ general evaluation of the course. Harker and Koutsantoni (2005) have not provided any information regarding the reliability of the questionnaire they have developed. Nonetheless, the authors of the current study have analyzed the reliability of the questionnaire by running Cronbach’s alpha on data from items 1–10 and 41–47. These two groups of items were separately tested for reliability since the first ten items had a four-level scale (strongly disagree, disagree, agree, strongly agree), whereas the rest of the items had been created on a three-level scale (no, to a certain degree, yes). The questionnaire was found to be internally consistent and reliable, with no items needing to be deleted (Cronbach’s alpha = .80 for both groups of items). Since all the items yielded a Cronbach’s alpha value above .70, the questionnaire can be considered reliable (Pallant, 2001).
The researchers had a thorough look at Vai and Sosulski (2011) checklist and the QM annotated rubric before embarking on designing and developing the course while attempting to take into account as many standards as possible. After the course was implemented, a self-review was conducted using the worksheet available on the QM Course Review Management System. The self-review was a reflective aid to facilitate making further amendments to the course before proceeding to the peer review. For the preparatory peer review, a QM-certified reviewer was given guest access to the course to both score it and give comments on the areas in need of amelioration. It is worth mentioning that in order to become a QM-certified reviewer, all applicants are required to complete an asynchronous online course titled “Higher Ed Peer Reviewer Course (PRC)” over a period of three weeks. The first round of review yielded a score of 70 out of 99, insufficient to meet the standards. The course was later revised based on the comments of the peer reviewer and a second application for review was started. As already mentioned, a copy of the non-annotated version of the rubric based on which this review had been carried out can be found in Additional file 1 :Appendix B.
Moreover, the evaluation questionnaire was created online and distributed to the students via CLE. Since this questionnaire is quite lengthy and analyzing responses to all the items is beyond the scope of this paper, only data from items which asked the participants to evaluate the course in general will be considered for analysis. Those items are in bold within Additional file 2: Appendix A.
QM review: Round 1
The first round of the QM peer review yielded a score of 70 out of 99, meaning the course did not meet the QM standards. The researchers then revised the course in accordance with the reviewer’s comments and suggestions. Table 1 contains a list of the six essential sub-standards which were not initially met. It is worth mentioning that STANDARD 3.3 was evaluated as “not met” although the course included rubrics for scoring speaking and writing assignments. This is mainly due to the fact that the rubrics on CLE are visible only once users attempt at submitting an assignment, and therefore the reviewer failed to notice them. This fact was mentioned in the amendment worksheet and was addressed during the second round of review.
QM review: Round 2
After making amendments to the course in accordance with the recommendations of the QM peer reviewer, the course was reviewed once again by the same reviewer, and it currently meets all the requirements of the Higher Education Course Design Rubric (Fifth Edition) with a score of 99/99. More details on the problems found with the course and the ways in which the reviewer’s comments were addressed are explicated below.
STANDARD 1.1 Instructions make clear how to get started and where to find various course components
According to the reviewer, the instructions were available, but they were not readily seen by the students. To address this issue, a welcome page was created and set as the course entry page, in which information about navigating the course menu and content was provided through written instructions and screenshots. Figure 3 displays a screenshot of the course homepage including a welcome message and instructions on website navigation.
STANDARD 2.4 The relationship between learning objectives or competencies and course activities is clearly stated
Previously, the course activities were not clearly linked to the course objectives and learning outcomes mentioned in the syllabus. The connection was clarified by assigning each type of activity to the corresponding learning outcome in the syllabus. Table 2 is an instance of the connection established between the learning outcomes and learning activities associated with each of the four language skills.
STANDARD 3.3 Specific and descriptive criteria are provided for the evaluation of learners’ work and are tied to the course grading policy
As already stated, this standard was met by sharing the evaluation rubrics for speaking and writing tasks with the students. Every writing and speaking assignment included a link to its respective rubric in order to assure consistency in evaluating and scoring students’ work. These rubrics were not immediately visible to guest viewers, and therefore the course was evaluated as lacking this essential component. However, the score for this standard was restored during the second round of review through writing a note to the reviewer on the amendment worksheet.
STANDARD 5.3 The instructor’s plan for classroom response time and feedback on assignments is clearly stated
This shortcoming was rectified by adding a new section to the syllabus titled “Response Time and Feedback Schedule” in which a rough schedule was provided for responding to inquiry emails and grading assignments.
STANDARD 7.2 Course instructions articulate or link to the institution’s accessibility policies and services
To address this issue, a new link was added to the course menu through which students could access a page containing information on Osaka University institutional policies for each faculty.
STANDARD 8.2 Information is provided about the accessibility of all technologies required in the course
In order to meet this standard, a new page was created which contained information on the technologies required in the course, for instance a computer with a standard browser, and links were provided to the accessibility pages of the websites introduced to the students, for instance Blackboard Inc. (http://www.blackboard.com/accessibility.html).
By making revisions in accordance with the reviewer’s comments, the course was evaluated as meeting all the standards after amendment.
The evaluation questionnaire
The evaluation questionnaire asked the participants to evaluate the course content and website as well as write any comments or suggestions they had for improving the course. Table 3 displays the students’ responses to items 1 through 10.
It is evident that in general, the students had a relatively high opinion of the course website; however, they rated item 4 as the lowest since during the semester, there were technical issues regarding the submission of videos on the website, and many students had difficulty uploading their video speaking assignments to CLE. Some of the students also believed that the website was not mobile-friendly and that the audio files were occasionally low in sound quality. Regarding task difficulty (item 9), there were various opinions. Some respondents desired for more challenging reading tasks, rating the current reading passages as too short and easy. Others believed that the speaking tasks were extremely difficult and time-consuming.
The students also evaluated the course by responding to the seven items displayed in Table 4. The responses to these items equally indicate that the students had a rather positive attitude toward the course despite the occasional technical difficulties caused by the malfunctioning of the learning management system.
Students’ responses to the open-ended questions
The content analysis of the students’ responses to the open-ended questions revealed their overall satisfaction with the course. The students were content with the integration of four language skills, i.e., reading, listening, speaking, and writing. They believed that the integrated approach was well-balanced, and they were pleased to have the opportunity to speak and write in English as they reported that the productive skills had been overlooked in their previous English courses. Here are two comments about the course in general:
I am very satisfied with this course, as it gave me the ability to improve my English in an interactive and productive way.
Before taking this course, I had written only 70-word paragraphs in English, and I had few opportunities to speak English, but in this course I had the chance to write 400-word essays and give 4-min presentations in English.
A number of students pointed out that offering the course at three levels provided them with the opportunity to learn English at their own level. The students also commented on the poster presentations they gave at the face-to-face sessions. They were trained how to use an Augmented Reality (AR) app, Blippar, to overlay a video on their posters (see Alizadeh, Mehran, Koguchi, & Takemura, 2017 for more details). They found the activity fun, interesting, and engaging, and they stated that they enjoyed the group work. One of the students said:
I think it is a fun and innovative way of learning, and it provides access to more content outside of regular classroom materials.
Some of the students remarked that the reading and listening topics were interesting to them especially because of their recency and relevance to global issues. The TED talks also interested the majority of students. Moreover, they were satisfied with instructor presence and responsiveness as they found it easy and quick to communicate with the instructor and teaching assistants. One of the students stated:
I hope that more Japanese people will be able to use English to show the charms of Japan to the world, argue their opinions, listen to others’ opinions, and to interact with them. Instead of leaving it to translators, one should be able to express their opinion in their own words and directly understand their conversation partners speaking in English. I strongly hope that with classes like this one, which strengthen all our four skills in English, there will be more internationalized Japanese people.
Another aspect of the course that the students felt satisfied with was the feedback they received on their speaking and writing tasks. They said that the feedback was polite, easy to understand, and accessible at any time especially on their mobile devices. More comments are as follows:
I found the feedback given on my assignments clear and constructive.
I was able to see the grammar mistakes I made, which helped me recognise the gaps I have in my knowledge of English.
Unlike feedback on paper, we can look back on the feedback whenever we want to and we do not have to worry about misplacing the feedback paper. We can review our mistakes at any time.
I am not so confident about my speaking and writing skills, but pointing out to the strengths of my assignments and the points needing improvement have given me a new sense of self-confidence.
It is worth mentioning that a few students preferred to receive face-to-face feedback on their speaking tasks. A student elaborated on the reason:
In general, it [the feedback] was good, but I wish I could get face-to-face feedback on the speaking assignments. That is because it is easier to immediately understand the problem, correct myself, and receive feedback again.
Notwithstanding, some of the students found the online environment less threatening which enabled them to express their thoughts more confidently in English. Below is a comment made by one of the students that summarizes her opinion about the online, individual submission of speaking tasks:
I am not confident enough to speak in the presence of others, but since the speaking assignments were submitted online, I was able to express myself freely.
One student did not feel confident filming himself; however, he was satisfied with the feedback he was provided with. He wrote:
To be honest, I was not confident to show my face in the videos and felt embarrassed to do so, but I felt that receiving advice on my assignments was easy. There was no ambiguity, and in my opinion the quality was high.
In addition, the students stated that the online course allowed them to learn at their own pace, anywhere, anytime, using mobile devices. The students found the weekly instructions, “Read Me First” in both Japanese and English, helpful. They also believed that structural format of the course folders was easy to navigate. Last but not least, some students wrote that they felt their English had improved.
Regarding the difficulties that the students encountered in the course, failing to upload the speaking assignments’ videos on CLE was mentioned by many students. One of the students suggested that the speaking assignments could have been submitted in audio format. This technical problem caused delay in sending feedback to the students and a number of students said that the feedback should have been provided more quickly. The students asserted that they were satisfied with the help they received to solve this problem and despite having difficulty in uploading the videos online, they could submit their videos face-to-face.
Some of the students reported the low sound quality of few listening audios. It should be noted that for some students the content of the course was too easy, while for some was too demanding due to weekly writing and speaking tasks. One student also mentioned that he was not informed of the feedback as CLE does not send notifications to the users when they receive feedback on the tasks. Lastly, CLE does not have spell and grammar checker which made the writing tasks challenging for a few students and they preferred to use Microsoft Word.
This paper reports on a study conducted at Osaka University which involves the design, development, delivery, and evaluation of a blended course of EGAP, referred to as Osaka University Global English Online (OUGEO). The course was peer-reviewed using the Quality Matters Higher Education Course Design Rubric (Fifth Edition), and it currently meets all the standards of this rubric upon amendment. The findings of the evaluation phase also indicate that despite the occasional technical problems, the majority of the students felt content with the course and believed that it met their language needs and helped them improve their English skills.
This study underlines the significance of continuous improvement in online/blended course design and development. The QM peer review has aided in improving the course design and development process in light of establishing clear links between learning objectives and learning activities as well as bringing more ease and convenience to students in course navigation. The course needs to be rerun before more conclusions can be drawn on the effectiveness of the changes made; however, the literature on the application of QM to online/blended learning programs—studies such as Harkness (2015) and Hollowell et al. (2017)—bears sufficient evidence to the effectiveness of the QM rubrics and peer review in assuring excellence in online/blended learning programs.
Not only does quality assurance in online/blended learning rely on scrupulous attention to design and development, but it is also related to students’ level of satisfaction with their online experience. Young and Norgard (2006) have identified several factors contributing to student satisfaction with online instruction. The factors include interaction among students and between student and professor, consistency in course design, provision of technical support, and flexibility of online courses, each of which will be discussed here.
Regarding interaction among students, the students enrolled in OUGEO were connected with their classmates either via the online discussion boards or other communication tools such as LINE (a freeware app for instant communications developed by Line Corporation, Ltd.) for a term project entailing poster presentations. The bulk of this interaction was related to communicating with team members in order to prepare for the poster presentation; however, the students also occasionally asked and answered each other’s questions related to the course content or technical problems. More details on this phase of the study can be found in Alizadeh, Mehran, Koguchi, and Takemura, (2017) They were also in touch with the instructor and teaching assistants via email and discussion boards. A constant attempt was made to respond to student inquiries as soon as possible, the majority of which were related to submitting speaking assignments. As some studies (e.g., Rush, 2015) have shown, lack of connection, interaction, and responsiveness in online courses can make students feel isolated and disconnected.
According to studies on blended learning experiences (e.g., Tuapawa, 2016), inconsistency in online course design can cause frustration among students. The course design in the present study was consistent in that all the contents were classified based on proficiency level week by week and were saved into distinct folders for listening, speaking, reading, writing, pronunciation, etc. In addition, there were clear instructions on study materials and assignments for each week provided in English and Japanese. Moreover, the face-to-face orientation session contributed greatly to the course consistency.
As Young and Norgard (2006) remarked, technical assistance is vital to satisfaction with online courses, and studies (e.g., Yang & Cornelius, 2004; Zeng & Perris, 2004) have reported that limited technical support can lead to students’ dissatisfaction with online courses. In this study, technical support was provided by creating a shared folder on Google Drive (a file storage and synchronization service developed by Google LLC) where students were able to upload their speaking videos in case they could not upload it to CLE. If it was impossible for a student to submit their video online, neither on CLE nor on Google Drive, an appointment was made to meet them face to face and receive the video file directly through AirDrop (an ad-hoc service in Apple Inc.’s macOS and iOS operating systems) or on a USB Drive.
Finally, with regards to flexibility, the students were given one week’s time to complete the online study portion and assignments for each week and they had to attend face-to-face classes only five times out of a total of fifteen weeks. Given all this and also regarding students’ positive responses to item 44, it is evident that the course was sufficiently flexible in comparison to traditional language classes. Flexibility is in fact the reason for greater satisfaction with learning online as reported in Romero and Barbera (2011) and Pardo-Gonzalez (2013).
A major challenge with implementing this course was the large number of enrollees, which translated into a large burden for the teacher and teaching assistants in view of dealing with technical problems due to insufficient manpower. In a study exploring learners’ perceptions on the usefulness of a blended EFL program, Kobayashi and Little (2011) have found that the interface of the online component is a determining factor correlated with students’ satisfaction with such programs. Online learner satisfaction has been demonstrated to be in close relation to the operability of the technology deployed. In case of OUGEO, the submission of speaking assignments caused problems for some students which was partially resolved by providing alternative ways for submission as explained above. The dissatisfaction with the submission of videos was also reflected in the students’ responses to the questionnaire as well as in their written comments. This technical issue should be resolved before rerunning the course.
Lessons learned and advice
Here are some lessons learned during the design, development, and delivery of the current blended course:
⎕ Be ready to change – Designing and developing an online/blended course is an ongoing process. It requires constant evaluation and reflection so as to improve future courses. In fact, the ability to make changes is one of the merits of online courses.
⎕ Do not forget about open educational resources (OER) – Instead of constantly reinventing the wheel, look for freely available resources. It not only saves you a tremendous amount of time but also adds more variety to your course.
⎕ Consider time demands – Developing effective online resources is often much more time-consuming than creating classroom learning materials. Be prepared to invest time and energy into this lengthy yet valuable process.
⎕ Always keep your course objectives in mind – Your objectives are the core component leading all your actions and decisions. Make sure they are well-aligned with your learning activities and assessment.
⎕ Check for course organization and navigation – No matter how professionally you have developed and compiled your online resources, they will not be effective as long as they are not well-organized. Make sure your course is clearly organized and easy to navigate. Also, take measures to enhance screen readability and responsive design.
⎕ Be clear as to what your requirements are – Be explicit in communicating your expectations to your students. Tell them clearly what your requirements are with respect to interaction with instructor, peers, and course content.
⎕ Set evaluation criteria – Provide clear-cut criteria for how students’ work will be assessed. Inform the students of your grading policy and any rubrics you utilize for evaluating their assignments.
⎕ Care about course accessibility and usability – Ensure that the course is accessible and usable for all the students. Include information on accessibility support as well as technical and academic support services provided by your institution.
⎕ Foster social presence – An easy way to create a sense of social presence in your course is to allow the students to build a learning community with their peers through simple activities such as introducing themselves to the class.
⎕ Be ready to deal with technical glitches – No matter how hard you have attempted at designing and developing your course, there are things that will not work occasionally or constantly. Think of alternative solutions to deal with technical difficulties and ask technical staff for help.
Limitations and implications
This evaluation report is based on the data collected during the first round of implementing the blended course. Running the course several times with various groups of students could add to the validity of the findings and also aid in further improving the shortcomings of the existing course. After all, quality assurance is an ongoing process rather than a one-shot procedure (Adair, 2014). Another limitation is related to lack of sustainability and discontinued practice. Since the course was designed and developed to fulfill the requirements of the doctoral program that the first two authors were enrolled in, other instructors may not be willing to adopt it to their contexts since it requires a great amount of time and effort on the part of the instructor.
The current study has implications for online or blended course designers and developers as well as teachers. It introduces methods and resources to design, develop, deliver, and evaluate such courses. The authors also recommend designers to take a look at an evaluation rubric before embarking on the task of course design so as to assure the appropriacy of their choices and decisions from the outset. This latter point further highlights the significance of faculty development in using rubrics such as the QM rubric. In fact, QM provides professional development courses and workshops for faculty who wish to learn about effective online course design as well as those who aim at becoming QM peer reviewers. Roehrs, Wang, and Kendrick’s (2013) study on preparing faculty to use the QM Model is a recommended source to refer to for universities and institutes of higher education which are considering the adoption of this model.
The aim of the present study was to examine the quality of a blended course of English for general academic purposes targeting Japanese undergraduate students at Osaka University. In order to assess the quality of the course, two courses of action were taken: (1) having the course peer-reviewed by a trained Quality Matters reviewer, and (2) conducting a survey study to measure the satisfaction of the students enrolled in the course. The main findings of the study are as follows:
The first round of peer review based on the QM Higher Education Course Design Rubric (Fifth Edition) yielded a score of 70 out of 99. The review process rigorously demonstrated areas in need of improvement. The course was further revised in accordance with the reviewer’s comments and suggestions and was evaluated as meeting all the standards upon amendment with a new score of 99 out of 99.
Students were in general satisfied with the course and believed that it met their language needs and helped them improve their practical English skills. Some of them reported struggling to submit their speaking assignment caused by the malfunctioning learning management system and unstable Internet connection.
Despite the technical problems, the researchers hold that the course has met its predefined objectives to a great extent, i.e. getting the students to practice all four language skills in an integrated manner and aiding them in improving their practical English skills within a course which met their language needs. In order to further refine the course design, development, and delivery, there is a perceived need to rerun the course with various groups of students so as to further ameliorate it in the future.
As a final word, like many institutions of higher education worldwide, Osaka University is adopting online and blended learning more than ever before. As suggested by Roehrs et al. (2013), more online courses will be implemented from now on, and this stresses the increasing need for more faculty development opportunities to assure quality in online education and student satisfaction.
The authors hope that this study can inspire instructors and researchers at Osaka University and other universities across Japan to consider the benefits and affordances of blended learning and to enrich their students’ learning experiences through offering quality blended courses.
Analysis, Design, Development, Implementation, and Evaluation
Collaboration and Learning Environment
English for General Academic Purposes
Massive Open Online Courses
Open Educational Resources
Osaka University Global English Online
Quality Matters Research
Successive Approximation Model
Adair, D. (2014). A process to improve course design: A key variable in course quality. In K. Shattuck (Ed.), Assuring quality in online education: Practices and processes at the teaching, resource, and program levels (pp. 81–90). Sterling, VA: Stylus.
Al Zumor, W. H. Q. (2015). Quality Matters rubric potential for enhancing online foreign language education. International Education Studies, 8(4), 173–178. http://dx.doi.org/10.5539/ies.v8n4p173 .
Allen, M. (2012). Leaving ADDIE for SAM: An agile model for developing the best learning experiences. Alexandria, VA: ASTD Press.
Alizadeh, M., Mehran, P., Koguchi, I., & Takemura, H. (2017). Learning by design: Bringing poster carousels to life through augmented reality in a blended English course. In K. Borthwick, L. Bradley, & S. Thouësny (Eds.), CALL in a climate of change: Adapting to turbulent global conditions – Short papers from EUROCALL, (pp. 7–12). Dublin, Ireland: Research-publishing.net. https://doi.org/10.14705/rpnet.2017.eurocall2017.680.
Mehran, P., Alizadeh, M., Koguchi, I., & Takemura, H. (2017). Designing and developing a blended course: Toward best practices for Japanese learners. In K. Borthwick, L. Bradley, & S. Thouësny (Eds.), CALL in a climate of change: Adapting to turbulent global conditions – Short papers from EUROCALL, (pp. 205–210). Dublin: Research-publishing.net. https://doi.org/10.14705/rpnet.2017.eurocall2017.714.
Boettcher, J. V., & Conrad, R. M. (2010). The online teaching survival guide: Simple and practical pedagogical tips. San Francisco, CA: Jossey-Bass.
Dietz-Uhler, B., Fisher, A., & Han, A. (2007). Designing online courses to promote student retention. J Educ Technol Sys, 36(1), 105–112. https://doi.org/10.2190/ET.36.1.g.
Garrison, D. R., & Kanuta, H. (2004). Blended learning: Uncovering its transformative potential in higher education. The Internet and Higher Education, 7(2), 95–105. https://doi.org/10.1016/j.iheduc.2004.02.001.
Garrison, D. R., & Vaughan, N. D. (2008). Blended learning in higher education: Framework, principles, and guidelines. San Francisco, CA: Jossey-Bass.
Graham, C. R. (2006). Blended learning systems: Definitions, current trends, and future directions. In C. J. Bonk, & C. R. Graham (Eds.), The handbook of blended learning: Global perspectives, local designs, (pp. 3–21). San Francisco, CA: Pfeiffer.
Grgurović, M. (2017). Blended language learning: Research and practice. In C. A. Chapelle, & S. Sauro (Eds.), The handbook of technology and second language teaching and learning, (pp. 149–168). Hoboken, NJ: Wiley-Blackwell.
Gruba, P., Cárdenas-Claros, M. S., Suvorov, R., & Rick, K. (2016). Blended language program evaluation. London, England: Palgrave-Macmillan.
Gruba, P., & Hinkelman, D. (2012). Blending technologies in second language classrooms. London, England: Palgrave-Macmillan.
Harker, M., & Koutsantoni, D. (2005). Can it be as effective? Distance versus blended learning in a web-based EAP programme. ReCALL, 17(2), 197–216. https://doi.org/10.1017/S095834400500042X.
Harkness, S. S. J. (2015). How a historically black college university (HBCU) established a sustainable online learning program in partnership with quality Matters™. American Journal of Distance Education, 29(3), 198–209. https://doi.org/10.1080/08923647.2015.1057440.
Hollowell, G. P., Brooks, R. M., & Anderson, Y. B. (2017). Course design, quality Matters training, and student outcomes. American Journal of Distance Education, 31(3), 207–216. https://doi.org/10.1080/08923647.2017.1301144.
Johnson, L., Smith, R., Willis, H., Levine, A., & Haywood, K. (2011). The 2011 horizon report. Austin, Texas: The New Media Consortium. Retrieved from http://www.nmc.org/pdf/2011-Horizon-Report.pdf.
Kimura, S., & Mori, Y. (2013). 平成24年度TOEFL-ITP実施に関する報告書:結果と分析・考察 [A report on the TOEFL-ITP administered in Heisei 24: The results, analyses, and inquiries]. Toyonaka, Japan: Center for Education in Liberal Arts and Sciences, Osaka University.
Ko, S., & Rossen, S. (2010). Teaching online: A practical guide, (3rd ed.). New York, NY: Routledge.
Kobayashi, K., & Little, A. (2011). Learner perceptions on the usefulness of a blended learning EFL program. JALT CALL Journal, 7(1), 103–117. Retrieved from http://journal.jaltcall.org/articles/7_1_Kobayashi.pdf.
Kwon, K., DiSilvestro, F. R., & Treff, M. E. (2017). Online graduate course evaluation from both students’ and peer instructors’ perspectives utilizing Quality MattersTM. Internet Learning, 5(1), 7-16. https://doi.org/10.18278/il.5.1.2.
Legon, R. (2006, September). Comparison of the quality Matters rubric to accreditation standards for distance learning. Retrieved February 24, 2018, from http://bit.ly/2HKfdk2.
Legon, R. (2015). Measuring the impact of the quality Matters rubric: A discussion of possibilities. American Journal of Distance Education, 29(3), 166–173. https://doi.org/10.1080/08923647.2015.1058114.
Lowenthal, P. R., & Hodges, C. B. (2015). In search of quality: Using quality Matters to analyze the quality of massive, open, online courses (MOOCs). Int Rev Res Open Distribut Learn, 16(5), 83–101 Retrieved from http://www.irrodl.org/index.php/irrodl/article/view/2027/3438.
Martin, F., Ndoye, A., & Wilkins, P. (2016). Using learning analytics to enhance student learning in online courses based on quality Matters standards. J Educ Technol Sys, 45(2), 165–187. https://doi.org/10.1177/0047239516656369.
Quality Matters. (2014). Standards from the QM higher education rubric (5th ed.). Retrieved from https://www.qualitymatters.org/sites/default/files/PDFs/StandardsfromtheQMHigherEducationRubric.pdf
McCarty, S., Sato, T., & Obari, H. (2017). Implementing mobile language learning technologies in Japan. Singapore: Springer International Publishing. https://doi.org/10.1007/978-981-10-2451-1.
Osguthorpe, R. T., & Graham, C. R. (2003). Blended learning environments: Definitions and directions. Quart Rev Dist Educ, 4(3), 227–233 Retrieved from https://www.jstor.org/stable/44428863.
Pallant, J. (2001). SPSS survival manual. Maidenhead, England: Open University Press.
Pardo-Gonzalez, J. (2013). Incorporating blended learning in an undergraduate English course in Colombia. In B. Tomlinson, & C. Whittaker (Eds.), Blended learning in English language teaching: Course design and implementation, (pp. 51–60). London, England: British Council.
Piña, A. A., & Bohn, L. (2014). Assessing online faculty: More than students surveys and design rubrics. Quart Rev Dist Educ, 15(3), 25–34.
Roehrs, C., Wang, L., & Kendrick, D. (2013). Preparing faculty to use the quality Matters model for course improvement. MERLOT Journal of Online Learning and Teaching, 9(1), 52–67 Retrieved from http://jolt.merlot.org/vol9no1/roehrs_0313.pdf.
Romero, M., & Barbera, E. (2011). Quality of learners’ time and learning performance beyond quantitative time-on-task. International Review of Research in Open and Distance Learning, 12(5), 125–137 Retrieved from http://www.irrodl.org/index.php/irrodl/article/view/999/1894.
Rush, P. (2015). Isolation and connection: The experience of distance education. Int J E-Learn Dist Educ, 30(2) Retrieved from http://www.ijede.ca/index.php/jde/article/view/936/1599.
Stavredes, T., & Herder, T. (2014). A guide to online course design: Strategies for student success. San Francisco, CA: Jossey-Bass.
Tuapawa, K. (2016). Interpreting experiences of teachers using online technologies to interact with students in blended tertiary environments. Int J Inform Comm Technol Educ, 12(4), 76–87. https://doi.org/10.4018/IJICTE.2016100107.
Vai, M., & Sosulski, K. (2011). Essentials of online course design: A standards-based guide. New York, NY: Routledge.
Wise, J.-M., & Im, T. (2015). Framework for assessment from an institutional perspective. In S. Koç, X. Liu, & P. Wachira (Eds.), Assessment in online and blended learning environments, (pp. 289–306). Charlotte, NC: Information Age Publishing.
Yang, Y., & Cornelius, L. F. (2004). Students’ perceptions towards the quality of online education: A qualitative approach. Association for Educational Communications and Technology, 27, 861–877.
Young, A., & Norgard, C. (2006). Assessing the quality of online courses from the students' perspective. The Internet and Higher Education, 9(2), 107–115. https://doi.org/10.1016/j.iheduc.2006.03.001.
Zeng, W. Y., & Perris, K. (2004). Researching the efficacy of online learning: A collaborative effort amongst scholars in Asian open universities. Open Learning, 19(3), 247–264. https://doi.org/10.1080/0268051042000280110.
Availability of data and materials
This research project has been fully documented on the following Google site: https://sites.google.com/view/ougeo/. Details on course design, development, implementation, and evaluation can be found under “Research Log”. Additional datasets are available upon request from the corresponding author.
Mehrasa Alizadeh is a PhD candidate at Takemura Lab, Graduate School of Information Science and Technology, Osaka University, Japan. Her research is focused on designing, developing, implementing, and evaluating an online academic English course.
Parisa Mehran is a PhD candidate at Takemura Lab, Graduate School of Information Science and Technology, Osaka University, Japan. Her doctoral dissertation concerns the design, development, implementation, and evaluation of an EGAP online course.
Ichiro Koguchi is a professor of English at the Graduate School of Language and Culture and Center for Education in Liberal Arts and Sciences, Osaka University, Japan. An EFL practitioner, his focus is on academic writing and online language education.
Haruo Takemura is a professor at the Graduate School of Information Science and Technology, Osaka University, Japan. His research interests include human-computer interaction and online education.
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Alizadeh, M., Mehran, P., Koguchi, I. et al. Evaluating a blended course for Japanese learners of English: why Quality Matters. Int J Educ Technol High Educ 16, 6 (2019). https://doi.org/10.1186/s41239-019-0137-2
- Blended learning
- English for general academic purposes
- Quality Matters