Comparison of pharmacy students randomized to receive drug information reference education via recording or interactive Moodle lesson

Abstract

This study assessed the efficacy of an interactive Moodle lesson teaching the subject of references commonly used to answer drug information (DI) questions on pharmacy student performance and retention. Students were randomized to receive education via either the lesson or a recorded lecture. Students completed a baseline knowledge assessment, an assignment following instruction, the same baseline assessment at the end of the semester to assess retention, and a survey on their experience with the assigned education technique. The differences of median grades on assessments and survey results between groups were compared via Mann-Whitney U tests. Student performance on the baseline assessment was comparable between groups (n = 40 per group; 47.22% vs. 48.61%). Retention of knowledge was less in students in the lesson group (63.89%) than in the lecture group (75%). Students taught by lecture performed better (90.63%) than those taught by lesson (87.5%) on the post-instruction assignment. Survey results from participants (n = 32/80) showed more students in the lecture group (n = 16) agreed time associated with educational technique was reasonable (69% vs. 31%), technique was valuable to learning (75% vs. 44%), and experience was satisfying (56% vs. 31%). These results show student retention of knowledge was not improved with use of a Moodle lesson in comparison with a recorded lecture. Student performance was better immediately following education when taught by lecture and more lecture-taught students agreed the educational technique was time-appropriate, valuable, and satisfying.

Introduction

The utilization of multimedia in teaching and learning has increased significantly over the past 10 years as tools have expanded and student-learning preferences have changed (Becker et al., 2018; Brooks & Pomerantz, 2017). Multimedia is a proven way to enhance learning since “people learn more deeply from words and pictures than from words alone” (Mayer, 2012). Designing multimedia learning activities with Richard Mayer’s Cognitive Theory of Multimedia addresses the issue of how to implement instructional methods that affect cognitive processing to help people learn efficiently and more effectively (Mayer, 2002). Meaningful learning happens when the learner brings in prior knowledge to engage with the multimedia by applying what they learned in new situations and problems (Ambrose, Lovett, Bridges, DiPietro, & Norman, 2010). This act creates new knowledge that can be stored in the long-term memory and can be applied in field experiences in the short and long term.

Teachers are working toward a more active learning approach to their content by developing multimedia and offering it online (Carmichael, Reid, & Karpicke, 2018; Newman, Kim, Lee, Brown, & Huston, 2016). Students are then able to review the content outside of class and come to class more prepared for interaction. For meaningful learning to happen, students should have a differentiated exploration of the course material where videos, readings, images, and words help guide the students in a self-directed activity (Clark & Mayer, 2016). In the education of healthcare professionals, researchers have investigated how use of the Modular Object-Oriented Dynamic Learning Environment (Moodle) learning management system (LMS) affects student outcomes in online learning environments (Oliveira Reis, Ikari, Taha-Neto, Gugliotta, & Denardi, 2015; Seluakumaruan, Jusof, Ismail, & Husain, 2011; Seixas et al., 2016).

One tool available in Moodle, the Moodle lesson, is an activity that provides a flexible, branching flow for presenting course material in a self-paced, self-directed, self-assessed manner. Moodle lessons are useful for self-directed learning or decision-making scenarios. More specifically, teachers develop text and multimedia content and question pages that allow students to “choose their own adventure”; teachers can use these learning activities for student practice or assessment. Typically, students are presented with a question and, based on the answer they select, the lesson will move the student to a new page that is specifically dictated based on their answer. Depending on the chosen answer, students will see different content; this makes the Moodle lesson a virtual learning experience that may be different every time the student accesses it (Moodle.org, 2018).

While experiences with Moodle as an LMS have been reported (Rehatschek, Hӧlzl, & Fladischer, 2011; Rehatschek & Hruska, 2013; Jones & Miller, 2012; Barge & Londhe, 2014), specific information on the use and impact of the Moodle lesson activity does not exist. In order to assess the utility of the Moodle lesson, investigators compared student retention of content taught via 2 different methods. The objective of the study was to assess whether using the Moodle lesson to teach first-year pharmacy students drug information (DI) references would improve their immediate performance and retention of taught content via an associated assignment and end-of-semester assessment, respectively.

Methods

Course structure

The research study was conducted in a 1-credit hour required DI class taught in the Fall 2016 semester of the first year of a traditional 4-year Doctor of Pharmacy (PharmD) curriculum. Goals of the course were to give students an overview of DI practice, expose them to basic DI skills, and teach them to answer DI questions using the systematic approach. The course used a mostly online structure, teaching students with weekly video lectures and assessing them via online assignments. Each week at 11:00 AM on Fridays, students accessed online materials, including recorded lectures and assignments, via Moodle (eg, Week 1). Students were required to review the videos and associated materials, then complete the affiliated assignment over the course of the next 7 days, specifically prior to the deadline of Friday at 10:59 AM; then at 11:00 AM, the next week’s materials (eg, Week 2) released. Six times during the semester, the course met in person to review taught content and provide students with opportunities to apply the concepts practically.

Most weekly assignments were multiple choice in nature, but in an attempt to diversify assignments and use new options available in Moodle, an interactive Moodle lesson was developed. The content of DI resources was selected because a self-directed lesson with ramifications for making various decisions aligns well with the requirement that students be able to utilize and search various DI resources and interpret identified information for applicability to various scenarios in order to execute the systematic approach to DI questions.

Educational methodology

Both the traditional recorded lecture and Moodle lesson educational techniques used the same learning objectives for students, specifically: 1) to select and be able to use the most appropriate DI resources to answer DI questions; 2) to identify DI resources by publication mechanism (ie, textbook, online, or both); and 3) to distinguish between DI references in the same category by being able to recognize differences between them. The instructor used the same information, pictures on slides, and script for both learning methodologies.

The traditional lecture organized 54 DI resources into 18 categories in a 46-min recorded PowerPoint presentation that used 89 slides. Students in this cohort were not provided a handout and there was no opportunity for practice questions.

The 109-page (88 content and 21 question pages) Moodle lesson covered the same 54 DI resources using videos with a maximum length of 2 min. These videos were incorporated into 6 scenarios covering the same 18 categories as the recorded presentation. Students were provided a handout aligned with the lesson and given the opportunity to answer 21 resource-specific practice questions during the lesson. Figure 1 (Title: Moodle lesson images) provides an example of how one of the sections of the Moodle lesson was organized.

Fig. 1

Moodle lesson images

Study methods

To establish similarity at baseline, students took a brief, 6-question quiz at the beginning of the semester; completion of the assessment was considered willingness to participate. Names of participating students were included in the first column of a Microsoft Excel (Microsoft Corporation, Redmond, WA) spreadsheet and a random number (ie, 1 or 2) was then generated in the next column. Based on the result of the random number, students were assigned to receive DI resources education by either the traditional method of recorded lecture or the new Moodle lesson. The local institutional review board (IRB) approved the study.

During week 6 of the semester, students learned about DI references. The LMS released either the recorded lecture or lesson educational materials to the students, based on their assigned group following randomization. Following review of the educational content, both groups of students completed an 11-question assignment for a grade. Students then completed a 3-item survey on their experience with the assigned education technique. The survey utilized a Likert scale (1 = strongly agree to 5 = strongly disagree) and students responded to the following 3 statements: 1) “The time associated with reviewing content prior to completing the assignment was reasonable”; 2) “The presentation of drug information resources was valuable to learning the material”; and 3) “The overall experience associated with learning about drug information resources was satisfying”. At the end of the semester, the students repeated the baseline assessment, which included identical questions, to evaluate retention of knowledge.

Characteristics of the 2 educational techniques were compared. The primary outcome of the study was student retention, as measured by the student median grade on the end-of-semester assessment. Secondary outcomes included the student median grade achieved on the post-lecture/lesson assignment and student opinion based on survey results. Group median grades were compared with Mann-Whitney U tests in SPSS v22 and student thoughts were compared via descriptive statistics.

Results

Student performance

Student performance results are detailed in Table 1. Student groups achieved comparable grades at baseline, showing that the randomization yielded similar groups. One student in each group failed to complete the immediate and end-of-semester assessments. The post-lecture/lesson assignment median grade was higher in the lecture group than in the lesson group, as was the end-of-semester retention assessment median grade.

Table 1 Student performance on baseline, post-class, and retention assessments

Student opinion

Only 32 of 80 (40%) students completed the opinion survey, 16 in each group. Students in the lecture group were more likely to strongly agree or agree with all 3 statements (Fig. 2) [Title: Student opinion of lesson or lecture educational techniques (n = 40)], showing that students preferred the lecture educational methodology to the lesson methodology.

Fig. 2

Student opinion of lesson or lecture educational techniques (n = 40)

Discussion

This investigation appears to be the first assessment of health-profession student performance and opinion of a Moodle lesson. Literature searches yielded studies and reports using Moodle in unique ways, but none that described utilizing or exploring the Moodle lesson. Some reports did mention virtual lessons (Rehatschek et al., 2011; Rehatschek & Hruska, 2013) and Moodle navigation (Scherl, Dethleffsen, & Meyer, 2012), but none of these examples were evaluating the Moodle lesson that is the subject of this study.

The results of this experience show that the Moodle lesson did not produce the expected results as utilized in this particular instance. Students in the lesson group performed worse, retained less, and had a worse experience than students in the lecture group.

Taking into account the educator perspective, the opportunity to incorporate the Moodle lesson into an online course was an exciting prospect. The time dedicated to the development of the lesson was substantial, as ensuring that all content and question pages were appropriately organized required a great deal of attention to detail and review. While not counted directly, it took at least 40 h of faculty time to develop and organize the Moodle lesson content. Based on faculty workload and the findings in favor of the traditional recorded lecture, the use of the lesson in this setting appears to be flawed.

Strengths of this study include that it was a randomized, direct comparison of students with similar knowledge at baseline. Assessing both retention of knowledge and immediate performance, as captured in this study, are important outcomes because they can help instructors learn what educational methodologies assist students in effectively gaining knowledge, rather than just memorizing information. The study was limited in that it did not assess power needed to demonstrate no difference in methodologies because the sample size was limited by class size; therefore, the results of no difference between groups is subject to a type 2 error. Other limitations include that assignment questions were originally written in a year prior to the study based on the traditional lecture content; no consideration was given to the lesson content when questions were written. While the information contained was similar across both methodologies, this method of question development may have led to better results in students taught via the recorded lectures. Providing students in the lesson group a handout may have introduced a potential confounder that improved these scores. While it made sense to the investigators to provide education on DI references via the Moodle lesson, the unexpected results may mean the content was not optimal for the Moodle lesson. Future investigative uses of the Moodle lesson should attempt to prove the appropriateness of the educational content to this format before comparing student performance with other teaching methodologies. Student opinion in the lesson group may have been influenced by the fact that they completed other course materials via the traditional recorded lecture method and were thus familiar with this educational methodology, while students in the traditional recorded lecture group had no exposure to the lesson methodology. Because of this, no direct comparisons of the opinion results have been made. Finally, the overall response rate to the opinion survey was low, limiting the applicability of the results.

These results, when combined with the fact that the course was redesigned in Fall 2017 to a more traditional weekly in-person meeting structure, led to this particular lesson not being reused for student assessment. The lesson, with some minor updating, could be made available for student practice in future course iterations. Additional future considerations would be to use the Moodle lesson for other content in this course where it makes the most sense. Answering DI questions and conducting literature searches are learning experiences that require students to make choices that can have different outcomes based on those decisions. While the amount of development time makes lesson creation burdensome, another attempt in one of these areas may be worthwhile to better determine if the results were due to failure of the lesson to help students retain knowledge or if the problem was with the specific content. Engaging graphic designers may also be helpful for more audience-friendly content development (Maggio et al., 2012).

Conclusion

In summary, a Moodle lesson used to teach first-year PharmD students did not lead to better immediate student performance or end-of-semester student retention in comparison with a traditional recorded lecture format. While both teaching methodologies were effective in regards to student performance and retention, more students receiving education via lecture, in comparison with those taught via Moodle lesson, found the learning methodology to require an appropriate amount of time, have value, and generate an overall satisfying experience.