Evaluation of a simulation-based hospital pharmacy training package for pharmacy students

Abstract

This study describes the process undertaken to develop, implement and evaluate a simulation-based training package focused on medication management and reconciliation processes for final year pharmacy students about a patient’s hospital journey. A five module training package was developed following a literature review and consultation with stakeholders. The simulation-based package immersed students in a real-life scenario and was delivered to final year pharmacy students over a six-week period in 2016. Data on knowledge, skills and confidence was collected via a survey in the week preceding engagement with the online training package and 1 week post completion of the training. The mean score was compared across four student categories: three categories incorporated students who had not completed a hospital pharmacy placement and one category comprised students who had completed a hospital placement. Qualitative feedback was collected via an online survey at the conclusion of the training program. Of the 79 participants, 44 (55.7%) completed both the pre and post- test surveys that showed the change in score was statistically significant. There was a significant positive change in mean test scores across all four student categories for the domains of skills, knowledge and confidence. Assessment of students’ confidence according to 16 ranking statements also improved markedly post-training. Thirty-one students provided qualitative feedback that was generally positive. The positive outcomes reinforce the rationale to include online simulation-based methodologies as part of pharmacy education programs. The model provides a reproducible framework for online simulated learning activities that could be applied within various professions and educational environments.

Introduction

Medication-related problems (MRPs) that include medication errors, adverse effects, medicine interactions and non-adherence are a major burden on health care systems. In Australia, it has been estimated that more than 190,000 Australian hospital admissions per year are attributable to MRPs, with an associated cost of AUS$660 million (Roughead and Semple 2009). A proportion of MRPs are caused by medication errors which often arise during prescribing, dispensing or administration of medicines and contribute to patient harm or injury, and in some cases mortality (Runciman et al. 2003). Research by Roughead and Semple (2009), Roughead et al. (2016) indicates that approximately 50% of Australia’s hospital admissions due to MRPs are considered potentially preventable. Consequently, appropriate and correct management of medicines by health professionals plays a central role in facilitating better health outcomes (Aronson 2009; Roughead et al. 2016, 2013). However, it was highlighted by Stowasser et al. (2004) that the medication management pathway involves complex steps, with the potential for errors and interventions at various stages. It is therefore important that health services integrate medication management processes and that health professionals are adequately trained to follow these processes to reduce incidences of MRPs.

The Australian Commission on Safety and Quality in Health Care 2016 in Healthcare prioritised the need to improve the safety and quality of medicine usage and developed a number of medication safety tools and resources to support hospital and community practices in achieving better outcomes. These initiatives align with quality use of medicines (QUM) principles as specified in the Australian National Medicines Policy (Australian Government Department of Health and Ageing 2002; Stowasser et al. 2004). The continued appropriate use of these tools and resources by all health professionals involved in patients’ medication management remains an important strategy to ensure QUM and continuum of care within clinical practices and across healthcare settings. Hospital pharmacists play an important role in the integration and application of these clinical resources and the coordination of medication management services (Society of Hospital Pharmacists of Australia 2013). Published studies continue to support the role of hospital pharmacist services in improving medication safety, including continuum of care services (Hennen and Jorgenson 2014; Hilmer and Ogle 2006; Semple and Roughead 2009).

Pharmacy students need to understand medication management processes within a variety of settings, including community and hospital sectors. Indeed, the Australian Pharmacy Council Accreditation Standards for Pharmacy Programs (2012) states that it is important for hospital practice settings to be experienced by students during pharmacy degree programs. Exposing pharmacy students to hospital pharmacy practices provides an opportunity for students to develop a working knowledge of hospital procedures and an appreciation of the role of hospital pharmacists. Hospital pharmacy placements also facilitate the integration and application of theory knowledge. However, Australian universities increasingly find it challenging to organise hospital placements for all pharmacy students. Research by Fejzic et al. (2016) suggested a variety of reasons for this including course time constraints and high competition for limited hospital places. Pharmacy academics therefore need to explore alternative approaches to equip students to practise in hospital pharmacy settings.

Simulation is a generic term that refers to the replication of real world experiences. According to Gaba (2004) simulation-based education is an educational approach that uses simulation, in its many applications and modalities, to achieve educational goals through experiential learning. International literature emphasizes the need to utilize an assortment of technology-based learning environments to train and educate health professionals (Cook et al. 2011; Keppell et al. 2015) and specifically for pharmacists (Fejzic and Barker 2015; Smith and Benedict 2015). Multiple studies support embedding technology-enhanced, simulation-based education into pharmacy curricula, such as online learning that is enriched by pre-recorded role plays with actors or simulated participants, virtual patients, task simulations and/or interactive platforms that simulate a specific skill or competency (Fejzic and Barker 2015; Kirwin et al. 2013; Loke et al. 2011; Regan et al. 2014; Salter et al. 2014; Smith and Benedict 2015; Veettil and Rajiah 2016; Vyas et al. 2013).

A systematic review and meta-analysis by Cook and colleagues that compared technology-enhanced simulation-based education with no intervention concluded that greater effects were consistently seen in a number of outcome areas including participants’ skills, knowledge and behaviours, with simulation based education and training (Cook et al. 2011). However, within Australia, there is a lack of evidence on the development, implementation and evaluation of simulation-based hospital medication management processes in the training of pharmacy students. A need was hence identified to develop a simulation-based training package that was focused on hospital medication management processes to assist pharmacy students in the application of medication management tools and utilization of relevant resources.

The overall aim of this project was to develop, implement and evaluate a simulation-based training package to expose final year Bachelor of Pharmacy (BPharm), BPharm Honors and Graduate Entry Master of Pharmacy (MPharm) students to a patient’s hospital journey focusing on medication management and reconciliation processes. It was hypothesized that the exposure of pharmacy students to the simulated hospital journey training package would assist them in gaining an understanding of hospital medication management processes and procedures, how hospitals function, how the pharmacist works within the hospital and the importance of the continuum of care. The purpose of introducing the simulated-based training package in the curriculum was to better prepare students before hospital placements.

Methods

This project involved the development of the simulation-based training package, pre-testing of students’ knowledge and skills, utilization of the training package, post-testing of students’ knowledge and skills and obtaining students’ feedback. Mixed methodology (Dowding 2013) was used for the evaluation of the training package and the impact of the training on students’ knowledge and skills. Evaluation involved a written pre- and post-test to explore students’ knowledge of medication management processes and skills in the application of medication management processes (i.e. through the use of clinical tools), and an end-of-training survey to obtain quantitative and qualitative feedback on student satisfaction with the training package. The pre-and post-tests and surveys were validated by a pharmacy academic and two hospital pharmacists.

Curtin University Human Research Ethics Committee approval was granted to conduct this research (RDHS-63-16). The project was divided into the four phases with Phase 1 being the development of the training package, Phase 2 the pre-testing of students’ knowledge, skills and confidence, Phase 3 the students’ self-directed learning through completing the training package and Phase 4 the post-testing of students’ knowledge, skills, confidence and obtaining their feedback.

Phase 1: Development of simulation-based training package

Phase 1 of the project focussed on the development of the simulation-based training package over a period of approximately 6 weeks (mid-March to end-April 2016). This included a review of the literature and available resources to consider and explore how resources could be incorporated. Consultation with hospital pharmacists and other hospital health professionals involved in medication management processes, representatives of the Western Australian Medication Safety Group and the Western Australian Hospital Pharmacy Research Alliance was undertaken. The training package comprised five on-line learning modules with specific learning outcomes to simulate a patient’s journey through hospital, summarised in Table 1. The learning outcomes were mapped against the National Competency Standards Framework for Pharmacists in Australia (Pharmaceutical Society of Australia 2016).

Table 1 Training package modules, learning outcomes and competencies addressed

The package was available through Blackboard®, the Curtin University learning management system, to students between end of April and early June 2016. The modules incorporated patient background information, 13 videos, nationally recognized policies, procedures and documentation that facilitate medication management, simulated medication management activities (i.e. admission medication reconciliation, interpret hospital notes and charts with abbreviations, identifying and addressing MRPs), practice questions, pre- and post-tests, and an evaluation and feedback questionnaire. In addition to the medication management activities, the simulation-based aspects of the package included videos with actors to portray an unfolding scenario as a patient received care in a hospital and community setting, as well as hospital and community pharmacists and other health professionals simulating various aspects of medication management processes, such as an initial assessment, medication reconciliation, interprofessional team meetings and a home visit. Woven throughout the package were modules that incorporated strategies to manage high risk medicines in order to emphasise potential for medication errors and approaches to improve medication safety (The Society of Hospital Pharmacists of Australia 2015).

Phase 2: Pre-testing

Phase 2, the pre-testing of students’ knowledge, skills and confidence, involved final year pharmacy students namely fourth year Bachelor of Pharmacy (BPharm), fourth year BPharm Honors and final year Graduate Entry Master of Pharmacy (MPharm) students. These students were provided with an information sheet about the project and invited to workshops scheduled at the end of April 2016 during which they were given an overview of the study. Students were categorised into four groups according to their program of enrolment and to identify those who had already completed a hospital placement: BPharm students who had already completed a hospital pharmacy placement, BPharm students who had not already completed a hospital pharmacy placement but completed a community placement instead, BPharm students enrolled in the Honors program and who had not completed any placement, and MPharm students who had not completed a hospital pharmacy placement, but similar to community students had completed a community pharmacy placement. It was assumed that the BPharm students who already completed a hospital placement would have a better understanding of hospital medication management processes and associated clinical tools compared with the other students.

Students were requested to complete the pre-learning test survey to obtain baseline information about their knowledge, skills and confidence. This survey was in written format and was completed in class during the workshops. Section A consisted of 15 short answer questions that assessed their ability to apply medication management processes and procedures through a hospital patient case study. The case study simulated real practice and incorporated medical notes, referral letters, pathology results and medication charts. Students were allowed 50 min to complete the 15 questions. These responses were de-identified and marked by an independent academic and a total ‘knowledge and skills’ score was obtained by adding up each student’s marks out of a total of 70. Section B consisted of 16 statements covering the learning outcomes of the various training package modules. Students had 5 minutes to rate their confidence in being able to apply these outcomes by rating statements on a 5-point Likert scale (strongly disagree - strongly agree). A total ‘confidence’ score was obtained by assigning numbers 1–5 to the responses and summing over the 16 statements and a total score for the section calculated as the sum of the 16 scores (giving a total score between 16 and 80). A higher total score indicated greater overall confidence.

Phase 3: Self-directed learning through online training package

Phase 3 involved students accessing the online training package over a five-week period (end-April to early-June). The release of the modules was staged with modules being released on a weekly basis. The staged release process provided students with progressive prompts to complete the modules in weekly sittings through self-directed learning.

Phase 4: Post-testing

Phase 4 was the post-testing of students’ knowledge, skills, confidence as well as obtaining their feedback. Students were requested to complete an in-class written post-test survey in early June 2016. The post-survey contained the same questions (Sections A and B) as the pre-test to enable comparisons of pre- and post-test data and assess the potential impact of the training. Additionally, the post-survey incorporated a section to provide feedback about the training package and aspects that worked well in addition to aspects that could be improved. Respondents assessed their level of agreement (strongly disagree - strongly agree on a 5-point Likert scale) with statements concerning the overall quality and delivery of the training package and the usefulness of the individual modules. Open-ended questions were included to capture qualitative feedback.

Standard descriptive statistics (means, standard deviations, medians and ranges for continuous variables) were used to summarize the survey responses. Fisher’s Exact Test was used to compare the response rates for the post-test survey between the student groups. An Analysis of Variance (implemented using a General Linear Model [GLM]) was used to compare the student groups with respect to pre-test and post-test scores as well as the changes from pre- to post-test for each participant. For each outcome, the statistical significance of the pairwise differences between student groups was obtained from the model. A GLM was used to explore whether the improvement in knowledge and skills score was associated with a change in each of the statements in Section B (confidence), as well as the student type. Finally, a Mixed model (random effects regression model) was used to explore whether the knowledge and skills test score was associated with the confidence score at each survey. For this analysis, each participant contributed either 1 or 2 records (pre- or post-test or both), and the participant identifier was named as a random effect so that correlation between records belonging to the same participant could be taken into account. Data analysis was conducted using the SAS version 9.2 software, and a p-value< 0.05 was taken to indicate a statistically significant association in all tests.

Results

A total of 105 students (31 BPharm students who had completed a hospital placement, 48 BPharm and 17 MPharm students who had completed a non-hospital community placement, and 9 BPharm Honors students who had completed no placement) were invited to placement orientation workshops and were invited to participate in the study. Pre-test data from the 79 students who agreed to participate (75.2%) were analysed. These participants comprised 27 BPharm students who had completed a hospital placement, 36 BPharm and nine MPharm students who had completed a non-hospital community placement, and seven BPharm Honors students who had completed no placement. Pre- test scores were available for all 79 participants, but one of these responders did not complete the ranking questions (Section B). Of these 79 participants, 44 (55.7%) completed the post-test questions. These included 17 students who had completed a hospital placement and 27 students who had not (18 BPharm, four MPharm and five BPharm Honors students).

The response rates to the post-test questions were similar across the 4 student types (p = .58 –Fisher’s Exact test). The scores on the knowledge and skills questions are shown in Table 2.

Table 2 Pre- and post-test scores for knowledge and skills

The change in score for the 44 participants who completed both the pre- and post- tests was statistically significantly greater than zero when using the one sample t-test to compare the mean change (p < .0001) or the signed rank test to compare the median change (p < .0001). Therefore, there seems to be strong evidence that the scores improved (by 11.2 points on average) following the training.

Because of the large proportion of participants who did not complete the post-test survey due to being off-campus to study for examinations, an independent samples t-test was performed to compare the baseline scores for those who did and did not complete the post-test. This analysis confirmed that these two groups performed similarly at baseline (mean scores: 29.5, 30.7 respectively; p = .61). This suggests no systematic bias in the post-test responders (i.e. students who completed the post-test performed no better or worse at baseline than those who did not do the post-test survey).

An analysis of test scores at pre-test, post-test and the change from pre to post was performed to identify any differences in these measures between student types. Table 3 shows that, at baseline, the BPharm students who had already completed a hospital placement obtained a statistically significantly higher score than either the other BPharm students or the MPharm students who had not already completed hospital placements, but had a similar score (not significantly different) to the Honors students. Analysis of the change in score from pre- to post-test showed no significant difference between the groups (p = 0.51), showing that the improvement was comparable for all student types. Analysis of the post-test scores showed that the scores had converged, with no significant overall difference (p = .0528). However, there was a trend, with the students who had completed a community placement scoring less than the Honors or hospital placement groups.

Table 3 Analysis of knowledge and skills test scores between participant type (results obtained from the General Linear Model)

The comparison of the pre-and post-test confidence levels of the ranking statements about confidence with medication management roles and processes are summarised in Fig. 1.

Fig. 1

Changes between pre- and post-test data for confidence with medication management roles and processes. Bars represent changes to students’ responses to agree/strongly agree, neutral and disagree/strongly disagree statements

Table 4 summarises the confidence scores pre-and post-test. A higher score indicates strong agreement through the questions, which points towards greater confidence and ability to deal with the various medication management issues. A low score indicates the opposite.

Table 4 Total student confidence scores at pre- and post-test

The change in scores for those participants who completed section B at both pre- and post-test was significantly greater than zero when using the one-sample t-test to compare the mean change (p < .0001) or the signed rank test to compare the median change (p < .0001). Therefore, there seems to be strong evidence that the agreements with the statements in section B increased following the training. Similarly to the knowledge and skills test scores, an analysis of the confidence scores at pre-test, post-test and the change from pre to post was performed to identify any differences in these measures between student groups (Table 5).

Table 5 Total scores on confidence statements, and pairwise differences between student types (obtained from the GLM)

The results show that the students who had completed a hospital placement gained significantly higher total scores at pre-test compared to the other students, while there were no significant pairwise differences between the remaining three groups. Analysis of the change in total scores showed that the hospital placement group showed the least improvement in score (probably because they were starting at a higher baseline), and this was a significantly smaller improvement than either the groups who completed community placements students or the Honors students. However, the improvement for the MPharm students was intermediate and not significantly different from any of the other groups. At post-test, there were no significant differences between groups overall (p = 0.14), showing that the groups had all converged by the end of the training.

A GLM was used to explore whether a change towards agreement with each of the confidence statements in section B was associated with a change in the students’ test scores (section A). It appeared that Question 2 (I am confident using a medication management plan) showed the greatest correlation with the improvement in marks: those whose agreement increased for this question obtained a score which was 13 marks higher than their pre-test score, while those whose Question 2 agreement was unchanged increased in score by 8.3 marks (p = .03). This suggests that greater confidence in using this plan was associated with a jump in the post-test score.

A second regression model (Mixed model) was used to explore whether the knowledge and skills test score was associated with the confidence score. This model accounted for the repeated measurements made on each person by treating the person as a random effect. The results showed that an increase of 1 point in the confidence score corresponded to an increase of 0.50 in the knowledge and skills test score (p < .0001). There was no influence of the student category type (1–4) on this relationship, which means that a similar improvement in test score with increasing section B score was noticed for all student groups.

Qualitative feedback

Thirty-one students provided qualitative feedback. Three specific positive feedback themes emerged and are summarised in Table 6 with supporting quotes.

Table 6 Positive qualitative feedback themes

Areas for improvement mainly related to students wanting more time to work through the modules, inclusion of face-to-face aspects to discuss the modules and one of the students who already completed a hospital placement commented that the training should take place before hospital placements.

Discussion

The training package appeared to significantly increase test scores, and agreement with the statements which assessed confidence in dealing with a range of clinical pharmacy situations. There were differences at baseline between the four different student types with those who had completed a hospital placement scoring the highest. Following the training, all groups showed improvements and gained similar test scores. Assessment of their confidence according to the 16 ranking statements also improved markedly post-training.

International literature continues to reinforce the need for and the positive effects associated with the use of simulation-based methodologies as part of pharmacy educational programs (Kirwin et al. 2013; Loke et al. 2011; Murdoch et al. 2014; Regan et al. 2014; Ruehter et al. 2012; Salter et al. 2014; Smith and Benedict 2015; Veettil and Rajiah 2016; Vyas et al. 2013). However, there is limited information on Australian studies that involved the development, implementation, and evaluation of hospital-based simulation training packages within tertiary education. One Australian study (Fejzic et al. 2016) that focussed on pharmacy students’ communication skills involved supplementing the curriculum with simulation-based activities with positive enhancement of student confidence and professional practice skills. This study showed that the integration of a simulation-based training package into the curriculum of final year pharmacy students improved students’ knowledge base of hospital pharmacy processes as well as their confidence levels in feeling prepared for working in a hospital pharmacy environment.

The 44 students in our study who completed both pre-and post-test questions improved test scores on average by 11.2 marks with a t-test showing a statistically significant improvement in their knowledge and ability to apply medication management tools. An American study by Douglass et al. (2013) evaluated the impact of a simulation-based virtual patient program on third year pharmacy students’ competence in relation to their clinical skills, with post-test results indicating that 78% of the students agreed that the simulation improved their problem solving clinical skills. Kirwin et al. (2013) described subsequent effects on students’ confidence and performance abilities with the implementation and evaluation of a simulated hospital pharmacy module that reflected the day-to-day tasks of a hospital pharmacist. The pharmacy students participating in that study had prior exposure to hospital practice experience, however, it was reported that their confidence and comfort at completing hospital pharmacy tasks was low. Despite the prior experience, students completing their simulation module significantly improved their practice skills and confidence to complete tasks typical of a hospital pharmacist. The authors concluded that exposure to hospital information systems used within the module would be of benefit to pharmacy students prior to their placements. Another study incorporated patient simulated scenarios with other traditional educational methods to prepare medical students for practice with authors suggesting that this combination enhanced third and fourth year students’ critical evaluation skills in order to assess medicine safety (Karpa et al. 2015). Our hospital pharmacy training package will similarly be completed by all pharmacy students at Curtin University before exposure to their first hospital pharmacy placement.

Benefits of incorporating simulation-based activities into pharmacy curricula could result in reduced MRPs and associated adverse events experienced by individuals (Battaglia et al. 2012; Daupin et al. 2016; Metzger et al. 2015). Multiple studies have shown that pharmacists who are involved in medication management processes, such as medication reconciliation programs, improve patients’ clinical outcomes and assist in the reduction of hospital re-admissions (Mekonnen et al. 2016a, b; Steeb and Webster 2012). Our study has shown that the use of an online simulation-based training package which details the complex nature of medication management processes within a hospital setting helps to raise awareness and educate pharmacy students. This may enhance pharmacy students’ practice-readiness by developing their confidence to address MRPs, identify medication errors, and collaborate effectively within interprofessional teams.

The effectiveness of simulated learning in pharmacy is not limited to the domain of higher education. A descriptive cross-sectional study conducted in a 500-bed university hospital identified that a simulation exercise within a ‘medication errors room’ to test the hospital’s medication-use processes resulted in almost 68% of the staff participants that included medical, pharmacy and nursing health professionals, correctly identifying the medication-related errors (Daupin et al. 2016). The researchers emphasised that this type of simulation activity encourages good practice, raises awareness of medication safety and offers improvements in management systems. Another study which compared simulation-based training of nurses to traditional methods such as lecturing demonstrated that simulation-based activities provided a significant reduction of almost 25% in administration-related medication errors, whereas traditional formats such as a didactic lecture, had no effect on reducing medication administration errors for the same group (Ford et al. 2010).

The benefits seen by pharmacy students’ participation in the online medication management training program was enhanced by the inclusion of a simulated interprofessional communication activity. A recent study involving nurses, doctors and a pharmacist based on two medical wards in an acute hospital identified the interpersonal, authoritative and instructive styles used when communicating with patients about medication and highlighted the need for improved interprofessional collaboration in order to ensure safe, patient-centred care is employed during medication management (Liu et al. 2016). By using simulation-based learning, this complex and high level discourse could be explored in ways that engender empathy, respect and safety whilst allowing students the opportunity to practise and rehearse dialogue with the interprofessional team and the patient.

Limitations

A small number of students completed the post-test survey. Some differences between student types may be present but not detected by the analysis due to the small numbers in some groups. As the research had to be completed within a strict time-frame (3 months), students had limited time (5 weeks) to complete the online simulation package. Additional improvements in knowledge and skills may have been observed if they had been able to complete the package over a longer period. In addition, the test that students completed to assess their knowledge and skills was time-limited. Further improvements in test scores may have been observed if students had longer to complete it.

Conclusion

Online simulation-based learning is a beneficial method for engaging final-year pharmacy students in authentic, self-directed, flexible learning, which fills the gap between traditional didactic teaching methods and experiential learning. It has an integral role in helping to prepare students for fieldwork by empowering them to take more responsibility for their own learning, improving their knowledge and skills, and contributing to them being more career ready. The evaluation of the simulation-based training package showed improvements in students’ knowledge, application of clinical tools as well as their confidence in medication management processes. However, the sample size was small and hence the next step will be to conduct a full validation of the methodology and an evaluation of the implementation of the training package with a bigger cohort of students. This model will then provide a reproducible framework for other online simulation-based learning activities and could be applied within various professions and educational environments.