Instilling purpose and value in the implementation of digital badges in higher education
International Journal of Educational Technology in Higher Education volume 16, Article number: 44 (2019)
This qualitative multi-case study examined digital badge programs being used at three higher education institutions, at the course level, the department level, and the university level. The study sought to explore the adoption of badges in higher education using Rogers’ (2003) innovation diffusion theory to identify factors associated with the adoption process in order to provide contextual insight into factors that impede and facilitate successful implementation of badge systems. Rogers established that there are five specific characteristics of innovations (relative advantage, compatibility, complexity, trialability, and observability) which are attributed to varying rates of adoption (2003). The results of the current study also attributed factors related to complexity as barriers to successful adoption of badges in each of the three cases. Results found that usability issues, increased faculty workload, and a lack of understanding of the badges’ purpose and value were the main factors which negatively impacted badge adoption.
Digital badges are electronic symbols used as micro-credentials to document achievement or skills mastered such as course completion, professional development participation, or training completion (Parker, 2015). Badge design and implementation processes are reliant on the purpose of the badge system and the context in which it will be used (O'Byrne, Schenke, Willis, & Hickey, 2015; Randall, Harrison, & West, 2013). Implementation is variable across institutions as a result of the uniqueness of each badge system, so factors such as badge system type, purpose, platform, and scope must be considered (Abramovich, 2016; Carey, 2018; Devedžić & Jovanović, 2015; Grant, 2016; Jovanovic & Devedzic, 2015).
Digital badges now cross multiple fields and are used in informal learning settings, corporate settings, government settings, and in all levels of education from K-12 to higher education. Typically used for non-tangible representations of skill achievement and accomplishments, the efficacy of badges in educational settings is still being explored.
Research is exploring whether or not digital badges are effective pedagogical tools as well as how they may be best utilized in higher education settings. Additional research is needed exploring their potential uses such as rewards, incentives, assessment, or skills recognition (Grant, 2016). Other uses for badges in higher education include virtual credentialing systems that would allow students and employees the ability to showcase newly acquired skills and mastery of new content.
In order for a new innovation, such as badges, to be successfully implemented and accepted, factors such as organization culture, participant experiences, and communication processes will also need to be considered by those introducing the innovation (Huang, Shadiev, Sun, Hwang, & Liu, 2017). The framework established by Rogers (2003) to describe innovation diffusion was applied to badge system implementation in order to better understand what drives successful adoption of badges.
Badges in higher education
With growing interest in the use of digital badges, research has been emerging to explore to what extent badge systems can be used in higher education (Abramovich, Schunn, & Higashi, 2013). One question that is being discussed in the badging community include whether badges are better suited for lower-level courses versus upper-level. Another question that is warranting further research includes the uses of badges in higher education. The effectiveness of whether badges are effective for improving class participation or should be limited to the acquisition of skills needs to be further explored. Others suggest using badging programs as a way to teach the required curriculum for an entire college course, rather than using badges as a supplement to the curriculum. Randall et al. (2013) suggest that giving students a choice in which badges they want to complete will promote autonomy and motivate them to learn.
Motivational effect of badges
Abramovich et al. (2013) suggested that badges awarded for participation may not be valued as much as badges awarded for knowledge and skills acquisition. The results of their study revealed that badges impact both intrinsic and extrinsic motivation and that the effects of badges vary across learner ability, prior knowledge level, and badge design. Incentives should be avoided as well, as they are forms of extrinsic rather than intrinsic motivators.
A recent study by Reid, Paster, and Abramovich (2015) found that badges were only slightly motivating for higher-level learners and even less motivating (in some cases, demotivating) for lower-level learners in undergraduate composition courses in which badges are tied to multiple course outcomes. Participants were identified with high or low expectancy-values and intrinsic motivation and attitudes toward badges were measured throughout a semester. The results supported findings from the motivation research which points to badges being more motivating when used as intrinsic rather than extrinsic motivators such as rewards. The findings also supported prior research (Abramovich et al., 2013) indicating that the efficacy of badges is tied to the learner type.
Peer review and feedback opportunities
Badges hold potential to generate peer discussion and feedback opportunities (O'Connor & McQuigge, 2013). Incorporated badges into a Big Open Online Course (BOOC) titled Educational Assessment. Peer discussion and feedback was a fundamental component of the course. Earned badges could be shared over social networks and email. Participants could also link their final work product along with peer endorsements and comments to the badge.
Digital badges were used by where they examined how badges could be used to provide feedback on student performance in an online learning environment. While the researchers did not provide the results of their analysis pertaining to their study, they did provide the survey tool, data set, and the descriptions created for the individual badges for other researchers to replicate. The data set serves to supplement future empirical studies in order to increase sample size and statistical power. The researchers note the small sample size of 44 participants as a limitation in this study and offer justification for their methodology.
Course-based and program-based badges
Credit-based badge systems in college courses, such as the one in the afore-mentioned Reid et al. (2015), are still a rare form of badge use in higher education. Wilson, Gasell, Ozyer, and Scrogan (2016) describe two badging systems at their university. The first badge system implemented at the individual course-level served as a pilot for a planned system to be implemented at the university-level and embedded in the school’s learning management system. The university-based researchers reference Rogers (2003) innovation diffusion theory and highlight the role of early adopters for both badge systems. Conducted a mixed-methods study in a graduate program for teacher education to study the effect of badges on student participation and interaction. The experimental group participated in a course which integrated badges and the control group was a traditional course. The researchers found that badges increased student interaction, although it did not significantly impact participation. Findings suggest that badges may be more useful in online courses that incorporate a read-write-reflect-comment model rather than courses that are already highly interactive such as project-based courses.
Faculty members at the University of Colorado have implemented badges at the course level and at the program level. Badges have been integrated into individual courses, and there are plans to incorporate badges into the university’s learning management system. These faculty members identify as early adopters and see the value of badges beyond the classroom and as a mechanism to document professional development and to assist in outreach and marketing activities. Badges also have potential to strengthen traditional degree programs, to support competency-based programs, and to link badge earners to potential employers and professional organizations (Randall et al., 2013; Wilson et al., 2016); however, it is important for badging activities to be facilitated by instructors in order for badges to be effective pedagogical tools. Instructors can provide necessary scaffolding as needed, especially for lower-level learners (Ahn, Pellicone, & Butler, 2014). Badge earners may need academic and technical support as they progress through learning activities associated with badges.
Badges as an assessment method
Coastal Carolina University faculty have implemented a badge system used in the undergraduate composition courses. The online badge system serves as the assessment model for these particular English courses. The badge system provides a framework for consistency with course assignments and evaluations across multiple course sections and instructors (Reid et al., 2015). Wilson et al. (2016) also refer to the course-level badge system as assessment-based. Badges are used to certify skill mastery within courses in this badge system. The authors warn that weak assessment threatens the validity of badges. The authors share other challenges that arise when badges are used in higher education. Some of the challenges include lack of support and buy-in, absence of program vision, and failure to align badge programs with university missions, as well as technical difficulties within badge systems.
An assessment platform was also developed by educational technologists at Purdue University using an e-portfolio system called “Passport”. This assessment platform aligns outcomes to challenges and incorporates scaffolding. Well-designed badges can provide a form of alternative assessment which supports collaboration and engagement (Parker, 2015). Poorly designed badges which are awarded when competencies are not completely met are a threat to badge validity within the badge system and in the wider badge community. Weak assessment can adversely impact a university’s reputation (Wilson et al., 2016). In summary, badges afford multiple types of assessment such as instructor, peer, and self-assessment (Gibson, Coleman, & Irving, 2016).
Micro-credentials and professional development
Researchers have explored the potential for digital badges to be valued as a reliable system to validate skills and knowledge by potential employees (Parker, 2015; Yu, Dyjur, Miltenburg, & Saito, 2015). Prospective employees have an opportunity to demonstrate soft skills as well as their content mastery related to particular subject matter. This was also explored by Gamrat, Zimmerman, Dudek, and Peck (2014) as they tested the use of digital badges, referred as “microcredentials” to showcase professional development opportunities.
Open badges can be used in higher education as a tool when teaching soft skills such as communication, decision-making, time management, leadership, problem-solving, etc. The University of Central Oklahoma identifies soft skills they expect graduates to demonstrate and uses badges to document and track these skills in addition to grades (Gibson et al., 2016; Parker, 2015). Badges hold potential to provide a method of credentialing skills. Badges verify skills mastered through informal and formal learning experiences. Micro-credentials can be used to document professional development at workshops and other trainings in order for university staff, faculty, and graduate students to catalogue their professional learning experiences (Yu et al., 2015). Wilson et al. (2016) suggest badges used as micro-credentials could position universities as credentialing institutions.
Possible forums for badges include the use of digital backpacks to collect and showcase skills to prospective employers (Goligoski, 2012). For example, Wilson et al. (2016) envision a badge system used in Master’s level courses where badges are used as micro-credentials in student online, professional portfolios. The authors also see potential for badges used to forge partnerships with employers and professional organizations.
Yu et al. (2015) describe a planned pilot study on the impact of a digital badges program at a Canadian University. Badges used as micro-credentials allow for criteria-based professional learning documentation rather than just participation-based certificates. Learners demonstrate a skill or knowledge attained at a professional development session and the badge can contain the artifact of the accomplishment along with the criteria for earning the badge. The pilot badge program at University of Calgary includes two courses, Course Design Program and Teaching Online Program, in which participants can earn badges. The Course Design Program requires participants to create a new course or revise a current course. Badge program participants are expected to create course outcomes and develop instructional and assessment activities to meet the course outcomes. The Teaching Online Program runs for 4weeks and requires participants to facilitate and create online learning activities. Participants can display earned badges on their university profile, export them to Mozilla’s Open Badges Backpack, or display them on LinkedIn (Yu et al., 2015). The literature points to the interest in badges used as micro-credentials and more specifically to badges used in faculty development programs in higher education. This interest warrants further exploration into micro-credentials and faculty development, especially extending the literature base to include more empirical studies on the topic.
Potential value of digital badging in higher education
Badges hold value as a system for potential employers to validate skills aside from content mastery and to allow learners to reflect upon and track their own soft skills (Parker, 2015; Yu et al., 2015). While badges are growing in popularity, standards are still being developed to promote consistency with implementation and recognition of skills. An analysis of the varying factors impacting implementation and adoption could be helpful for better understanding the unique nuances related to badge implementation and establishing consistency among badge design standards.
Micro-credentials can serve as skill documentation within an institution or with the inclusion of metadata, they can be made visible to other institutions in the form of open badges. Gamrat et al. (2014) tested a digital badge system’s capability to support custom professional development opportunities. They based their study on the theoretical framework of customized decision making to allow teachers choice in which professional development activities they pursue. Badges can be used in higher education as a tool when teaching soft skills such as communication, decision-making, time management, leadership, problem-solving (Ahn et al., 2014; Randall et al., 2013). The University of Central Oklahoma identifies soft skills they expect graduates to demonstrate and uses badges to document and track these skills in addition to grades (Gibson et al., 2016; Parker, 2015). Micro-credentials can be used to document professional development at workshops and other trainings in order for university staff, faculty, and graduate students to catalogue their professional learning experiences (Yu et al., 2015). Wilson et al. (2016) suggest badges that are recognized as micro-credentials could position universities as credentialing institutions.
In a needs assessment to explored perceptions related to implementing badge programs in higher education institutions and design considerations to be taken, Carey (2018) interviewed professors, administrators, and instructional designers who had experience with implementing badges in higher education contexts. The participants reported a number of challenges and barriers that impacted their ability to design badge initiatives. These challenges included obtaining buy-in from faculty, the need for training badge faculty and administrators, establishing rigor within badge programs, and ensuring sustainability of programs. While a number of participants admitted to challenges, they also noted a number of learner and institutional benefits such as increased learner motivation (Ash, 2012; Catalano & Doucet, 2013; standardizing the documentation and reporting of learner accomplishments (Devedžić & Jovanović, 2015; Olneck, 2012); and providing an infrastructure for transferring accomplishments for employee recognition (Carey, 2018).
Purpose of the study
The purpose of our study was to identify and analyze the factors that impact the diffusion and adoption of digital badges used across higher education institutions. The intent was to identify challenges and solutions in badge system implementation so that others can learn from the experiences of badge users from three higher education institutions. As more higher education institutions explore the adoption of digital badging in coursework (Abramovich, 2016; Berge & Muilenburg, 2016; Cheng, Watson, & Newby, 2018; Mah, 2016), it is important for educators to be aware of and able to address constraints that may impact implementation (Carey, 2018).
There are limited empirical studies on the use of badges which may be attributed to their relatively short existence (Abramovich et al., 2013; Gamrat et al., 2014; O'Connor & McQuigge, 2013). There is a notable lack of discussion in the literature on the diffusion and adoption of badge programs in higher education (Casilli & Hickey, 2016; Jovanovic & Devedzic, 2014; Ostashewski & Reid, 2015). An understanding of what factors drive the adoption of badges used in higher education could help those involved in badging to capitalize on the potential use of badges and successfully implement programs.
Roger’s model provides a framework for understanding how new innovations are designed, introduced, communicated, and adopted by society over periods of time. Time is viewed from three angles: the innovation-decision process of an individual, the innovativeness of the adopter, and the rate of adoption. The innovation design process is defined by Rogers (2003) as:
The process through which an individual (or other decision-making unit) passes from first knowledge of an innovation to forming an attitude toward the innovation, to a decision to adopt or reject, to implementation and use of the new idea, and to confirmation of this decisions (p.475).
Innovations which involve larger systems rather than just a few individuals take longer to adopt due to the complexity of the decision process involving more people. The adopter categories in which individuals can be sorted according to their innovativeness are: innovators, early adopters, early majority, late majority, and laggards (Rogers, 2003). The five characteristics of innovation which are attributed to varying rates of adoption:
Relative advantage: The degree to which an innovation is perceived as better than the idea it supercedes.
Compatibility: The degree to which an innovation is perceived as being consistent with the existing values, past experiences, and needs of potential adopters.
Complexity: The degree to which an innovation is perceived as difficult to understand and use.
Trialability: The degree to which an innovation may be experimented with on a limited basis.
Observability: The degree to which the results of an innovation are visible to others (p. 15).
In addition to the five attributes identified by Moore and Benbasat (1991), Rogers (2003) identified additional attributes related to user perception being an influencer of adoption success, voluntariness, and image. These additional constructs were developed to further dissect factors that influence the diffusion and adoption of an innovation as they relate to Rogers’ and emphasizes individual characteristics of those responsible for leading an innovative initiative.
Innovations which individuals view as having more relative advantage, compatibility, trialability, and observability, yet less complexity, are expected to have higher adoption rates. Rogers (2003) strongly suggests that potential adopters should determine the measure of degree that an innovation contains as an attribute. We investigated the following two questions:
What are factors that facilitate diffusion and adoption of badges by higher education institutions?
What are challenges that impact diffusion and adoption of badges in higher education?
IRB approval was obtained from Old Dominion University prior to data collection for this study. We conducted a collective case study (Stake, 1995; Yin, 2017), consisting of a contextual examination of badge programs used in three different higher education institutions. Institutions which are using badges in various ways were studied in order to widen the scope of experiences and increase the relevance of the results to other institutions. The cases included: a university using badges in undergraduate writing courses for credit purposes, a university using badges in graduate instructional technology courses for skill certification, and a university using badges in graduate teacher education courses for purposes of peer feedback. A case study approach allowed for participants to share their experiences and knowledge in a way that may inform others in similar contexts who are interested in implementing a badge program.
We obtained a purposeful sample for this study. We approached three professional, academic contacts from higher education institutions currently running badge programs and invited them to participate. Case A promoted digital badges in graduate level educational technology courses. The badges were related to the development of an instructional design website portfolio. Students were given the option to complete additional tasks in order to earn a badge. This digital badge program was led by one faculty member who served as the designer and system administrator and was our primary point of contact for data collection during this study.
Case B incorporated the use of digital badges in multiple sections of undergraduate English composition courses. Similar to Case A, students were given the option to complete additional tasks to earn up to eight digital badges for their work related to the courses (i.e. research skills, summarization skills, quoting skills). This course is typically taught by 80 instructors each semester. For purposes of this study, we interviewed two faculty member and received completed anonymous questionnaires from nine others detailing their experiences with the badge program. Three students completed questionnaires explaining their participation in the badge program. The instructor who developed the badge initiative served as our primary contact for the study, providing us with archival data.
Case C incorporated digital badges in graduate level instructional design courses. Students have the opportunity to earn approximately 55 badges for technology courses each semester. A total of four faculty participate in this badging initiative. All participating faculty members were interviewed and two completed anonymous questionnaires to provide additional information. Four students completed an anonymous questionnaire to provide us with additional information from a student perspective.
Data collected for this study included interviews with faculty responsible for implementing the badge programs at the institutions, a questionnaire issued to students and faculty involved in the badging programs measuring information technology innovation adoption and diffusion (Moore & Benbasat, 1991), and archival documents such as meeting notes related to badge system design and roll-out, planning documents, training materials (i.e. course syllabi, assignments, and rubrics) and evaluation documents related to the overall operation of the badge system. Additionally, a review of the associated research and articles for the badge programs took place. Data were supplied by the primary contacts at each of the institutions.
Case Study A provided us with five artifacts representing course syllabi, assignments, and rubrics related to their badge program. They also provided a research article that they had published showcasing their badge program. The sole program administrative participated in an interview for this study.
Two program administrators from Case Study B participated in interviews with our research team. Eight faculty and eight students completed questionnaires detailing their experiences with the badge program. Our document analysis consisted of reviewing course syllabi, assignments with rubrics, training materials provided to badge users, research and articles related to the badge program, and an overall evaluation of the operation of the badge program.
A total of four faculty and administrators participated in interviews for Case Study C. Two faculty and four students completed questionnaires. Our document analysis consisted of reviewing materials related to badge system roll-out, course syllabi, training materials for badge users, and research articles related to the badge program.
Within-case analysis was conducted followed by a cross-case analysis took place to check for theme consistency, similarities, and differences across cases (Creswell, 2013). Questionnaire items were sorted and categorized by the constructs noted by Moore and Benbasat (1991): voluntariness, relative advantage, compatibility, image, ease of use, result demonstrability, visibility, and trialability. These constructs were then aligned to the following innovation attributes: relative advantage, compatibility, complexity, trialability, and observability (Table 1).
The innovation attributes (relative advantage, compatibility, complexity, trialability, and observability) served as priori codes for each piece of data and emergent coding was used as other codes surfaced. Information was reviewed, memoed, segmented, and coded by a primary reviewer. A second reviewer evaluated the materials to validate the work of the first reviewer according to the innovation attributes. If there were discrepancies with codes, we used a peer debriefing technique to discuss and arrive at a consensus (Lincoln & Guba, 1985; Willig, 2008).
Case study a: graduate level educational technology courses
Case A is the smallest of the three badge programs studied. Although limited in scale and operated at the course-level, this case illustrates that one individual can design and implement a badge system. Additionally, it is important to note that the other cases also began at the course level during the piloting phases. The professor/participant, who is also the course developer, of the graduate course designed the system using an open source platform for badges and implemented it 5 years ago in an emerging technologies course to increase student interactions with one another and for the use of peer feedback on assignments. The participant explained, “…rather than simply having them read about badging, I wanted them to experience it.”
The incorporation of badges used as peer review has increased participation and “richer comments” on the discussion boards along with an enhanced sense of community in the courses. The participant noted that students seemed to work harder and produce better quality technology projects when the badges were implemented. In this particular program, the professor generates the badges once students have evaluated each other’s work and provided feedback. The students then vote on who should receive the badges based on criteria of aesthetics, navigation, end user, and clear purpose. This added element of competition has also been a positive contribution to the courses as well. Participation is required as part of the class assignments and expectations.
The main themes which surfaced for the innovation attribute of relative advantage were direct experience, competition, and community. The design of the system supports a peer review framework where students evaluate each other’s technology projects and offer critical feedback to one another. These lateral learning benefits, including modeling and community building, were the main themes for the innovation attribute, compatibility, and were mentioned several times during the interview. A program administer explained “they get to know each other’s work; they get to know each other’s background, because they’ve looked at someone else’s work. They are learning from the model of somebody else’s work, so I see many subtle, what I call lateral benefits.” The badge system has an informal evaluation process, aside from a published, mixed methods action research study conducted to determine the impact of the badging system.
The participant noted that there are no reported usability issues that prevented students from being able to claim and export their badges; however, a recurring theme throughout the interview was that of increased faculty workload. The increased time and effort required to run the badge program is the main theme categorized under the innovation attribute, complexity. The badge system is separate from the learning management system, which is a limitation and time consideration.
The innovation attribute, observability, is built into the system that is built to support peer review and modeling of technology projects. After reviewing each other’s projects, the students vote for which of their peers should win 1st through 3rd place. The Google Form voting ballot allows the professor to export data to a Google Sheet and calculate votes and analyze the data. Once the winner is calculated, an email that announces the winners is generated for the class. The professor sends the badge link to the winners, and they can navigate to the Credly website and claim their badge. Then they can export their badges to social media sites, such as LinkedIn, for future employers to view.
The final innovation attribute, trialability, was tied to the pilot phase where the participant began to notice the outcomes of better work being produced and “richer comments” on the discussion board. The downside to the implementation was again noted as the added faculty workload. The badge program studied in this case, uses badges in a specific way that may not be the most common application. This case study is a window into just one way that badges are being used. The participant believes that, “used judiciously, [badges] can encourage a type of social interaction you might not get otherwise”.
Case study B: undergraduate composition courses
Case B presents a use of badges in higher education where they are used for skill assessment and comprise the fourth credit hour of undergraduate composition courses. This use of badges is unique in that the badges are tied to credit, tuition, and faculty salaries. Leadership in the composition department of the university implemented badges to assure that consistent instruction across the many sections of the courses that form the undergraduate writing courses. They had noticed there was an achievement gap and that students were not receiving consistent instruction across sections. The number of sections is very high, and there are many adjuncts teaching these courses.
The solution was to incorporate a badging program in order to unify composition courses and ensure students were receiving instruction central to the intended course learning outcomes. The badge system solution came about at a time that the traditional 3-credit hour approach was being questioned, and the department head was interested in incorporating a digital, multi-modal element into the courses.
The badge program in this case study undergoes the most formal evaluation process of the three cases studied. The annual, intensive evaluation is a “mini-study” conducted each summer at the close of the academic year. Multiple faculty members participate as members of the evaluation team. Three random writing samples are collected from each section, and the committee evaluates the samples using a standard rubric that measures the learning outcomes. This process informs decisions about which badges need to be revised or emphasized more in class. Survey data from the instructors is also collected and analyzed. The program administrators believe that the badge system will continue, since all stakeholders are on board now, they are committed to the fourth credit hour, and the multimodal component has been ingrained in the system.
The badge website was designed by the primary contact for this case study who currently maintains the system. One program administrator took on the role of ensuring pedagogical soundness. In the fall of 2013, one of the program administrators first heard about badges, and soon afterwards suggested badges as a potential solution to the issues the writing department was facing. The department chair fully supported the idea and required that all faculty for the upcoming semester resubmit their letters of intention to agree to the badge program being implemented in the courses they would teach. There was strong faculty resistance at first, because they were being asked to fundamentally change the way they had been teaching, some of them for many years. Conversely, the administration was excited; they saw the initiative as innovative and progressive. During the initial phases, one of the program administrators was pitching the badge program all across the campus, while the other program administrator was building the system and sorting through technical issues. One program administrator stated, “It was really hard. We got a lot of resistance from our faculty…. We had to convince people that this was a smart way to teach writing.”
The program administrators developed a training video presentation for potential badge users in order to obtain buy-in of the badging process. They also placed emphasis on conducting research related to their badge program as a means of developing notoriety of the program. It was the administrators’ intent to establish credibility by developing a scholarly presence to appeal to students as badge users.
The pilot phase was very brief and informal. One of the program administrators had implemented a badge system with a small class of five or six students over half a semester while at another university so that they could be sure that Credly was integrated properly with WordPress. They wanted to be sure that the “infrastructure and technology work[ed]” before the full-scale implementation went live almost overnight in the fall of 2014. Some initial issues with the implementation were that faculty were unable to locate their students’ work. One participant stated that “…there could be 3000 student submissions for all kinds of different badges, and if you are a faculty member, you just want to find your students or a specific badge”. They needed a way to organize the submissions, so a drop-down menu was created with faculty names so that students could choose their instructor’s name when submitting their badge assignments.
Since the badges are tied so specifically to the learning outcomes for the class, they have found that students do not typically share the badges externally, although it is an option. Students can create a profile on Credly and then share the badge to social media sites, but typically the badges are not shared outside of the program. The value of the badges is internal rather than external in this particular case. One participant explained:
…we use badges very differently. We don’t really see students sharing them externally, and that’s one of the major questions that we have for digital badges, which is how do we get students to engage with these badges and share them publicly? (Program Administrator01).
The benefits to the institution are increased visibility and notoriety from press, published articles, and research studies. The site is also public, so other departments at the university are able to access it. This is one of the benefits of the system not being outsourced. One participant remarked, “Well institutionally, they are very much supportive of the program. I think they mostly like it for its innovation … and it’s gotten quite a bit of press and some notoriety, so they of course like that”.
It is certainly a benefit for both faculty and students that the system is mobile friendly; however, there are some limitations. One of the major usability issues is the lack of integration of the badge system into the learning management system. The usability issue related to lack of sorting capability was mentioned and is an ongoing problem that likely will not go away due to the platform’s constraints. Instructors are unable to segregate badge assignments into specific sections and add due dates for the work. One participant said, “…there’s been no way for us to organize submissions into different sections, and that’s something that a lot of students -- a lot of faculty members would have liked to have seen”.
Case study C: university open badge platform
Case C traces the adoption of the badge program using the initial system which was implemented 4 years ago, and the current system on the new platform that rolled out at the beginning of 2017. The system was updated due to the badge program growth, and so others outside of the university could access it as an open source option for badges. The primary purpose of the badge program at this institution is to support technology integration courses in the teacher education program. The current focus is on providing pre-service teachers with technical skills using badges that are “chunked into categories”. Some of the badges act as prerequisites to the course, so they must be earned before starting the class. The education technology classes serve approximately 20 different education majors who need technology training that is specific to their subject area. A program administrator explained that badges allow instructors of the course to “compartmentalize” projects that are specific to each subject area. The instructors who teach the various sections of the course each have a group of badges in which they specialize, so the badge assignments can be graded by a person who has expertise in that area. This approach supports learning that is more personalized and customized for the students in the various programs.
Although the initial system did not require approval outside of the department and was run “as a hack out of WordPress”, it continued to grow, so they designed the newest site with the college dean’s approval and support. The new system is their current badge “repository and issuer”. The badge system is available to individuals, higher education institutions, and school districts searching for a platform in which to use badges. The system designers and administrators in this case study have a fair amount of technical ability, which may play a significant role in its ability to success and scalability.
There was not a formal piloting phase when the department first started using badges. A couple of instructors tried the badges out in a class one semester, and they liked the results well enough that they extended the use of badges to the other sections by the next semester. One of the lessons learned during the initial implementation was that creating badges for all of the content areas included in the secondary education teacher preparation courses was not possible for the design team. They soon learned to make use of subject matter experts sourced from undergraduate students in specific content areas such as science majors. The designers took on the role as instructional design “mentors” to these undergraduate students who offered their content knowledge expertise in return.
One of the program administrators shared that one of the benefits of badges in this particular system is that “learners get an opportunity to see a variety of different tools and different technologies that are out there”. Additionally, it affords the students an opportunity to decide which of the different tools they would actually like to use in their teaching practice. Participants shared that an added benefit is recognizing students for specific skills and promoting agency in choosing what particular technologies they want to learn.
The benefits to the university are marketing-related, since the badges are branded with their university logo. The way the system is set up allows for alumni to earn badges even after they graduate. This is a form of alumni support and continuing professional development for students. Furthermore, the badges are endorsed by and branded with a major, national, professional association in the field of educational technology. This particular case has not experienced notable faculty resistance; however, a participant remarked:
The value of badges are not easily understood by students all the time. I think part of it is that it’s not fully understood by … employers, and so as employers become more cognizant of what they represent, I think that students will see the value in it, but I think that’s something that we learned early on and that we are continually kind of learning and struggling with. (Program Administrator01).
Usability issues have arisen and been addressed throughout the years, where possible. One of the main challenges has been that the badge system is not linked to the university’s learning management system. This causes frustrations for both students and faculty and creates what one program administrator refers to as a “pain point”. Other participants cited issues of glitches in the search function, along with problems with permission settings, lack of instructor notification on resubmissions, and filtering issues.
Increased faculty workload since the time of badge system implementation was a common theme among all three cases. Rigorous badge assignments increase evaluation time, and as Case A’s participant noted, running the badge program takes a lot longer than just grading the discussion board. It takes time to pitch, design, develop, promote, implement, and manage a badge program. The three cases have seen instructional benefits due to the implementation of badges, and participants from each case made it clear that they are committed to keeping the badge systems running in their programs.
Case B models how badges can be indicators of benchmarks and how they can create unification while maintaining flexibility and instructor autonomy. Badges can be shaped & reshaped based on assessment results and programmatic needs. Case C illustrates how institutions can partner without outside institutions and offer badges as extracurricular or competency-based instructional components. Wilson et al. (2016) suggest a shared vision for the badge system that involves multiple levels at the institution.
One similar thread which weaves through each case includes the importance of “buy-in”. Case A shows that one person with a bit of autonomy and some technical skills can fully design and implement a badge system at the course level without having to convince anyone to support it. Case C exhibits how a few tech-savvy individuals with an interest in badging can design and implement a system that eventually is outgrown, rebuilt, and scaled with full support of the college. Case B requires many layers of institution support and faculty buy-in. Including multiple stakeholders in the design and implementation phases of a new badge program can create a sense of ownership for all those involved (Wilson et al., 2016). Faculty and student buy-in is crucial to a badge program’s success. Faculty and students from the cases mentioned the importance of creating opportunities for buy-in repeatedly.
Considerations for the alignment of purpose and value
The idea of building rigor and challenge into badge assignments is a recurring theme in this study. Rigor not only makes the badges more instructionally sound, but it also increases credibility and value outside the awarding institution (Hickey et al., 2014). Argue the importance of rigor for open badges and maintain that two components must be present in badge design in order to ensure rigor. The first component is to establish strong criteria for those to earn the badge (Grant, 2016; Lemoine & Richardson, 2015; Olneck, 2012; Wilson et al., 2016). The second component is to support interrater reliability by establishing consistent badge evaluator practices.
If a badge program is designed to support programmatic needs and has more benefits for the instructors than the students, perhaps the consideration of including metadata in order to increase external value would be beneficial to students (Casilli & Hickey, 2016; Dyjur & Lindstrom, 2017; Mah, 2016). Of course, for students to understand the value, potential use must be explained to them. Additionally, exporting and sharing to public sites must be explained and supported by the system (Ahn et al., 2014; Casilli & Hickey, 2016; Dyjur & Lindstrom, 2017). Aside from that, potential employers must understand and value badges, and that is not always the case (Hurst, 2015; Lemoine & Richardson, 2015). Grant (2016) claims that most badge earners do not claim or display their earned badges in a public repository and that these factors would need to change in order for badges to become more widely valued.
Alignment of purpose and value
The two prevalent themes that continually resurfaced in this study were badge value and purpose. The recommendations from badge participants emphasized the importance of creating badges with internal and external value in order to maximize buy-in. The recommendations also emphasized the importance of a comprehensible purpose for the badges. Complexity negatively impacts badge diffusion, so increased faculty workload and usability issues need to be mitigated and the benefits of the badges need to offset increased workload. Table 2 aligns the themes of purpose and value to Rogers’ (2003) innovation attributes by showing how each recommendation supports building in purpose and value to badges Strategically aligning badge systems to purpose and value and communicating these aspects of the system to stakeholders supports successful badge system implementation.
Consider badge purpose
The primary theme for participant responses was to “ask why digital badges?” when considering the use of badges. Participants indicated that those considering the implementation of badges should think about how badges can support the intended learning outcome goals and how badges can enact best practices in the field. It was suggested that designers of badge systems let pedagogical goals drive the decisions. It was also recommended that those responsible for implementation communicate to users the badges’ purpose and utility outside of the awarding institution. Gamrat, Bixler, and also argue the importance of considering purpose when designing badges. As with other technology tools, badges should not be used just for the sake of using badges, yet they should be the best tool for which to achieve a particular learning objective.
Consider badge design
The participants made suggestions related to badge design. The suggestions included to incorporate sound criteria that can be verified by evidence via metadata embedded in the badges. It is important to make badges valuable, so they will be intrinsically motivating. Badge designers should ensure that badges represent real skills and that they are robust & rigorous, but not overwhelming. Participants shared that it is beneficial to invite stakeholders to contribute during the process to instill a sense of ownership over the product in order to increase buy-in. Wilson et al. (2016) also recommend participation at all levels when initiating a badge system. Badge designers should also consider how to invite community involvement in the badge program. One tip was to protect the institution’s brand with the use of logos on the badge icons. Another recommendation was to determine if badges would expire and be renewable. Also describe badge expiration dates. As technology evolves, particular tool functions may become obsolete, so expiration dates can ensure that badges remain relevant.
Consider technical aspects
It was recommended for those interested in implementing a badge system to start out small and keep it simple. It is important to question the affordances of available technology and limitations of the platform and users. It is wise to use an existing open-source badge platform and to proceed in stages. Also describe the benefits of using an existing commercial badge system. Finally, when drafting the implementation plan, it is recommended to provide faculty training to ensure grading consistency. Interrater reliability is key to ensuring that badge activities maintain rigor and credibility.
As we reviewed materials provided by each of the case sites, we began to observe a common theme focused on instilling purpose and value in attribute to the diffusion constructs identified by Moore and Benbasat (1991) and Rogers (2003). A framework (Fig. 1) for successful badge program implementation is provided below. The framework divides the recommendations into three categories which emerged from the cases’ data: badge instructional design, badge system platform, and badge program implementation.
The Badge Instructional Design category contains recommendations related to the badge instructional components. One of the strongest messages from the participants was to consider the purpose for using badges and to be certain that badges are the right tool to achieve that purpose. The learning activities associated with the badges should be meaningful, which means they should be robust and linked to evidence such as the final work product. Directly tying badges to learning objectives is an effective strategy. The badges can be made more valuable by also embedding the criteria that was met to earn the badge.
The Badge System Platform category comprises recommendations related to ensuring the badge platform is manageable and sustainable. When starting a new badge program, it is a good idea to use one of the open source badge systems available and to start out with a simple plan, possibly at the course level. If the badge system is too difficult to use or increases faculty workload beyond reason, the users will resist adopting the new system. The value of using the badges must outweigh the increased workload. One way to share the workload and to involve multiple levels of stakeholders is to outsource roles to students or other faculty who have expertise in areas such as graphic design, information technology, or specific content areas.
The Badge Program Implementation category consists of recommendations for successful badge program roll-out and evaluation processes. Purpose and value are two of the most important considerations for using badges, and these foundational attributes must be communicated to everyone involved from the beginning of the implementation process. Building in an evaluation plan which involves multiple stakeholders also is integral to the badge program’s success, sustainability, and possible scalability.
If a badge program is designed to support programmatic needs and has more benefits for the instructors than the students, perhaps the consideration of including metadata in order to increase external value would be beneficial to students. Of course, for students to understand the value, potential use must be explained to them. Additionally, exporting and sharing to public sites must be explained and supported by the system. Aside from that, potential employers must understand and value badges, and that is not always the case. Grant (2016) claims that most badge earners do not claim or display their earned badges in a public repository and that these factors would need to change in order for badges to become more widely valued.
The recommendations from badge participants emphasized the importance of creating badges with internal and external value in order to maximize buy-in. The recommendations also emphasized the importance of a comprehensible purpose for the badges. Complexity negatively impacts badge diffusion, so increased faculty workload and usability issues need to be mitigated and the benefits of the badges need to offset increased workload.
When designing badges, the first step is to be able to articulate how badges will address an instructional or curricular need. Table 3 suggests strategies to add purpose and value for badges and addresses these strategies from each level of the Framework for Successful Badge Program Implementation.
This study focused on three higher education institutions to provide data using a case study approach. Other types of institutions and organizations using badges were not studied. Rogers (2003) describes other factors, besides the five specific characteristics central to this study, that can influence diffusion and adoption of innovations. This study did not analyze personal characteristics of adopters or change agents in an effort to classify their adopter categories. Further, organizational innovativeness was not measured in this study. Instead, the unit of measure was the innovation (badges) and the focus was on the attributes of badge programs that facilitate and impede diffusion and adoption.
Another significant limitation to the study is that on-site observations were not conducted by the researcher due to geographical constraints. All communication and data collection took place via email and electronic communication platforms, so rich data could have been lost in the process. The abbreviated length of the study was also a limitation to consider. Finally, due to the limited nature of the case study methodology, results are not generalizable to larger populations outside of higher education due to the contextual nature of this particular study.
The existing body of badge literature includes empirical studies related to motivation (Abramovich et al., 2013; Reid et al., 2015) and badges used as peer review in college courses (O'Connor & McQuigge, 2013) and teacher professional development programs (Gamrat et al., 2014). More research is necessary to establish the need for a common framework or standardization of badges, particularly those used for credentialing. Drawing upon the main themes from the current study, purpose and value, future research could focus on how a clearly communicated purpose and perceived value of badges may impact motivation to earn and share badges.
Additional research is needed to refine the ways that digital badges are being used by employers. If more and more organizations are giving credence to digital badges during job interviews, it will help if there is some standardization as to how these badges are validated. The long-term impact of badge systems has not been explored due to the relatively small number of years of badge existence, but as time goes on, it will be possible to study the long-term effects of badges in specific contexts.
This study traced the diffusion and adoption process of digital badge system implementation across three universities that are purposing badges in various ways. Rogers (2003) indicated that the innovation attributes of relative advantage, compatibility, observability, and trialability positively impacted innovation adoption. The main factors found in this study to facilitate diffusion and adoption of badges were compatibility of the badge program with the institution’s values and needs, observability of the value of badges both internally and externally, and relative advantage of badges grounded by a clear purpose that is communicated to stakeholders. Trialability was not shown to play a significant part in the successful adoption of the badge programs in this study.
The findings suggest how to move the innovation-decision process forward through potential barriers and challenges, and offer solutions and suggestions based on lessons learned from the individual case studies. The findings indicate a need for understanding the potential uses of badges outside of the awarding institutions as badge users seek to have their earned badges recognized beyond college walls and in professional settings. The across-case study findings also inform instructional design practice by formulating recommendations for badge program implementation.
Ongoing involvement of multiple stakeholders increases buy-in and promotes a sense of ownership in the institution’s badge program. It is important that program administrators establish roles early on in the design and development stages of the system. Additional studies focusing on the role of the program administrator as an influencer are needed in order to address challenges related to large scale implementation of badge programs in higher education. We believe that Moore and Benbasat’s (1991) emphasis on the user (program administrator) as an influencer can be further enhanced by delving into how their role in communicating purpose and promoting value of the badging opportunities.
Availability of data and materials
The authors do not have taken to be shared with a repository.
Abramovich, S. (2016). Understanding digital badges in higher education through assessment. On the Horizon, 24(1), 126–131.
Abramovich, S., Schunn, C., & Higashi, R. (2013). Are badges useful in education? It depends upon the type of badge and expertise of learner. Educational Technology Research and Development, 61(2), 217–232.
Ahn, J., Pellicone, A., & Butler, B. S. (2014). Open badges for education: What are the implications at the intersection of open systems and badging? Research in Learning Technology, 22, 1–9.
Ash, K. (2012). Digital badges would represent students’ skill acquisition. Education Week, 5(3), 24–25.
Berge, Z. L., & Muilenburg, L. (2016). Competency-based education and the relationship to digital badges. In L. Y. Muilenburg, & Z. L. Berge (Eds.), Digital badges in education: Trends, issues, and cases, (pp. 44–52). New York: Routledge.
Carey, K. L., & Stefaniak, J. E. (2018). An exploration of the utility of digital badging in higher education settings. Educational Technology Research and Development, 66(5), 1211-1229.
Casilli, C., & Hickey, D. (2016). Transcending conventional credentialing and assessment paradigms with information-rich digital badges. The Information Society, 32(2), 117–129.
Catalano, F., & Doucet, K. J. (2013). Digital ‘badges’ emerge as part of credentialing’s future Retrieved September 20, 2017, from http://www.proexam.org/images/resources/ProExam_DigitalBadgesPaper_071713.pdf.
Cheng, Z., Watson, S. L., & Newby, T. J. (2018). Goal setting and open digital badges in higher education. TechTrends, 62(2), 190–196.
Creswell, J. W. (2013). Qualitative inquiry and research design: Choosing among five approaches, (3rd ed., ). Los Angeles: Sage.
Devedžić, V., & Jovanović, J. (2015). Developing open badges: A comprehensive approach. Educational Technology Research and Development, 63(4), 603–620.
Dyjur, P., & Lindstrom, G. (2017). Perceptions and uses of digital badges for professional learning development in higher education. TechTrends, 61(4), 386–392.
Gamrat, C., Zimmerman, H. T., Dudek, J., & Peck, K. (2014). Personalized workplace learning: An exploratory study on digital badging within a teacher professional development program. British Journal of Educational Technology, 45(6), 1136–1148.
Gibson, D., Coleman, K., & Irving, L. (2016). Learning journeys in higher education: Designing digital pathways badges for learning, motivation and assessment. In D. Ifenthaler, N. Bellin-Mularkski, & D. Mah (Eds.), Foundations of digital badges and microcredentials: Demonstrating and recognizing knowledge and competencies, (pp. 115–138). New York: Springer.
Goligoski, E. (2012). Motivating the learner: Mozilla’s open badges program. Access to Knowledge: A Course Journal, 4(1).
Grant, S. L. (2016). History and context of open digital badges. In L. Y. Muilenburg, & Z. L. Berge (Eds.), Digital badges in education: Trends, issues, and cases, (pp. 3–11). New York: Routledge.
Hickey, D. T., Otto, N., Itow, R., Schenke, K., Tran, C., & Chow, C. (2014). Badges design principles documentation (DPD). Interim project report Retrieved from Indiana University, Center for Research on learning and technology website http://iudpd.indiana.edu/JanuaryReport.
Huang, Y. M., Shadiev, R., Sun, A., Hwang, W. Y., & Liu, T. Y. (2017). A study of the cognitive diffusion model: Facilitating students’ high level cognitive processes with authentic support. Educational Technology Research and Development, 65(3), 505–531.
Hurst, E. J. (2015). Digital badges: Beyond learning incentives. Journal of Electronic Resources in Medical Libraries, 12(3), 182–189.
Jovanovic, J., & Devedzic, V. (2014). Open badges: Challenges and opportunities. In Proceedings of the 13th international conference on web-based learning- ICWL 2014, (pp. 56–65). Berlin: Springer.
Jovanovic, J., & Devedzic, V. (2015). Open badges: Novel means to motivate, scaffold and recognize learning. Technology, Knowledge and Learning, 20(1), 115–122.
Lemoine, P. A., & Richardson, M. D. (2015). Micro-credentials, nano degrees, and digital badges: New credentials for global higher education. International Journal of Technology and Educational Marketing (IJTEM), 5(1), 36–49.
Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic inquiry. Newbury Park: Sage.
Mah, D. K. (2016). Learning analytics and digital badges: Potential impact on student retention in higher education. Technology, Knowledge and Learning, 21(3), 285–305.
Moore, G. C., & Benbasat, I. (1991). Development of an instrument to measure the perceptions of adopting an information technology innovation. The Institute of Management Sciences, 2(3), 192–222.
O'Byrne, W., Schenke, K., Willis, J. E., & Hickey, D. T. (2015). Digital badges: Recognizing, assessing, and motivating learners in and out of school contexts. Journal of Adolescent & Adult Literacy, 58(6), 451–454.
O'Connor, E. A., & McQuigge, A. (2013). Exploring badging for peer review, extended learning and evaluation, and reflective/critical feedback within an online graduate course. Journal of Educational Technology Systems, 42(2), 87–105.
Olneck, M. (2012). Insurgent credentials: A challenge to established institutions of higher education Retrieved August 10, 2017, from HASTAC website http://www.hastac.org/files/insurgent_credentials__michael_olneck_2012.pdf.
Ostashewski, N., & Reid, D. (2015). A history and frameworks of digital badges in education. In T. Reiners, & L. C. Woods (Eds.), Gamification in education and business, (pp. 187–200). New York: Springer.
Parker, H. E. (2015). Digital badges to assess bloom’s affective domain. The National Teaching & Learning Forum, 24(4), 9–11.
Randall, D. L., Harrison, J. B., & West, R. E. (2013). Giving credit where credit is due: Designing open badges for a technology integration course. TechTrends, 57(6), 88–95.
Reid, A. J., Paster, D., & Abramovich, S. (2015). Digital badges in undergraduate composition courses: Effects on intrinsic motivation. Journal of Computers in Education, 2(4), 377–398.
Rogers, E. M. (2003). Diffusion of innovations. New York: Free Press.
Stake, R. E. (1995). The art of case study research. Thousand Oaks: Sage.
Willig, C. (2008). Introducing qualitative research in psychology: Adventures in theory and method, (2nd ed., ). Maidenhead: McGraw Hill/Open University Press.
Wilson, B. G., Gasell, C., Ozyer, A., & Scrogan, L. (2016). Adopting digital badges in higher education: Scoping the territory. In D. Ifenthaler, N. Bellin-Mularski, & D.-K. Mah (Eds.), Foundation of digital badges and micro-credentials, (pp. 163–177). Switzerland: Springer.
Yin, R. K. (2017). Case study research and applications: Design and methods. Thousand Oaks: Sage.
Yu, L., Dyjur, P., Miltenburg, J., & Saito, K. (2015). Micro-credentialing: Digital badges in faculty professional development. In P. Preciado Babb, M. Takeuchi, & J. Lock (Eds.), Proceedings of the IDEAS: Designing responsive pedagogy conference, (pp. 82–89). Calgary: Werklund School of Education, University of Calgary.
Both authors declare that they have no competing interests associated with this submission.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Stefaniak, J., Carey, K. Instilling purpose and value in the implementation of digital badges in higher education. Int J Educ Technol High Educ 16, 44 (2019). https://doi.org/10.1186/s41239-019-0175-9
- Digital badges
- Higher education
- Diffusion and adoption