How stakeholders’ data literacy contributes to student success in higher education: a goal-oriented analysis

Abstract

Student success is becoming a shared vision for quality in higher education. Majority data in higher education have not been transformed into actionable insights for quality enhancement. Data are dispersed among stakeholders, and stakeholders’ data literacy influences the effectiveness of using data for student success. However, existing studies mainly focus on students’ data literacy; the analysis of other stakeholders’ data literacy for student success is still few. This study aims to explore how stakeholders’ data literacy contributes to student success in a holistic view. The salience model is used to identify core stakeholders. The goal-modeling language iStar is used to present how stakeholders contribute to student success. A competencies matrix of data literacy is used to discuss the specific data literacy competencies that stakeholders should focus on promoting student success. A survey is conducted to validate the goal-oriented analysis and the discussions on specific competencies of data literacy for stakeholders. The goal-oriented analysis presents the complexity of interactions and dependencies among stakeholders for student success. This study helps to raise stakeholders to be aware of the importance of their data literacy and the necessity of collaboration on exploiting vast available data for student success.

Introduction

Data are one of the most significant assets in the era of big data, and data literacy empowers us to transform data into actionable insights. Quality is the lifeline of higher education (Bing, 2003). It is crucial to understand the potential of data to enhance quality in higher education. Some positive efforts have been made in learning analytics. Specifically, computer scientists exploit data in massive open online courses and learning management systems to understand and optimize student learning and their learning contexts. These efforts attempt to enhance the quality of teaching and learning - an essential part of quality in higher education. However, the findings offer limited implications in educational practices. It is because computer scientists lack the domain knowledge in educational sciences for correct data interpretation (Al-Shabandar, Hussain, Liatsis, & Keight, 2018; Beneito-Montagut, 2017; Wen, Yang, & Rosé, 2014). Data can enhance quality in several ways. First, data generated in the process of teaching and learning can be used to understand and optimize student learning and their learning contexts (Ferguson, 2012; Brown, 2016). Second, data generated in the operation of HEIs can be used to support decision-making through better understanding the impact of different variables (Buckingham, 2012). Third, data enable governments to understand national higher education ecosystems, affect the resource allocation (Ifenthaler, 2017; Norris, Baer, Leonard, Pugliese, & Lefrere, 2008) and impact on transforming higher education and academic models (Buckingham, 2012).

There is a vast amount of data in the higher education system, which is dispersed among different stakeholders. Existing studies mainly focus on students’ data literacy. Specifically, some studies explore the impact of students’ data literacy for their professional development (Carlson & Johnston, 2015; López-Meneses, Sirignano, Vázquez-Cano, & Ramírez-Hurtado, 2020; Maybee & Zilinski, 2015). Some studies explore faculty members’ perception of data literacy that aims to design courses for students (Burress, Mann, & Neville, 2020). Few studies focus on other stakeholders’ data literacy. Higher education is a complex system. Stakeholders function in different parts of this system, and they interact with each other. Therefore, it is crucial to explore their data literacy in a holistic approach for quality enhancement in higher education. A holistic analysis can reveal interactions and dependencies among stakeholders, which brings insights that are impossible to discover when we only focus on a particular stakeholder. Quality is a value-laden concept, and different people mean different things (Harvey & Green, 1993). Thus, a holistic analysis of stakeholders’ data literacy for quality enhancement in higher education requires a mediated quality shared by stakeholders (Ghislandi, Raffaghelli, & Yang, 2013). In this study, we choose student success as the common goal for stakeholders, for it is emerging as a shared vision of quality in higher education (Coates & Matthews, 2018; Matthews, 2018; Vossensteyn et al., 2015). The next section discusses related studies on stakeholders’ data literacy and the literature gap that triggers the proposed research questions. The methodology section explains the design of the research with frameworks and techniques used in the analysis. The result section presents the research findings. Finally, the last section concludes the study with limitations and future work.

Related studies

Data literacy is an essential ability required in the era of big data; the manipulation of data occurs in daily processes across all sectors and disciplines. Thus, it is becoming a focus in higher education curricula (Prado & Marzal, 2013). Studies on data literacy mainly focus on students. Maybee and Zilinski (2015) argue that students’ data literacy should be taught in close combination with their disciplinary contexts instead of taught as a general skill. In this way, data literacy can not only help students improve academic performance but also provide a sound basis for their professional development.

Similarly, Bracke and Fosmire (2015) present a case study about a student-centered pilot program on data literacy. This program was structured to be flexible to incorporate each student’s particular field of study. Phillips, Fosmire, Turner, Petersheim, and Lu (2019) explore the need for engineering students’ data literacy by comparing it with the experience of engineers. Two online surveys - one for undergraduate engineering students, one for practicing engineers - are designed with similar questions for comparison. The result of this study help identify gaps in preparing new engineering graduates for the requirements of the industry. Besides, some study explores students’ data literacy in general. López-Meneses et al. (2020) explore students’ data literacy at one Italian and two Spanish universities by the questionnaire with 1073 participants. The result shows students, who have a more significant online social presence, develop a more active and cohesive form of communication in group and collaborative learning process. Some study explores the collaboration between librarians and faculty members for designing a curriculum on data literacy. The result shows that the collaboration of librarians and faculty members helps to establish campus-specific data literacy competencies across the curriculum (Burress et al., 2020).

From the existing literature on data literacy, we can draw three points. First, the main research object of data literacy is focused on students, even from the teacher’s point of view, the ultimate goal is to improve the students’ data literacy. Second, the discussion of students’ data literacy for student success mainly focuses on academic learning. Third, collaboration among stakeholders potentially contributes to student learning. We find two literature gaps: first, for the research object, current studies mainly focus on students and ignore other stakeholders. Second, the contribution of data literacy to student success does not cover all the elements of student success. Student success is a broader concept (Respondek, Seufert, Stupnisky, & Nett, 2017; Smallhorn, 2017; York, Gibson, & Rankin, 2015). It also refers to employability (Hinton, Towell, MacFarlane, Refling, & Amesbury, 2017; Kift, 2019; Yorke & Knight, 2007) and well-being (Conner, Pope, & Galloway, 2010; Schreiner, 2010; Zepke & Leach, 2010). Therefore, we think it is valuable to explore various stakeholders’ data literacy for student success. It not only helps to broaden the research object of data literacy but also helps to get a full understanding of how data literacy contributes to student success. The research questions of this study are:

• 1. How do various stakeholders contribute to student success?

• 2. What data do they possess to contribute to student success?

• 3. What data literacy do they require to facilitate their contribution to student success?

Methodology

To answer the research questions mentioned above, we adopted the following frameworks and techniques. First, the salience model is used to select essential stakeholders in higher education for student success, which is a prerequisite for further analysis. Second, the goal-modeling language iStar is used to build a model for presenting how selected stakeholders contribute to student success. Third, a competencies matrix of data literacy is used to explain the requirement of stakeholders’ data literacy according to their contribution to student success. Finally, a survey is used to validate the goal-oriented analysis and the discussion on the requirement of stakeholders’ data literacy.

The salience model for stakeholders

The salience model was developed by Mitchell, Agle, and Wood (1997), which aims to identify stakeholders based on one or more of three attributes they possess: power, legitimacy, and urgency. Power presents that stakeholders can bring about the outcomes they desire. The power can be further categorized into three types in terms of the resources used to exercise power. The first type is coercive power, and it is based on the physical resources of force, violence, or restraint. The second type is utilitarian power, and it is based on material or financial resources. The third type is normative power, and it is based on symbolic resources such as prestige and esteem. Legitimacy is a generalized perception or assumption that the actions of an entity are desirable, proper, or appropriate within some socially constructed system of norms, values, beliefs, and definitions. Urgency is defined as the degree to which stakeholder claims call for immediate attention. Figure 1 presents seven types of stakeholders with a combination of one or more of the three attributes. The salience model, as a typology of stakeholders, offers three attributes to describe the stakeholder-manager relationship (Mitchell et al., 1997). This model is also used in the context of higher education for discussing stakeholder-manager relationships (Lyytinen, Kohtamäki, Kivistö, Pekkola, & Hölttä, 2017; Powell & Walsh, 2018). The challenge of using this model in this study is to adapt it for the research purpose. Three attributes in the model present stakeholders’ potential contributions to student success rather than the stakeholder-manager relationship. Definitive stakeholders, possess all the three attributes, have the most potential to contribute to student success. Students, faculty members, professional and support staff at universities, and employers are selected, for they possess all the attributes in the salience model.

Fig. 1

The salience theory for stakeholders

iStar goal-modeling language for building the model

The iStar modeling language is designed by Yu (1995) as a goal- and actor-oriented modeling and reasoning framework. It consists of a modeling language along with reasoning techniques for analyzing created models (Dalpiaz, Franch, & Horkoff, 2016a). iStar covers a fundamental modeling dimension and supports decision-making and analysis. Thus, researchers adopt it in various contexts (Dalpiaz, Franch, & Horkoff, 2016b, Koch & Landes, 2014; Koch & Landes, 2015). iStar is adopted to build a model to explain how stakeholders interact with each other that contribute to student success. Figure 2 shows the elements of iStar used in this study with explanations. iStar has several benefits. First, it succinctly presents how the actor can achieve the ultimate goal by refining it into several sub-goals. Second, it visually presents the complex interactions among actors through their dependencies on resources that required to achieve (sub-) goals. Third, it offers a holistic view of all the actors and their activities abstracted from reality, which seems to be impossible for other analysis methods that focus on a particular actor.

Fig. 2

Description of iStar elements used in this study

A data literacy competencies matrix

Ridsdale et al. (2015) proposed a data literacy competencies matrix (Table 1). It divides data literacy into four competencies areas with 21 specific competencies. This matrix is adopted in further studies (Pothier & Condon, 2019). Specifically, for data collection, data discovery and collection are to identify useful data and to collect data. Evaluating and ensuring the quality of data and sources is to critically assess the trustworthiness of data sources and the quality of datasets. For data management, data organization is to evaluate the requirements of data organization and to organize data. Data manipulation is to clean data by identifying duplicates, outliers, and anomalies. Data conversion is to convert data from one type to another. Metadata creation and use are to create metadata descriptors and to assign appropriate metadata descriptors to the datasets. Data curation, security, and reuse are to curate data by considering the requirements of data curation (storage, accessibility, and sharing) and data security (such as restricted access, and protected drives). Data preservation is to preserve data according to the requirements of preservation. For data evaluation, data tools refer to the ability to analyze data by selecting appropriate tools and techniques. Basic data analysis is to develop an analysis plan, to conduct exploratory analysis, and to evaluate analysis results. Data interpretation is to identify critical points, to integrate them with other important information, and to identify discrepancies within the data. Identifying problems using data is to identify problems in practical situations and environments of policy, marketing, and economics. Data visualization is to create meaningful tables or graphs and to evaluate the effectiveness and accuracy of graphical representations. Presenting data is to prepare the visualization according to the audience’s needs and to present arguments clearly and coherently. Data-driven decision making is to convert data into actionable information, to consider the impacts of possible decisions, and to implement them. For data application, critical thinking is aware of high-level issues and challenges associated with data and to think critically in working with data. Data culture is to recognize the importance of data and to support an environment that fosters the critical use of data. Data ethics is aware of legal and ethical issues associated with data and to work with data ethically. Data citation is to know widely-accepted data citation methods and to create correct citations for secondary datasets. Data sharing is to assess methods and platforms for sharing data legally and ethically. Evaluating decisions based on data is to assess decisions by conducting follow-up data analysis and comparing analysis results with other findings for retraining original decisions or implementing new ones.

Table 1 A data literacy competencies matrix (Ridsdale et al., 2015)

Survey for validation

A web survey is used in this study to validate the goal-oriented analysis of stakeholders’ data literacy for student success. Questions of the survey are designed based on the results of the goal-oriented analysis. In this study, we directly use the dichotomy form of questions “agree/disagree” instead of the Likert scale to collect stakeholders’ opinion for two reasons. First, Likert scales are affected by the central tendency bias, which reflects participants’ desire to avoid extreme positions in their responses to a particular topic. In contrast, extreme positions are their real positions (Gracyalny, 2017). Second, this survey is disseminated via social media Wechat, and participants will use their mobile phones to answer questions. Because most phones have a much smaller display size than an A4 sheet of paper, a standard five or seven Likert scale might cause difficulties in choosing specific options.

Research findings

This section is composed of three parts. The first part is a holistic view of goal-oriented analysis on stakeholders’ data literacy for student success. The second part is to zoom into each stakeholders’ goal and their contributions to student success. The third part is a validation of this analysis by the survey.

A holistic view of goal-oriented analysis

Figure 3 shows the holistic view of goal-oriented analysis, where is composed of four types of stakeholders in higher education: student, faculty member, professional and support staff, and employer. In Fig. 3, each stakeholder’s goal is highlighted in yellow while their sub-goals are in green. The goals and sub-goals are insider each stakeholder’s boundary (grey circle with a dotted black line). For students, student success is the goal, and it is composed of well-being, academic performance, and being employable. These three elements respectively represent the expected learning outcome in terms of the condition of an adult, of a graduate, and a prospective employee. Well-being refers to diverse and interconnected dimensions of physical, mental, and social well-being that extend beyond the traditional definition of health (Naci & Ioannidis, 2015). Academic performance is a core part of student success in terms of student learning. Student engagement that composed of cognitive, behavioral, and emotional dimensions (Blumenstein et al., 2018) is usually considered as a proxy for academic performance. Being employable emphasizes that students are well-prepared to contribute to society. Besides the professional competences that already presented in the academic performance, twenty-first-century skills (also called transversal skills or soft skills) are common competencies for all the students to prepare for employability. Twenty-First-century skills are composed of three areas’ skills (Trilling & Fadel, 2009): (1) learning and innovation, (2) life and career, (3) information, media, and technology skills. Professional and support staff refers to non-faculty members at universities. Their tasks range from clerical activity to high-level managerial activity to student engagement support in regards to learning, personal needs, and transactional activities to knowledge creation support to physical plant maintenance and security (Bossu, Brown, & Warren, 2018). For them, the goal that relevant to student success is quality services, which is composed of a series of specific services such as library service. For faculty members, the goal relevant to student success is quality teaching. It is decomposed into design, teaching and learning process, assessment, and reflective practice according to the previous research (Yang, 2015). Since the goal of the employer is not directly connected to student success, but they do provide resources for other stakeholders. Thus, only resources are presented inside the boundary of the employer. All the resources outside the boundaries are connections among stakeholders. In the next section, we will zoom into each type of stakeholder to understand how they contribute to student success.

Fig. 3

A goal-oriented analysis of definitive stakeholders for student success

Stakeholders’ contribution to student success

This section respectively focuses on particular types of stakeholders to explain what data do they possess and what type of data literacy they need for student success.

Students

Since the goal-oriented analysis is very complicated. Figure 4 highlights some goals and dependencies in Fig. 3 to focus on what data students possess and how they are dependent on other stakeholders. In Fig. 4, we only highlight seven sub-goals as seven types of data (i.e., career counseling results, study counseling results, psychological counseling results, health check result, data literacy training experience, students’ voice on teaching, and learning feedback) contribute to them. Students are the most important type of stakeholders to be discussed for student success because other types of stakeholders are facilitators to help them achieve success. Students’ data literacy for student success can be discussed in two aspects. First, data literacy is a part of twenty-first-century skills that contribute to being employable. Thus, students who have skills in data collection, management, evaluation, and application are more competent than those who do not. The more skills they obtain, the more competitive they are when seeking for an ideal position in the talents market. Second, students possess data on their status (physical, psychological, academic learning, etc.) and their perception of teaching performance. Thus, they should improve the abilities of data sharing with teachers and data-driven decision-making for their time management in order to facilitate the enhancement of teaching quality and a more balanced personal life and academic performance.

Fig. 4

Students for student success

Faculty members

From the perspective of faculty members, they mainly contribute to student success in academic performance. Figure 5 highlights some data possessed by faculty members. In general, faculty members possess two types of data: data about teaching and learning, data about research. These data contribute to students’ learning experience and student engagement. Faculty members should improve the reuse of research data into their teaching practice and the ability of basic analysis and data interpretation for adjusting the instructional design, implementation, and assessment.

Fig. 5

Faculty members for student success

Professional and support staff

Figure 6 shows that professional and support staff provide a large amount of support to student success that we often ignore. Different to faculty members that mainly contribute to academic performance, professional and support staff contribute to all the three aspects of student success. First, they provide a study counseling service, which is helpful for students to have a proper study plan. Second, they offer training that assures students’ being employable. Third, they provide extracurricular activities that enhance students’ social well-being. Professional and support staff possess various types of counseling data, health check data, alumni data, and application and admission data. All these data have the potential to develop a complete student profile and provide more supports and assistance for a better educational experience. Thus, professional and support staff should improve the ability of data management and data analysis (such as identifying problems using data and data-driven decision-making). For example, it is helpful to recognize violent or suicidal students if technicians can filter sensitive words in the Internet records of students.

Fig. 6

Professional and support staff for student success

Employers

The employer is a special type of stakeholder in this analysis, for they do not have their relevant goals for student success (Fig. 7). However, they own resources that support all the other stakeholders’ goals. First, they offer students internship opportunities, and these chances empower students’ life and career skills. Second, professional experts bring interesting content from industries, which enhance students’ cognitive engagement. Third, their employees’ performance can be used as feedback on graduates’ professional development. These data facilitate redesigning the programs for quality education. Therefore, employers should improve their ability of data management (such as data preservation on employees’ performance, students’ internship performance) and data application (such as data culture, data ethics, data sharing). These aspects of data literacy facilitate a deepen collaboration between higher educational institutions and employers for quality education.

Fig. 7

Employers for student success

Validation on the goal-oriented analysis

The survey is designed based on the results of the goal-oriented analysis and discussions on the requirement of data literacies for stakeholders. It was disseminated on social media Wechat. The estimated response rate is 90.83% because some participants who completed the survey also disseminated the survey to their social circles. In total, 218 (male 41.74%, female 58.26%) participants filled this survey. Among these participants, there are 58 faculty members, 35 university students, 26 professional and support staff, and 99 representatives for employers (38 managers and above, 61 employees). More details about participants’ demographic information can be found in appendix A.

In general, the survey result confirms the majority of the goal-oriented analysis and discussions on the requirement of data literacy for stakeholders. Specifically, almost all the questions in the survey get more than 80% positive response. Only the question about counseling data possessed by professional and support staff has a 70.64% positive response. Besides, questions about the requirement of data literacy for stakeholders have a higher positive response by participants in the survey compared to questions about data possessed by stakeholders. Specifically, the former has 92.60% positive responses on average, while the latter has 88.62% positive responses on average.

Since different types of stakeholders answered the same survey, we compare different stakeholders’ positive response rates on each question. Among 21 questions, six questions have a difference in positive response that more than 10%. All the professional and support staff agree that staff in the alumni office possess alumni data. In contrast, only 77.14% of students agree with this statement. 95.96% of employed workers agree that employers possess data about employees’ performance. In comparison, only 84.62% of professional and support staff agree with this statement. 94.29% of students agree that they possess teachers’ performance data. In comparison, only 63.79% of faculty members agree with this statement. These differences among stakeholders’ perception of the data possession might reveal areas of improvement for data sharing and data use. The details of the positive response rate for each question by types of stakeholders are in appendix B.

Conclusion

The vast data in higher education are available for data integration and data analysis to enhance quality in higher education. However, the majority of them have not been transformed into actionable insights for quality enhancement. Stakeholders’ data literacy is an essential topic as they directly influence how they understand their data and how they take advantage of these data. This study aims to explore stakeholders’ data literacy for student success in a holistic view. Based on the salience model of stakeholders, we identified students, faculty members, professional and support staff, and employers as core stakeholders for student success in this study. iStar goal modeling language is used to build a model that presents how stakeholders contribute to student success. It presents the complexity of interactions and dependencies among stakeholders.

For students, data literacy contributes to student success in two ways. First, as a twenty-first-century skill, it directly contributes to students’ employability. Therefore, students should improve data literacy in all its aspects. Second, students possess data on their status and perception of teaching performance. Thus, the ability of data sharing and data-driven decision-making will contribute to the enhancement of teaching quality and a more balanced personal life and academic performance. For faculty members, they contribute to student success by conducting quality teaching practice. They possess their research data and data about teaching and learning. Thus, the ability to reuse of research data into their teaching and the ability of basic analysis and data interpretation all facilitates adapting the design, implementation, and assessment. For professional and support staff, they contribute to student success in terms of academic performance, employability, and well-being. The ability of data management and data analysis helps to develop a complete student profile for offering a better educational experience to students. For employers, although they do not have their relevant goals directly contribute to student success, the data they possess supports all the other stakeholders’ goals. Their abilities in data management and data application will facilitate deepening collaboration between higher educational institutions and employers for quality education and student success.

The significance of this study is to offer a holistic view of how stakeholders’ data literacy contributes to student success. Besides, it also broadens the research objects of data literacy from mainly focusing on students to more stakeholders in higher education. However, this study has several limitations. First, the validation result of survey questions has a bias, for we do not design it with Likert scales. It might make participants lose the opportunities to present their accurate, neutral opinions. Second, professional and support staff is an umbrella term composed of different staff at universities; data they possess are much different. Hence, their data literacy also varied a lot. We group all of them into one type – professional and support staff - in order to analyze with other stakeholders at the same level. Third, the concept of “data possession” is a tricky concept in our research questions. According to the goal-oriented analysis, we consider both the data producer and the data user to possess the data. For instance, students’ voices on teaching are data produced by students and used by teachers to improve their teaching practice. In this case, we consider both of them to possess these data. However, the reality is much complex then what we analyze. Therefore, future research could further proceed with the exploration of stakeholders’ data literacy for student success. Two lines of investigations are possible. The first track is to continue a holistic analysis by adding more dependencies and interactions among stakeholders that ignored in the current analysis. The second track is to study one type of stakeholder for deepening the analysis and clarifying the “possession” between data producer and data user.

Appendix

Table 2 Demographic information on participants

Table 3 A Brief Summary of the Survey Result