E-assessment process: giving a voice to online learners

Abstract

Considering the expansion of active pedagogies that consider the student as the centre of the teaching and learning process, their role during the assessment process cannot be overlooked. That implies a change in the teacher’s role from a communicator of the student’s mark to an open system in which teachers and students get involved in the learning process and develop online assessment skills.

From a description of a concrete experience of continuous e-assessment in a subject at the UOC, this paper explores the advantages of the active role of students in the e-assessment of competency acquisition. In this subject, the student is assessed, using a project based learning methodology, from two dimensions provided by continuous assessment: on the one hand, the assessment of the process followed during the development of the activities based on the outcomes of each phase of the projects and process monitoring and, on the other hand, the assessment of the final outcome. Hence, students participate in both the assessment of the process and the final outcome. The process assessment, developed in the framework of teamwork, is produced in a self-assessment process in which both the student and peer participation in a work group are analysed, thereby developing a dynamic peer-assessment. In addition, a process of reflection regarding teamwork is carried out at the end of each phase of the project, based on an interactive process among the members of the group.

The evaluation of the results focuses on an interactive assessment based on the analysis of the projects developed by other groups at the final stage of the subject.

In order to analyse the students’ perception about the e-assessment methodology and their role during this process, a quantitative and online questionnaire was designed and administered at the final stage of the course. The results revealed a high level of satisfaction with the e-assessment activities of the course and an improvement of the learning process.

Theoretical framework

In Higher Education (HE), the construction of an integrated European space has been an adaptation challenge for universities (González & Wagenaar, 2003). The students should become the central focus of the teaching and learning process in terms of the organization of the learning strategies, time of teaching, contact time, but also in the assessment methodologies (Ferrão 2010). One good strategy to promote a student-centred educational model is assessment; which can be a core component for effective learning. In fact “teaching and learning processes need to be assessment-centred to provide learners with opportunities to demonstrate their development of skills and receive support to enhance their learning” (Gikandi et al. 2011, p. 2334).

The accelerated evolution of Information and Communication Technologies (ICTs) to learning processes advocates a reconsideration of assessment systems, fostering a change of the teacher’s role from a communicator of the student’s mark to that of an open system in which teachers and students get involved in the learning process and develop online assessment skills. ICTs have turned them into essential tools for improving current assessment strategies. E-assessment is playing an increasingly important role in the transformation of HE about how to use the full potential of e-assessment procedures to transform students’ learning and teacher’s work (Whitelock, 2009).

Comparing it with face-to-face environments, assessment online contexts encompass different characteristics, as stated by Gikandi et al. (2011), given the asynchronous interactivity among online participants (both teachers and learners). This fact has led teachers to rethink their pedagogy to perform assessment activities that support meaningful learning.

Following Osuji (2012), e-assessment could be conceptualized as the use of ICTs to facilitate the entire assessment process, from the designing and delivering of assignments to marking (by computers, or human-assisted by digital tools), reporting, storing the results and/or making the statistical analysis. In other research, Crisp (2010) used the term e-assessment to refer to all the assessment tasks conducted through a computer and/or the web. Besides, e-assessment is playing an increasingly important role in the transformation of HE (Whitelock, 2010), however, new assessment paradigms and methodologies are still needed.

Based on the theoretical principles of assessment in virtual environments, this paper approaches e-assessment (Ibarra Saiz et al. 2012), defined as any assessment process in which ICT is used in order to introduce and carry out assessment activities and tasks, record the answers and evaluate them from different perspectives: learners, teachers, institutions (Joint Information Systems Committee JISC 2007). The variety of applications of e-assessment and its innovation and efficiency reaffirm its potential as a booster of the learning process and learner outcomes.

Pachler et al. (2010) stress the importance of formative e-assessment in which one of its strategies is to enable students to take over their own learning process and to become a resource for other students. Consequently, in this paper, the advantages of the active role of students in the e-assessment of their competency acquisition will be explored.

Some authors (Beaumont, et al., 2011; Ibarra Saiz et al. 2012) stress on a form of e-assessment based on the active role of the student and some of them, as Carless et al. (2011), highlight the necessity of this role to avoid problems during the assessment process, given the fact that students often view comments by teachers on their work as difficult to understand; lacking specific advice on how to improve; or difficult to act upon. Many of these problems are accentuated by the one-way feedback process, so the student role in the feedback process also needs to be enhanced.

The feedback process can also be undertaken by students and is not only an individual activity, in that peer feedback during the e-assessment process has some benefits for students as stated by Guasch et al. (2013) and Cheng et al. (2015), reinforcing a kind of feedback that provides a positive feeling or recognition of the work and suggestive feedback provided by peers are most useful for subsequent learning. In some cases, this type of feedback can be seen as equivalent to instructor feedback in terms of reliability and value.

Methodological approach

Bearing in mind that this paper analyses the role of students in e-assessment activities centred in a concrete course and their perception about this process, the following research questions are considered:

How students appraise the different e-assessment strategies of the course?

What do these assessment strategies that involve an active role contribute to the learning process?

We developed a quantitative and qualitative research, the design of which is explained in the following sections.

Scenario

The research is conducted at the Open University of Catalonia that is a fully online university, with an e-learning model defined as “a mode of teaching and learning that can represent all or part of the educational model being applied, which exploits the media and electronic devices to facilitate access, evolution and improvement of the quality of education and training”(Sangrà et al. 2011, p.5). The university bases the assessment of its entire course on a continuous assessment model, which, according to Sangrà & Guàrdia (2005), has to be integrated in the course design from the beginning and thus meet the needs of students to build knowledge step by step.

The experience presented in this paper concretely focuses on the subject “ICT competencies” (ICTC) which is part of all the UOC university programs. This subject helps students to develop key methodologies and skills to work in digital environments from a rational and critical perspective, and its objective is that students begin in a gradual and integrated way with the acquisition of transversal competencies at the UOC; «Use and application of ICT in an academic and professional environment» and «Online team work» (Guitert et al. 2008). These competencies are outlined as: Search and selection of information online, Processing and development of digital information, Presentation and dissemination of digital information, Basic notions of digital technology, Work planning in a virtual environment, Management of a digital project, Communication strategies in the Net, Teamwork in an online environment and Digital attitude (Pérez-Mateo et al. 2014). The methodological approach is project-based learning (Railsback, 2002), concretely, the developing of a collaborative digital project is considered. To undertake it, students form groups of four, and have their own group space which integrates a variety of tools.

In order to develop their projects, students follow the process of the subject based on the subsequent phases (Author et al. 2012) (see Fig. 1).

Fig. 1

Dimensions of continuous assessment in ICTC

The starting phase provides an environment to create working teams and perform the initial searches. This assists the students in setting out the theme of the project.

The second phase (structuring) involves making a deeper search for information to structure the project.

Subsequently, (developmental phase) the project is developed and the processing and the development of the gathered information are carried out. Through this step, the first version of the project is achieved.

Finally, the closing and dissemination of the project are done, as well as the sharing and discussing of the final version of the project.

In order to assess the acquisition of competencies during the subject’s development, some assessment criteria were defined and shown in the learning activities.

The assessment activities of most subjects at the UOC are based on continuous assessment and developed online but the final evaluation is carried out by a face-to-face test. In ICTC, all assessment activities (including the final ones) are developed online and from a formative and continuous perspective so it becomes a good example of continuous e-assessment. Bearing this model in mind, assessment becomes, following Delgado, A. Oliver R (2006), progressive, and the teacher can monitor the progress in student learning more and in a better way as it allows for comprehensive assessment.

In addition, the subject shown in this paper is appropriate for students to acquire a relevant role in the assessment process, in that the UOC’s educational model is student-centred. In fact, student learning is assessed from two dimensions provided by continuous assessment: on the one hand, the assessment of the process followed during the development of the activities based on the outcomes of each phase and process monitoring and, on the other hand, the assessment of the final outcome.

The fact of developing a digital project collaboratively facilitates the implication of the students not only in their individual learning process but also in the collective learning process. This is further corroborated when the definition of collaborative work is defined as a “shared, coordinated and interdependent process, in which students work together in order to achieve a common goal in a virtual environment and based on a process of activity, interaction and reciprocity between students, thus facilitating the collaborative construction of meanings and individual progress towards reaching higher levels of development” (Guitert & Pérez-Mateo, 2013, p.24).

This scenario boosts the development of a new teacher’s role as an advisor and facilitator of the learning process (Pérez-Mateo et al. 2014), providing a more active role of the students through reflecting on their own learning process and peer assessment.

Considering this perspective, the concept of 360° e-assessment can be formulated from the bases of 360° Communication, which is “considered as a state of constant dialogue in which organizations take on a Communicator role with their clients (internal and external)”, so it becomes a comprehensive strategy that connects companies constantly with their public, online and offline (López & Martinez 2012) Based on the 360° communication theory, the characteristics of 360° e-assessment can be described as the following (adapted from Curcoll, 2014) (see Fig. 2):

• Strategic: considers the identification of the key elements for improvement based on the acquisition of competencies.

• Integral: assures the integral acquisition of the competencies, considering the set of competencies to be acquired as a whole, and not as a segmented process.

• Holistic: takes into account all the internal agents –understood as work groups-, external –the whole classroom- and the digital environment in order to understand collaborative learning as more than an addition of several parts,

• Transversal: affects all of the learning actions and activities and the interactions that take place during the learning process,

• Coherent: coordinates and gives sense to the whole teaching-learning process; being global, e-assessment takes into account the different processes as interrelated and not isolated, giving coherence to the assessment,

• And dynamic: conceives the learning process as a living system, in constant change and dynamic in which all the agents participate. So assessment is understood as a process of constant interaction among students and between students and teachers that can be redefined depending on the inputs received.

Fig. 2

360° e-assessment characteristics

Following these characteristics, the aim of 360° e-assessment is to reinforce and boost the students’ learning process in order to help them to acquire the competencies of the ICT competency course.

In the subsequent sections, some evidence of the active role of the student is given and some of the mentioned characteristics are reinforced such as those elements like being integral, dynamic and coherent.

Students’ involvement in the e-assessment process

In order to understand the role of students during the e-assessment process in ICTC, it is, first of all, necessary to describe the role of the teacher during this process.

During the development of the subject, the teacher carries out a process and final outcome assessment. The teacher monitors and assesses the work developed in each of the groups during the process in order to improve their dynamics. During this assessment, the active role of the students is quite relevant, since the self-assessment they develop in each phase of the digital project is one of the elements that determine the teacher’s post-assessment of their work.

The process assessment is finished with both individual and group feedback: individual feedback is based on the observation of the participation of each student within the group and the group one is based on the monitoring of the group’s dynamics and processes.

In the assessment of the final outcomes, the teacher assesses the final product of each phase following the predetermined criteria shown in the learning activities. This assessment is developed using a set of indicators that are a concretion of the afore-mentioned assessment criteria.

In this scenario, students subsequently develop the assessment activities.

Students participate in both the assessment of the process and the final outcome. The process assessment, developed in the framework of teamwork, is produced in a self-assessment process in which both the student and peer participation in a work group are analysed, thus developing a dynamic peer-assessment.

In addition, a process of reflection about the teamwork is carried out at the end of each phase of the project, based on an interactive process among the members of the group.

The assessment of the results focuses on an interactive assessment (Crisp, 2010) based on the analysis of the projects developed by other groups at the final stage of the course. During this phase, a defence process takes place during which every student participates individually as an evaluator and as part of a group participating in the development of the responses that their specific group receives from other individual evaluators.

The participation in the assessment process provides students with an awareness of their grade of competency acquisition, thereby allowing them to be an assessment agent of their own learning process from different perspectives or dimensions. In consequence, e-assessment becomes, within the ICTC’s subject framework, a very valuable resource to foster students’ implication in their own learning, from both the individual and collective perspective. In addition, the consideration of this dual perspective provides a higher level of personalization of student assessment: each student of the same team can obtain different marks according to the quality of their individual work in the framework of the team. In this context, a network assessment process is created.

The network assessment process becomes tangible during the different phases of the course because there is the group’s internal peer evaluation during the process and during the final phase of the project, and also each student becomes an evaluator of another group’s project, so the group class becomes a network of evaluators in order to improve their learning processes. In the subsequent figure, the different strategies to assure the students’ protagonism during the assessment project are laid out.

Data gathering and analysis

In order to gather the data, a questionnaire was designed and applied once at the end of the first semester of the course 2014–15. The questionnaire was composed of one section in which students rated the whole course (how it helps them to acquire the competencies of the course and its methodology) and another one based on ten statements that students had to rate their grade of agreement with them. In this last section, all items were drawn up considering their satisfaction with the different processes of e-assessment in the subject and with the active role that it implies. All items from both sections were rated using a 5 point Likert scale. In addition, open text questions were included in order to gather students’ testimonials. To validate the questionnaire’s content; it was previously revised by four experts in e-assessment and administered to a reduced sample of students. Once the validation process was completed, the questionnaire was administered to all students.

Considering that the course was undertaken by 2,303 students, we were able to gather a sample of 913 students thus reaching a representative sample of 39.6 %.

In order to complete the data gathered with some testimonials of the teachers, a focus group session involving 15 of them was developed during a coordination meeting.

Quantitative data has been analyzed with descriptive statistical methods (means and standard deviation) and, in order to determine the relationship between the items in the different sections, the Spearman’s r correlation coefficient has been calculated in some cases. The analysis of the items has been complemented with some of the students’ and teachers’ written testimonials.

Results

According to the questionnaire, the results regarding student’s satisfaction with the continuous assessment of the course are highly rated, obtaining a mean of 4.40 (SD. 0.845), placing all the means above 4 in a scale from 1 to 5.

So most of the students are satisfied, in general terms, with the course’s evaluation model based on continuous assessment. As some students stated, this model helped them to improve their learning despite its grade of thoroughness: “Even though it is a very demanding course, our teacher was very fair in her assessments; the fact of being continuously assessed makes you have to work harder and consequently learn more” (Student 1). Other students directly explain their satisfaction with the assessment of the course: “continuous assessment is fine with me; I think it is coherent with the activities developed during the course; it helped me to learn” (Student 2). Some teachers state their satisfaction with this type of assessment as well: “It’s a more time consuming assessment but I think that the fact of assessing students during the different activities improves their learning“ (Teacher 1).

In order to determine the coherence between the students’ perception about the methodology of the course and the assessment model, a statistical correlation was calculated between both items of the questionnaire and results were positively related (Spearman’s r = +0.471, p < 0 0001, n = 913).

As explained in the previous sections, the assessment criteria have been shown to students in the learning activities. This fact is highly rated by the students (Mean: 4.45, SD: 0.822) because they agree that it helps them to understand the learning activities and what the teacher expects from them.

Process assessment

Regarding the assessment of the process, our research presented the subsequent results (Table 1):

Table 1 Means and SD of process’ assessment items in the course’s questionnaire

Given the high ratings of the items of the questionnaire presented, the subsequent results have to be highlighted complementing them with the statements of some students:

Self-assessment gains an important role in the learning process of the students because it lets them reflect on their individual participation during the developing of the activities of the subject and to understand and be aware of the grade of their acquisition of competencies. As one of the students stated: “I was reticent to assess myself at first, but now I think it helped me to improve how I worked with my teammates and helped me to learn better” (Student 3), so most of the students do not agree with the idea of self-assessment at first, but they finally understand the benefits of it. In fact, the correlation between the students’ perception about their acquisition of the competences and about how self-assessment makes them aware of this acquisition is positive and statistically significant (Spearman’s r = +0.412, p < 0 0001, n = 913). So most of them believe that the reflection they develop during the self-assessment activities helps them to understand their learning process: “the fact of assessing my activities helped me to understand what I learnt and in what I have to improve during the next phase of the course” (Student 4).

Peer-assessment is seen by the students as an instrument that lets them assess their peers using objective criteria. It is important to bear in mind that this assessment process lets students reflect on their own role as a component of the group through comparing their actions with the actions of their peers, so it enriches their learning processes by helping them understand the role of their peers. Most students are not at first comfortable with assessing their peers, but they finally understand the benefits of this activity: “Assessing my teammates made me see that I was doing better than I thought” (Student 5), and for some of them, it is an instrument to critically analyse their activity during the course: “assessing my partners made me see that I had to be more involved in the development of the activities; I understand if my mates don’t give me the maximum score” (Student 6). According to teachers, this is the most complicated assessment activity to perform: “Sometimes it is difficult to convince students to do the peer assessment activities; they tell us that it’s not their duty to assess their peers, but after explaining its benefits, most of them end up being true believers” (Teacher 2).

As stated in previous sections, each group of students reflect on the process of the teamwork at the end of each phase of the digital process. This reflection lets students know each other more deeply and to understand their collaboration process, and their dynamics and interaction as some of the students stated: “the reflection at the end of each activity was a little repetitive at first but, thinking about it now, It helped us to solve our weaknesses as a group” (Student 7), and, in some cases, makes them improve their group’s dynamics: “The last part of each activity makes us reflect and discuss more deeply what we were doing as a group. In fact, in the case of the second activity, we decided to modify our group’s agreements because we detected some things that didn’t work” (Student 8). The teacher gives them some items in which they can base this reflection on but it is only a guide that students can have to enrich their experience as part of a group. Considering the teachers’ perspective, this activity provides evidence for them to assess their students: “the fact of making them reflect in groups about their teamwork makes it easier for me to assess them, because I can see more clearly what they have done” (Teacher 4).

Students highly value teacher’s feedback irrespective if it is done individually or in group form, as they stated: “We had some problems within our group and the feedback from our teacher was very useful to improve the way we were working” (Student 9). It is important to highlight how some students value the quality of the teacher’s feedback: “the teacher’s feedback was very thorough. It let me know what I was doing right and what I was doing wrong” (Student 10).

These three elements previously presented have a collaborative perspective. Having this fact in mind, the correlation is done between the items of the questionnaire related to peer assessment and the assessment of group dynamics during the process and the student’s perception about the acquisition of the competency “online teamwork”. It can be seen, once more, that there is a significant statistical relationship among them, so the students that were satisfied with teachers’ group feedback, peer assessment and group reflection think they highly acquired this competency (Table 2).

Table 2 Spearman’s r between process’ assessment and their level of acquisition of the competency “Online Teamwork”

The monitoring of the groups provides the teacher sufficient information to adequately assess the group and highlight the most relevant interactions within each group, in that he/she gets to know the group’s dynamics and uses his/her expertise to give advice to the students in order to improve it. Some of our teaching staff stated during the focus group that “if I couldn’t watch what the different groups of students do in the virtual campus, I couldn’t assess them. In fact, when some of them got in touch with each other outside the campus, I ask them to send me a report in order to figure out what they had been discussing” (Teacher 5).

Final outcome assessment

Regarding the final outcome assessment, students are, as can be seen in Table 3, in general, highly satisfied about how this kind of assessment helps them to improve the products of each phase of the process.

Table 3 Items related to the assessment in the course’s questionnaire

Students consider that the teacher has an important role in improving their projects, due to the fact that they highly rate group feedback, and most students stated that this feedback is very valuable for their learning process and they take this advice into serious consideration as they develop the project through its different phases: “the teacher gives us very good advice which makes our project easier to understand” (Student 11), and, once again, they rely on the quality of this feedback: “the teacher’s feedback was very detailed and analysed every part of our project, thanks to him, it looks much better now” (Student 3).

Not only is the feedback given by the teacher highly rated by the students, they think that the assessment of other projects gives them a critical perspective of their own projects by comparing their final result with the results of other groups. Some students stated that “some of the assessments of my classmates were very valuable to us! They really were able to find good points!” (Student 11). Some of them simply appreciate the change to look into other projects: “it was great to see other projects besides ours!” (Student 5); but, some of them, point to this activity as an instrument to critically analyse their own work: “I saw one that really impressed me and make me think how we could improve ours to be excellent” (Student 12); “After I analysed the project the teacher assigned me, I understood what we could improve the format and the content of our project” (Student 13).

Conclusions

Bearing in mind the research questions previously presented, we can conclude, regarding the first one, that students highly rate the fully online continuous evaluation model for their competency acquisition, which is coherent with their perception about the fact that the quality criteria are shown in the learning activities, and the teacher’s role during the process and final assessment (both individual and group) helps them to understand and promote their learning process. This fact can be a key to reconceptualise the way of giving feedback to students, in order to make them follow the learning process (Beaumont et al. (2011). In a fully online educational context and, considering that the students enroll in the ICTC subject during their first semester in the university, the feedback given by the teacher is a key element to avoid student dropout.

In fact, the continuous development of assessment activities is very highly rated by most of the students, due to the fact that they stated that it improves their learning process. One of the reasons that make it easier for students to participate in assessment activities during the course is the fact that some of these activities are developed during the different phases of the project, so, in the same sense as the experiences developed by Ibarra Saiz et al. (2012), the continuous assessment during the course promotes that students find self and peer-assessment easier to perform.

Regarding the second research question, students stated a high satisfaction to the fact of being actively involved in the assessment process and, in consequence, with their benefits for their own learning process. As seen in the results section, they highly rate the following activities:

• Self-assessment as a resource to reflect on their own competency acquisition and to improve their learning process. So, data analysed supports the hypothesis of students being more aware of their competency acquisition through self-assessment. As other research developed in other educational levels (Brown & Harris, 2013) and in Higher education (Boud 2003; Falchikov, 2005; Boud et al., 2013; Martínez-Figueira et al., 2013), our work demonstrates that self-assessment makes students more involved in their own learning processes.

• Peer-assessment processes as a resource that allows them to:

  • o Promote the objective assessment among group peers, and, as students stated, improve the group dynamics since it highlights the detection of any dysfunction in the group. In fact, as some authors mention at some point (Hatzipanagos & Rochon, 2011; Gikandi et al. 2011), peer assessment becomes an incentive in order to improve the work of the group and individual efforts and the involvement of students increases when teacher shares the responsibility of the assessment with the students.

  • o Improve their individual activity in the framework of the group. Considering that most of the course is developed in groups, it is important to highlight how student’s individual learning is increased through collaborative work and assessment. As some students’ statements and some research pointed out (Carless et al. 2010; Boud et al., 2013; Demiraslan et al. 2015), the fact of being assessed by peers makes students more aware of their contributions to group work, so they can see ways of increasing their involvement during group activities.

  • o Improve their products during the learning process since it reinforces group dynamics and the relationship with peers through both reflection and discussion processes. According to Hwang et al. 2014, peer-assessment activities make students improve both their processes and products. Our students, like other research participants, consider peer-assessment as a valuable educational experience that helped them to provide their learning processes with a better structure, a feature that is reflected in their final products. As stated by our students in the same sense of the research developed by Prins et al. (2005), they positively value if their peers develop sincere assessment according to their work, even if it is not positive.

• And the interactive assessment developed at the end of the course has two clear advantages for the students:

  • o The assessment of other projects promotes the critical analysis of students’ own projects.

  • o The assessment received from peers of students’ projects helps them to detect their own project’s weaknesses.

So, this activity makes them become, like in the case of similar works in Higher Education (KulKarni et al., 2014), better judges of their own work. In fact, as some statements of the students shown highlighted that it makes them be more critical about their own projects and their work as a team.

In addition, the assessment process of the course is carried out from different perspectives, so the development of these activities becomes a validation process in itself. In fact, as stated by Martínez-Figueira et al. (2013) students consider that it is necessary to integrate different assessment procedures which provide additional information in order to enrich the valuation; so they consider that the development of different activities and assessment roles is the best perspective to approach the assessment process.

The carrying out of collaborative activities in digital environments “facilitates the gathering of evidence of the interactions and the process of a shared construction of knowledge” (Guitert & Pérez-Mateo, 2013). In that sense, the realization of an online assessment process allows teachers to triangulate assessment from the interactions in the common spaces of the online environment, fostering the individual assessment of both the process and the final outcomes, based on the participation of all the afore-mentioned agents, creating a network assessment process.

All of the elements highlighted reinforce the 360° vision of e-assessment, in that it helps students to be more aware of their acquisition of competencies throughout their learning process. In fact, the features of this model and the active role of students during their e-assessment process provide evidence of the strategic, holistic and transversal characteristics explained in a previous section. So the 360° e-assessment model considers different perspectives regarding who performs the process (teacher and student), how they do it (individual and group work), what is assessed (process and results), and where the e-assessment is performed (group space and online classroom).

These conclusions are transferable to other teaching and learning processes that can become recommendations that can be applied in the design of learning of any subject based on continuous e-assessment.

Further steps will be directed to systematize this e-assessment model in order to apply it to other learning situations.

So the 360° model is a methodological proposal that improves and visualizes the learning results, and hence provides more evidence of the process. In fact, it is a way of give a voice to students in the learning process.