E-assessment process: giving a voice to online learners

Scenario

The research is conducted at the Open University of Catalonia that is a fully online university, with an e-learning model defined as “a mode of teaching and learning that can represent all or part of the educational model being applied, which exploits the media and electronic devices to facilitate access, evolution and improvement of the quality of education and training”(Sangrà et al. 2011, p.5). The university bases the assessment of its entire course on a continuous assessment model, which, according to Sangrà & Guàrdia (2005), has to be integrated in the course design from the beginning and thus meet the needs of students to build knowledge step by step.

The experience presented in this paper concretely focuses on the subject “ICT competencies” (ICTC) which is part of all the UOC university programs. This subject helps students to develop key methodologies and skills to work in digital environments from a rational and critical perspective, and its objective is that students begin in a gradual and integrated way with the acquisition of transversal competencies at the UOC; «Use and application of ICT in an academic and professional environment» and «Online team work» (Guitert et al. 2008). These competencies are outlined as: Search and selection of information online, Processing and development of digital information, Presentation and dissemination of digital information, Basic notions of digital technology, Work planning in a virtual environment, Management of a digital project, Communication strategies in the Net, Teamwork in an online environment and Digital attitude (Pérez-Mateo et al. 2014). The methodological approach is project-based learning (Railsback, 2002), concretely, the developing of a collaborative digital project is considered. To undertake it, students form groups of four, and have their own group space which integrates a variety of tools.

In order to develop their projects, students follow the process of the subject based on the subsequent phases (Author et al. 2012) (see Fig. 1).

The starting phase provides an environment to create working teams and perform the initial searches. This assists the students in setting out the theme of the project.

The second phase (structuring) involves making a deeper search for information to structure the project.

Subsequently, (developmental phase) the project is developed and the processing and the development of the gathered information are carried out. Through this step, the first version of the project is achieved.

Finally, the closing and dissemination of the project are done, as well as the sharing and discussing of the final version of the project.

In order to assess the acquisition of competencies during the subject’s development, some assessment criteria were defined and shown in the learning activities.

The assessment activities of most subjects at the UOC are based on continuous assessment and developed online but the final evaluation is carried out by a face-to-face test. In ICTC, all assessment activities (including the final ones) are developed online and from a formative and continuous perspective so it becomes a good example of continuous e-assessment. Bearing this model in mind, assessment becomes, following Delgado, A. Oliver R (2006), progressive, and the teacher can monitor the progress in student learning more and in a better way as it allows for comprehensive assessment.

In addition, the subject shown in this paper is appropriate for students to acquire a relevant role in the assessment process, in that the UOC’s educational model is student-centred. In fact, student learning is assessed from two dimensions provided by continuous assessment: on the one hand, the assessment of the process followed during the development of the activities based on the outcomes of each phase and process monitoring and, on the other hand, the assessment of the final outcome.

The fact of developing a digital project collaboratively facilitates the implication of the students not only in their individual learning process but also in the collective learning process. This is further corroborated when the definition of collaborative work is defined as a “shared, coordinated and interdependent process, in which students work together in order to achieve a common goal in a virtual environment and based on a process of activity, interaction and reciprocity between students, thus facilitating the collaborative construction of meanings and individual progress towards reaching higher levels of development” (Guitert & Pérez-Mateo, 2013, p.24).

This scenario boosts the development of a new teacher’s role as an advisor and facilitator of the learning process (Pérez-Mateo et al. 2014), providing a more active role of the students through reflecting on their own learning process and peer assessment.

Considering this perspective, the concept of 360° e-assessment can be formulated from the bases of 360° Communication, which is “considered as a state of constant dialogue in which organizations take on a Communicator role with their clients (internal and external)”, so it becomes a comprehensive strategy that connects companies constantly with their public, online and offline (López & Martinez 2012) Based on the 360° communication theory, the characteristics of 360° e-assessment can be described as the following (adapted from Curcoll, 2014) (see Fig. 2):

• Strategic: considers the identification of the key elements for improvement based on the acquisition of competencies.

• Integral: assures the integral acquisition of the competencies, considering the set of competencies to be acquired as a whole, and not as a segmented process.

• Holistic: takes into account all the internal agents –understood as work groups-, external –the whole classroom- and the digital environment in order to understand collaborative learning as more than an addition of several parts,

• Transversal: affects all of the learning actions and activities and the interactions that take place during the learning process,

• Coherent: coordinates and gives sense to the whole teaching-learning process; being global, e-assessment takes into account the different processes as interrelated and not isolated, giving coherence to the assessment,

• And dynamic: conceives the learning process as a living system, in constant change and dynamic in which all the agents participate. So assessment is understood as a process of constant interaction among students and between students and teachers that can be redefined depending on the inputs received.

Following these characteristics, the aim of 360° e-assessment is to reinforce and boost the students’ learning process in order to help them to acquire the competencies of the ICT competency course.

In the subsequent sections, some evidence of the active role of the student is given and some of the mentioned characteristics are reinforced such as those elements like being integral, dynamic and coherent.

Students’ involvement in the e-assessment process

In order to understand the role of students during the e-assessment process in ICTC, it is, first of all, necessary to describe the role of the teacher during this process.

During the development of the subject, the teacher carries out a process and final outcome assessment. The teacher monitors and assesses the work developed in each of the groups during the process in order to improve their dynamics. During this assessment, the active role of the students is quite relevant, since the self-assessment they develop in each phase of the digital project is one of the elements that determine the teacher’s post-assessment of their work.

The process assessment is finished with both individual and group feedback: individual feedback is based on the observation of the participation of each student within the group and the group one is based on the monitoring of the group’s dynamics and processes.

In the assessment of the final outcomes, the teacher assesses the final product of each phase following the predetermined criteria shown in the learning activities. This assessment is developed using a set of indicators that are a concretion of the afore-mentioned assessment criteria.

In this scenario, students subsequently develop the assessment activities.

Students participate in both the assessment of the process and the final outcome. The process assessment, developed in the framework of teamwork, is produced in a self-assessment process in which both the student and peer participation in a work group are analysed, thus developing a dynamic peer-assessment.

In addition, a process of reflection about the teamwork is carried out at the end of each phase of the project, based on an interactive process among the members of the group.

The assessment of the results focuses on an interactive assessment (Crisp, 2010) based on the analysis of the projects developed by other groups at the final stage of the course. During this phase, a defence process takes place during which every student participates individually as an evaluator and as part of a group participating in the development of the responses that their specific group receives from other individual evaluators.

The participation in the assessment process provides students with an awareness of their grade of competency acquisition, thereby allowing them to be an assessment agent of their own learning process from different perspectives or dimensions. In consequence, e-assessment becomes, within the ICTC’s subject framework, a very valuable resource to foster students’ implication in their own learning, from both the individual and collective perspective. In addition, the consideration of this dual perspective provides a higher level of personalization of student assessment: each student of the same team can obtain different marks according to the quality of their individual work in the framework of the team. In this context, a network assessment process is created.

The network assessment process becomes tangible during the different phases of the course because there is the group’s internal peer evaluation during the process and during the final phase of the project, and also each student becomes an evaluator of another group’s project, so the group class becomes a network of evaluators in order to improve their learning processes. In the subsequent figure, the different strategies to assure the students’ protagonism during the assessment project are laid out.

Data gathering and analysis

In order to gather the data, a questionnaire was designed and applied once at the end of the first semester of the course 2014–15. The questionnaire was composed of one section in which students rated the whole course (how it helps them to acquire the competencies of the course and its methodology) and another one based on ten statements that students had to rate their grade of agreement with them. In this last section, all items were drawn up considering their satisfaction with the different processes of e-assessment in the subject and with the active role that it implies. All items from both sections were rated using a 5 point Likert scale. In addition, open text questions were included in order to gather students’ testimonials. To validate the questionnaire’s content; it was previously revised by four experts in e-assessment and administered to a reduced sample of students. Once the validation process was completed, the questionnaire was administered to all students.

Considering that the course was undertaken by 2,303 students, we were able to gather a sample of 913 students thus reaching a representative sample of 39.6 %.

In order to complete the data gathered with some testimonials of the teachers, a focus group session involving 15 of them was developed during a coordination meeting.

Quantitative data has been analyzed with descriptive statistical methods (means and standard deviation) and, in order to determine the relationship between the items in the different sections, the Spearman’s r correlation coefficient has been calculated in some cases. The analysis of the items has been complemented with some of the students’ and teachers’ written testimonials.

Process assessment

Given the high ratings of the items of the questionnaire presented, the subsequent results have to be highlighted complementing them with the statements of some students:

Self-assessment gains an important role in the learning process of the students because it lets them reflect on their individual participation during the developing of the activities of the subject and to understand and be aware of the grade of their acquisition of competencies. As one of the students stated: “I was reticent to assess myself at first, but now I think it helped me to improve how I worked with my teammates and helped me to learn better” (Student 3), so most of the students do not agree with the idea of self-assessment at first, but they finally understand the benefits of it. In fact, the correlation between the students’ perception about their acquisition of the competences and about how self-assessment makes them aware of this acquisition is positive and statistically significant (Spearman’s r = +0.412, p < 0 0001, n = 913). So most of them believe that the reflection they develop during the self-assessment activities helps them to understand their learning process: “the fact of assessing my activities helped me to understand what I learnt and in what I have to improve during the next phase of the course” (Student 4).

Peer-assessment is seen by the students as an instrument that lets them assess their peers using objective criteria. It is important to bear in mind that this assessment process lets students reflect on their own role as a component of the group through comparing their actions with the actions of their peers, so it enriches their learning processes by helping them understand the role of their peers. Most students are not at first comfortable with assessing their peers, but they finally understand the benefits of this activity: “Assessing my teammates made me see that I was doing better than I thought” (Student 5), and for some of them, it is an instrument to critically analyse their activity during the course: “assessing my partners made me see that I had to be more involved in the development of the activities; I understand if my mates don’t give me the maximum score” (Student 6). According to teachers, this is the most complicated assessment activity to perform: “Sometimes it is difficult to convince students to do the peer assessment activities; they tell us that it’s not their duty to assess their peers, but after explaining its benefits, most of them end up being true believers” (Teacher 2).

As stated in previous sections, each group of students reflect on the process of the teamwork at the end of each phase of the digital process. This reflection lets students know each other more deeply and to understand their collaboration process, and their dynamics and interaction as some of the students stated: “the reflection at the end of each activity was a little repetitive at first but, thinking about it now, It helped us to solve our weaknesses as a group” (Student 7), and, in some cases, makes them improve their group’s dynamics: “The last part of each activity makes us reflect and discuss more deeply what we were doing as a group. In fact, in the case of the second activity, we decided to modify our group’s agreements because we detected some things that didn’t work” (Student 8). The teacher gives them some items in which they can base this reflection on but it is only a guide that students can have to enrich their experience as part of a group. Considering the teachers’ perspective, this activity provides evidence for them to assess their students: “the fact of making them reflect in groups about their teamwork makes it easier for me to assess them, because I can see more clearly what they have done” (Teacher 4).

Students highly value teacher’s feedback irrespective if it is done individually or in group form, as they stated: “We had some problems within our group and the feedback from our teacher was very useful to improve the way we were working” (Student 9). It is important to highlight how some students value the quality of the teacher’s feedback: “the teacher’s feedback was very thorough. It let me know what I was doing right and what I was doing wrong” (Student 10).

The monitoring of the groups provides the teacher sufficient information to adequately assess the group and highlight the most relevant interactions within each group, in that he/she gets to know the group’s dynamics and uses his/her expertise to give advice to the students in order to improve it. Some of our teaching staff stated during the focus group that “if I couldn’t watch what the different groups of students do in the virtual campus, I couldn’t assess them. In fact, when some of them got in touch with each other outside the campus, I ask them to send me a report in order to figure out what they had been discussing” (Teacher 5).

Final outcome assessment

Students consider that the teacher has an important role in improving their projects, due to the fact that they highly rate group feedback, and most students stated that this feedback is very valuable for their learning process and they take this advice into serious consideration as they develop the project through its different phases: “the teacher gives us very good advice which makes our project easier to understand” (Student 11), and, once again, they rely on the quality of this feedback: “the teacher’s feedback was very detailed and analysed every part of our project, thanks to him, it looks much better now” (Student 3).

Not only is the feedback given by the teacher highly rated by the students, they think that the assessment of other projects gives them a critical perspective of their own projects by comparing their final result with the results of other groups. Some students stated that “some of the assessments of my classmates were very valuable to us! They really were able to find good points!” (Student 11). Some of them simply appreciate the change to look into other projects: “it was great to see other projects besides ours!” (Student 5); but, some of them, point to this activity as an instrument to critically analyse their own work: “I saw one that really impressed me and make me think how we could improve ours to be excellent” (Student 12); “After I analysed the project the teacher assigned me, I understood what we could improve the format and the content of our project” (Student 13).