Aligning learning outcomes and assessment methods: a web tool for e-learning courses

Abstract

A learning outcome approach to teaching and a learning-oriented assessment are increasingly framing higher education. In an attempt to facilitate professors’ teaching activities, the TALOE (Time to Assess Learning Outcomes in E-learning) project has developed a web tool that aims to help trainers decide on the assessment methods to use in their online courses. Based on a conceptual model (ALOA) built upon Bloom’s revised taxonomy of learning objectives of the cognitive domain, the web tool automatically aligns learning outcomes stated by a user with recommended assessment methods. The web tool has been developed with the European Commission support within the Lifelong Learning Programme and it is publicly available. This paper presents the background and description of the tool and the results of the testing, where various stakeholders have provided feedback about the easiness and friendliness of the tool, as well as about the utility and quality of the recommendations it provides. Prospective of the web tool, limitations and strengths are highlighted in the paper.

Introduction

The Bologna process and the implementation of the European Higher Education Area (EHEA, 2014) brought a need for a substantive change in the pedagogical model in higher education, requiring improvement of the traditional ways of describing qualifications and qualification structures, modules and programmes. The focus moved from the mere accumulation of knowledge to skills acquisition by students. The result is a shift from a content-based approach to a learning-centred approach, a change from teaching objectives to students’ learning outcomes. Learning outcomes are becoming fundamental for structuring the standards and guidelines of quality assessment in higher education and continuing education institutions in Europe and worldwide. In this context, the assessment of learning outcomes becomes a crucial process in the education system.

With information and communication technologies increasingly used in education, new opportunities for improving teaching, learning and assessment are arising. Time to Assess Learning Outcomes in E-learning – TALOE (http://taloe.up.pt), is a project financed by the European Commission from January 2014 to December 2015 (Ref. 543097-LLP-1-2013-1-PT-KA3-KA3MP), which intends to promote the internal consistency of online courses. The main goal of the international project TALOE is to develop a web-based tool to help teachers and trainers decide on the e-assessment methods to use in their online courses.

To be able to develop the practical tool, the TALOE consortium¹ has set and worked on the following specific tasks:

• 1. Research and selection of innovative e-assessment practices that take advantage of the use of technology.

• 2. Development of a web-based tool that is easy to use by the stakeholders.

• 3. Testing of the implementation of the tool with real case studies.

• 4. Distribution and dissemination of the TALOE tool among stakeholders.

This paper focuses on tasks 2 and 3, i.e. development and testing of the TALOE web-tool.

The first task, selection and analysis of innovative e-assessment practices, was completed during 2014 and has been addressed in previous publications (Lõssenko, 2014; Gil-Jaurena et al. 2015). It consisted on the collection of 18 cases of assessment practice in online courses from different institutions around Europe, mostly from the project partners. They were selected according to a set of criteria, built upon the Standards and Guidelines for Quality Assurance in the European Higher Education Area (ENQA, 2009) and some principles agreed within the project partnership that the selected e-assessment practices had to meet:

• Selected practices should develop assessment of more holistic, complex activities using knowledge and skills in problem-solving or authentic tasks.

• Selected practices should use a diverse range of assessment methods, resulting in qualitative descriptions or judgements and where assessment is often integrated with teaching and learning and may involve students as active participants.

• Selected practices should identify and describe achievements according to relevant criteria and standards.

Those principles refer to “authentic” assessment with a learning-oriented approach (Rodríguez-Gómez & Ibarra-Saiz, 2015). The goal of this selection of case studies was to obtain examples that can be used as a showcase of current practice and also as testing material during the second year of the project (as will be explained in a later section). It as well enabled the insight how much teachers acquainted with learning outcomes and about the relation between learning outcomes and assessment methods. This selection also provided information about assessment methods that are usually used in online courses. The case studies are written on predefined templates and detail information about the learning outcomes, course content, teaching practice and assessment methods. In addition, a separate section was developed to linking the assessment to the selection criteria.

Theoretical framework: the ALOA model

Not all assessment methods are valid for each type of learning outcomes. The TALOE project uses an existing framework called the ALOA model (Aligning Learning Outcomes and Assessment) (Falcão 2013). This model highlights the connection between the intended learning outcomes and the assessment strategy used during a course. So, it provides tools for linking learning outcomes and assessment tasks.

The ALOA model was developed by Falcão (2013) from the concept of alignment (Anderson & Krathwohl, 2001; Biggs & Tang 2007; Bloom, 1956; Boud & Falchikov, 2006), which means that the learning outcomes of a course should be used to define the teaching and learning activities and the assessment tasks. The main principle that supports the ALOA model is that, in order to ensure the validity of assessment in relation to what is intended from a course, it is necessary that the outcomes measured by the assessment tasks are the same as the ones expressed in the learning outcomes. In this sense, the ALOA model matches the basis of learning-oriented assessment (Carless, 2007), “an attempt to emphasise the learning features of assessment and promote their development” (p. 58).

The ALOA model uses the revised version of Bloom’s Taxonomy of the cognitive domain (Anderson & Krathwohl, 2001) to establish the link between the learning outcomes and general assessment methods. Thus, the model has three components:

• a) The learning outcomes:

  • They are “statements of what a learner is expected to know, understand and/or be able to do at the end of a period of learning” (Ministry of Science, Technology and Innovation 2005, p. 29). A learning outcomes approach has become fundamental in the European Higher Education Area (EHEA), the so-called Bologna process.

  • The revised Bloom’s taxonomy of educational objectives (Anderson & Krathwohl, 2001) includes six verbs representing a continuum of increasing cognitive complexity: remember, understand, apply, analyse, evaluate and create. 19 specific cognitive processes identified by these authors in relation to each verb are used in the ALOA model to state learning outcomes (see Fig. 1 and Fig. 3 as an example).

• b) The assessment methods:

  • Based on the work of Brown et al. (1997), the model includes a selection of six categories of general assessment methods:

  • Multiple choice questions (MCQ)

  • Essays

  • Problem solving

  • Practical work

  • Short- answer questions

  • Reflective practice assignments

  • Each of the categories is divided into more specific assessment methods. In the TALOE project, each subcategory was then matched to specific e-assessment practices (Soeiro et al. 2014) (see Fig. 4 as an example).

• c) The alignment:

  • It is the relationship between the first two components. A course is aligned or consistent if the description and classification of the learning outcomes and the assessment tasks match.

Fig. 1

The cognitive process dimension, revised Bloom’s taxonomy. Adapted from Anderson and Krathwohl (2001), 67–68. Source: Iowa State University (2015). http://www.celt.iastate.edu/teaching-resources/effective-practice/revised-blooms-taxonomy/

The TALOE project intends to apply the ALOA model to the specific context of e-learning.

Development and description of the TALOE web tool

The TALOE web tool (available at http://taloetool.up.pt) is being developed to help teachers and educators (users) decide on the assessment strategies that will be used in their online courses. It provides users with a possibility to analyse the provided courses’ or modules’ learning outcomes, and offer the most appropriate e-assessment methods consistent with the intended learning outcomes. The tool is envisioned to be used by teachers/faculty/trainers, either to check if the existing assessment methods of their course or module are consistent with the stated learning outcomes, or to be advised on the most appropriate assessment methods for the existing or new course or module.

The TALOE project partnership produced a first version of the web-tool at the beginning of 2015. At that point it was decided by project partners to simplify the tool procedures during the first phase of testing due to the complexity of the ALOA model (Falcão 2013). The present version of the TALOE web tool does not discriminate between knowledge types, as suggested by the theoretical model. After initial testing was completed (phase 1, explained in the coming section), it has been confirmed that the setup matrix is working properly. The best (most appropriate) e-assessment methods are selected on the base of the absolute matches between input (learning outcome) and the e-assessment methods.

The welcome page of the TALOE web tool (Fig. 2) explains how to use it.

Fig. 2

First page of the TALOE web tool

In the first step (Fig. 3), the user describes the learning outcome that students should achieve in the course or module. The learning outcome should be described in a clear way and kept simple. For example: Apply algorithms based on prime numbers on practical problems.

Fig. 3

Describing the learning outcome in the TALOE web tool

Basic information about writing learning outcomes and the ALOA model are available within the TALOE web tool.

After describing a learning outcome, in the Step 2 the user chooses up to three verbs that best describe it (Fig. 3).

This step also enables the user to check and review the stated learning outcome. After describing the learning outcome with verbs, the user sets the process in motion and receives an assessment advice for the defined learning outcome (Fig. 4). The received outcome provides a list of potential e-assessment methods to use, with the corresponding description.

Fig. 4

The TALOE web tool page with results

A specific section in the TALOE website provides information and examples about assessment methods.

The decision engine consists of the estimation of the score that measures the best match between the cognitive processes submitted by the user and the specific ones of a given assessment method. The assessment methods are selected on the basis of score, calculated as the ratio between the number of matches between input, and the model assessment over the number of the selection of the assessment method.

Based on the received inputs, the tool calculates the best assessment methods for the defined learning outcomes. For each suggested assessment method, a description is provided to give users a better idea on how to use and integrate it in the course.

Therefore, the TALOE web tool can be used in two ways: to check if the existing assessment methods in a existing course are in line with the stated learning outcomes; and to help users decide on the most appropriate assessment method for a new course or module. The TALOE web tool can be used to learn more about learning outcomes, assessment and e-assessment, as it provides guidance to teachers to formulate the learning outcomes in accordance to Bloom’s revised taxonomy (Anderson & Krathwohl, 2001).

Testing of the web tool

One of the commitments and tasks of the project was to test the implementation of the web tool. In this section, we present both the testing method and the results.

Method

The dimensions considered in the evaluation of the tool were:

• Easiness: The TALOE web tool is easy to use

• Friendliness: I can use the TALOE web tool without any written instructions

• Usefulness: The TALOE web tool is useful

• Skilfulness: I quickly became skilful with TALOE web tool

• Quality: The TALOE web tool provides good, quality results

• Agreement: The provided results are in agreement with my plan

• Validity: The TALOE web tool enabled me to better align assessment methods to the learning outcomes

• Help to define assessment methods: The TALOE web tool helps me understand how to define the assessment methods better

• Recommendation: I would recommend the TALOE web tool to a friend

• Changes: What features would you change?

• Missing aspects: What features would you add?

• Removable aspects: What features would you remove?

These criteria were agreed among project partners after some working sessions (face-to-face in the project meeting that took place at the University of Padova (Italy) in February 2015, and virtually in the audio meetings we have held monthly). The instrument used was an evaluation form, which included nine close-ended questions (Likert scale) and three open-ended questions. It was first piloted within the project consortium (phase 1); annex 1 includes the evaluation form used in that first phase. Based on the results of phase 1, the web tool was improved and converted into a web form (phase 2). We used Google forms to collect the information.

The structure of the evaluation form was the following:

• General information about the course/module/lesson: course name, scientific field, programme level, number of credits.

• Evaluation of the TALOE web tool (nine close-ended questions – Likert scale, from 1-totally disagree to 5- totally agree). The questions are above in italics in the list of dimensions.

• Suggestions for improvement (three open-ended questions)

• Demographical questions and contact information (optional) about respondents: institution, country, academic profile.

The evaluation of the TALOE web tool was done in two phases.

• 1) First phase testing was done with the 18 collected case studies collected in the first task of the TALOE project (Lõssenko, 2014, as explained in the introduction) and performed by the project partners to see if the suggested e-assessment methods are closely related to the defined learning outcomes. For each case study one learning outcome was chosen and described with three verbs of the revised Bloom taxonomy (Anderson & Krathwohl, 2001), and then checked with the TALOE web tool for the e-assessment method. Learning outcomes in each case study were chosen randomly. Partners encountered some difficulties when trying to describe a learning outcome with three verbs, as some learning outcomes were not clearly defined. Another difficulty encountered in this testing was that some teachers did not clearly correlate specific e-assessment methods with the learning outcomes, and it was difficult to make comparisons between planned (by teachers) and suggested e-assessment method (by TALOE web tool) in such cases. The information was used to improve the web tool and its usability, and to improve the evaluation form itself. This phase took place in March 2015.

• 2) In order to perform the second phase testing, the evaluation form was added to the TALOE web tool site to make it available online. Second phase testing was done with invited stakeholders. Each project partner invited at least four stakeholders to test the TALOE web tool. Professors, researchers and heads of quality or innovation departments were approached for the testing. In total 42 participants (stakeholders) evaluated the tool. The sample is described in the next section. Feedback was collected from mid April to mid June 2015 (two months).

The participants in the online testing could remain anonymous, as demographical questions were optional, except the question about the home institution. Nevertheless, the majority of participants left their contact and wanted to receive further information about the TALOE project and web tool. This was important since it reveals the relevance of the TALOE project results to the stakeholders and helps the Project Consortium build a database of stakeholders for dissemination and exploitation activities.

Sample description

The profile of the 42 respondents (i.e., stakeholders invited by the project partners) to the evaluation form in phase 2 is presented in Figs. 5, 6, 7, and 8.

Fig. 5

TALOE web tool testers’ scientific field

Fig. 6

TALOE web tool testers’ course level

Fig. 7

TALOE web tool testers’ country

Fig. 8

TALOE web tool testers’ academic profile

Different scientific fields have been represented among the web tool testers: 38 % come from the field of Education, followed by 19 % from Computer Science (Fig. 5). Twelve percent come from Humanities and another 12 % from Social Sciences. Seven percent work in Health Studies and another 7 % in Mathematics.

More than 92 % have tested the TALOE web tool in relation to a higher education level course (Fig. 6). Forty-five percent have tested it in a Bachelor or Undergraduate level course; 31 % in a Master or Postgraduate level course; 14 % in a continuing education level course.

Testers of the web tool work in 11 different countries. Most respondents are from Hungary (24 %), Spain (21 %) and Portugal (14 %). All the project partners’ countries are represented in the sample, plus some stakeholders from other regions not directly involved in the project, such as Germany, Ireland, Finland and Serbia (Fig. 7).

Almost 80 % are professors and/or researchers in a higher education institution (Fig. 8). Thus, they have direct and practical experience in defining learning outcomes and deciding about assessment methods in higher education.

Besides this demographic information that was included in the feedback survey, given the home institution we can state that most respondents (specially those in higher education) have experience in e-learning: some of them work in distance education universities and others use a blended learning mode.

In the next section we present the findings related to each dimension considered in the evaluation of the tool, as explained in “method” section.

Research findings

The testing done in the first phase (with the TALOE case studies) showed that the majority of the teachers have defined the appropriate e-assessment methods for the learning outcomes. In some cases, the TALOE web tool suggested additional possibilities in terms of e-assessment methods. For several cases the TALOE web tool indicated that the existing assessment methods should be revised.

The analysis of received feedback in the second phase (through the web form) from 42 participants is presented in Figs. 9, 10, 11, 12, 13, 14, 15, 16, and 17 and explained in this section. The legend in those figures ranges from 1 (totally disagree) to 5 (totally agree). In the interpretation of results, for calculating percentages we have added replies 1 and 2 as disagreement and replies 4 and 5 as agreement. We have considered reply 3 as neutral.

Fig. 9

Easiness of the TALOE web tool

Fig. 10

Friendliness of the TALOE web tool

Fig. 11

Usefulness of the TALOE web tool

Fig. 12

Skilfulness of the TALOE web tool

Fig. 13

Quality of the TALOE web tool results

Fig. 14

Agreement of the TALOE web tool results with previous assessment plans

Fig. 15

Validity of the TALOE web tool

Fig. 16

Help provided by the TALOE web tool

Fig. 17

Recommendation of the TALOE web tool

The majority of participants found TALOE web tool easy to use (Fig. 9). This is an important feature as it influences teacher’s attitude towards the use of the web tool.

Answers to the question about use of the web tool without written instructions are distributed along all ranges (Fig. 10). The majority of participants (64 %) did not need any written instruction to use the tool. This indicates that the tool is well designed, but also that participants may be experienced in defining learning outcomes and familiar with the revised Bloom taxonomy. Nineteen percent of participants needed additional instructions to be able to use the tool.

Although some participants needed instructions to use the tool (Fig. 10), the majority of respondents found it is very useful (Fig. 11), which is very relevant for the project. Some participants (17 %) did not have an opinion about the usefulness of the tool. This perception can depend on their knowledge and experience in defining learning outcomes and the extent to which they have been achieved. Those more experienced may find it not as useful as they are quite experienced in writing learning outcomes and linking them with teaching and assessment.

Eighty-six percent of respondents found the web tool easy to use or become quickly skilful with it after first use or by reading the posted instructions (Fig. 12). This result is relevant for the project, as it compensates to same extent the difficulties that first users may experience (Fig. 10).

Results provided by the TALOE web tool, i.e, recommendations about assessments methods for each learning outcome, are of good quality for most testers (Fig. 13). Some participants (26 %) could not conclude about the quality of the received results.

The majority of participants (81 %) found that received results are in agreement with their own assessment plans (Fig. 14). Only 5 % of respondents found that the assessment methods recommended by he TALOE web tool don’t match their methods. This is an interesting result that would require further research: at first sight it suggests that professors are correctly aligning their learning outcomes and assessment methods, and that the TALOE web tool identifies this alignment.

In relation to the previous question (Fig. 14), most respondents state that the tool helps them to better align learning outcomes and assessment methods (Fig. 15). It could mean that the tool reinforces the assessment plan they were already using (given that there is a high match between the recommendation provided by the tool and the existing plans, Fig. 14). Almost 30 % of respondents don’t have a clear opinion about this question.

The majority of participants (76 %) find the TALOE web tool can help them to understand how to better define the assessment methods in their courses (Fig. 16).

Finally, the majority of participants would recommend the TALOE web tool to a friend; still, there are some of them (14 %) who are not sure about it (Fig. 17).

The information collected from the stakeholders in the open-ended questions of the TALOE feedback form provides some clues about missing elements or interface recommendations that could help improve the tool. From the 42 respondents, 23 have submitted their comments about changes needed in the web tool, missing aspects and removable aspects.

Regarding changes in the TALOE web tool procedure or interface, some relevant feedback is present in the following quotations:

“The explanations are organized along a two-level description so they are difficult to find” (tester 3)

“Separate steps 1 and 2 (otherwise, people will look into the options of step 2 before filling in step 1) - I did it” (tester 19)

“Possibility to choose more than 3 verbs and split into parts of LO could be useful” (tester 40)

About missing aspects in the web tool, we could highlight a demand for examples of use of the different assessment methods, expressed by the testers in the following terms:

“There should be more detailed descriptions of methods, examples of use” (tester 4)

“Perhaps include some examples to better explain how to relate assessment methods to outcomes and implement them” (tester 17)

“I would some examples of the types of assessment. For example for multiple choice, one or two examples of different MCQ. Add advises how to construct a good question for essay or MCQ” (tester 25)

There are not remarkable removable elements addressed by testers.

In summary, the feedback shows a positive evaluation regarding the usability of the tool and the quality of the output/results, i.e. the e-assessment method suggestions. Invited users found this tool easy to use and useful, but would like to see it improved for more complex testing. These results also confirmed that the web tool provides support and guidance to teachers to formulate the learning outcomes in accordance to Bloom’s revised taxonomy, as intended and planned by the task. It also increases the accuracy of e-assessment methods received by the tool and the alignment between learning outcomes, assessment techniques and teaching methods.

Discussion and conclusions

The European project TALOE has successfully transformed the ALOA model into a web tool that is public and freely available to any interested user. The web tool helps users to align their intended learning outcomes in a course (defined using the Bloom’s revised taxonomy of learning objectives in the cognitive domain) with appropriate assessment methods. After a period of testing of the tool, the evaluation results presented in this paper report that the tool is easy to use, written instructions are helpful for users, the tool is useful for its purposes and provides valuable recommendations. The requested improvements that the TALOE project will try to address are related to two main aspects:

• Improving the interface, usability and instructions. These changes have been taken into account to some extent in the last version of the web tool, which includes information about how to better write learning outcomes, about the ALOA model, defines the diverse types of assessment methods and provides some examples of assessment methods in the field of e-learning. The project consortium is also considering the translation of the web tool into languages other than English, in order to improve usability.

• Increasing the complexity of the web tool, thus providing more complex recommendations, especially examples of use of the diverse assessment methods. Users need more information about how to move beyond the identification of appropriate assessment methods to the implementation phase. The tool will be further developed in phases of complexity of the definition of the procedures relating to the learning outcomes and assessment methods. This is an extra effort to achieve consistency of the web-tool performance and simplicity of procedures by potential users.

Besides this prospective that the TALOE project wishes to achieve, we would like to briefly highlight some concluding remarks about strengths and limitations of the TALOE web tool.

A main limitation that users ought to be aware of is that the tool is focused on the revised version of Bloom’s Taxonomy of the cognitive domain (Anderson & Krathwohl, 2001) (Fig. 1), thus it does not contemplate other learning domains, namely the affective and psychomotor domains (Bloom et al. 1964; Kennedy et al. 2006), and the social domain of learning (Vygotsky, 1978; Wenger, 2000; Garrison & Anderson, 2003). It is important to notice this limitation, so users can be aware that not all the learning outcomes they state for their courses would be represented in the TALOE web tool (but only those in the cognitive domain).

Among the strengths, we would like to highlight the value that testers of the tool have stated and the fact that they would recommend its use to their colleagues. In the basis of the European projects it is the dissemination and exploitation of the developed products. In this sense, we feel encouraged to share the TALOE web tool with any user interested in checking if the existing assessment methods in an existing course are in line with the stated learning outcomes, and in making decisions on the most appropriate assessment methods for a new course or module. An extended use of the TALOE web tool could lead to another research project, focused on analyzing the impact of the tool on the improvement of e-assessment and, furthermore, on exploring the impact of the use of the TALOE web tool on the improvement of learning, which is, at the end, the purpose of the educational experiences we design and develop.