Skip to main content
International Journal of Educational Technology in Higher Education
Download PDF

Ascertaining the Relevance of Open Educational Resources by Integrating Various Quality Indicators

Determinando la relevancia de los recursos educativos abiertos a través de la integración de diferentes indicadores de calidad

International Journal of Educational Technology in Higher Education volume 8, pages 211–224 (2011)Cite this article

Abstract

The aim of the open educational resource (OER) development movement is to provide free access to high-quality educational materials in repositories. However, having access to a large amount of educational materials does not provide any assurance of their quality, and the mechanisms so far used to recommend educational resources have shown themselves to be lacking for a variety of reasons. Most evaluation systems are based on a costly manual inspection, which does not allow all materials to be evaluated. Moreover, it is often the case that other useful pieces of information are ignored, such as the use that users make of the materials, the evaluations that users perform on them and the metadata used to describe them. To try and improve this situation, this article presents the shortcomings of existing proposals and identifies every possible quality indicator that is able to provide the necessary information to enable materials to be recommended to users. By studying a significant set of materials contained in the MERLOT repository, the relationships among various, currently available quality indicators were analysed and numerous correlations among them were established. On the basis of that analysis, a measure of relevance is proposed, which integrates all existing quality indicators. Thus, the explicit evaluations made by users or experts, the descriptive information obtained from metadata and the data obtained from the use of the latter are employed in order to increase the reliability of recommendations by integrating various quality aspects. In addition, this measure is sustainable because it can be calculated automatically and does not require human intervention; this will allow all educational materials located in repositories to be rated.

Resumen

El propósito del movimiento de desarrollo de recursos educativos abiertos es proporcionar libre acceso a materiales educativos de alta calidad disponibles en repositorios. Sin embargo, tener acceso a una gran cantidad de materiales educativos no garantiza que estos sean de calidad, y los mecanismos empleados para recomendar los recursos educativos utilizados hasta la fecha se han mostrado insuficientes por diferentes motivos. La mayoría de los sistemas de evaluación están basados en una costosa inspección manual que no permite tener evaluados todos los materiales; además, muchas veces no se tienen en cuenta otras informaciones útiles como la utilización que hacen los usuarios de los materiales, las evaluaciones hechas por los usuarios y los metadatos que describen el material educativo. Para intentar mejorar esta situación, en este documento se exponen las carencias de las propuestas existentes y se identifican todos los posibles indicadores de calidad que pueden aportar información sobre qué materiales recomendar a los usuarios. A través del estudio de un conjunto significativo de materiales del repositorio Merlot se analizan las relaciones existentes entre los distintos indicadores de calidad disponibles, para constatar que existen numerosas correlaciones entre ellos. Posteriormente y a partir de este análisis, se propone una medida de relevancia que integre todos los indicadores de calidad existentes. De esta manera se utilizarán las evaluaciones explicitas realizadas por usuarios o expertos, la información descriptiva proveniente de los metadatos y los datos que proceden del uso de estos, para lograr aumentar la fiabilidad de las recomendaciones al integrar diferentes perspectivas de la calidad. Además, como esta medida se puede calcular de forma automática se garantizará su sostenibilidad, ya que no necesitará de la intervención humana para su cálculo, lo que permitirá que todos los materiales educativos ubicados en repositorios estén valorados.

References

  • AKPINAR, Y. (2008). “Validation of a Learning Object Review Instrument: Relationship between Ratings of Learning Objects and Actual Learning Outcomes”. Interdisciplinary Journal of Knowledge and Learning Objects. Vol. 4, pages 291–302.

    Google Scholar 

  • BARANIUK, R. G. (2007). “Challenges and Opportunities for the Open Education Movement: A Connexions Case Study”. In: Opening Up Education: The Collective Advancement of Education through Open Technology, Open Content, and Open Knowledge. California: MIT Press. Pages 116–132.

    Google Scholar 

  • BOSKIC, N. (2003). Faculty Assessment of the Quality and Reusability of Learning Objects. PhD thesis. Alberta (Canada): Athabasca University.

    Google Scholar 

  • BROWNFIELD, G.; OLIVER, R. (2003). “Factors influencing the discovery and reusability of digital resources for teaching and learning”. In: G. Crisp, D. Thiele, I. Scholten et al. (eds.). Interact, Integrate, Impact: Proceedings of the 20th Annual Conference of the Australasian Society for Computers in Learning in Tertiary Education. Adelaide. pages 74–83.

  • CLAYPOOL, M.; LE, P.; WASED, M. et al. (2001). “Implicit interest indicators”. Proceedings of the 6th international conference on Intelligent user interfaces. New Mexico. Pages 33–40, doi: 10.1145/359784.359836.

  • CUEVA, S.; RODRÍGUEZ, G. (2010). “OERs, Standards and Trends”. Revista de Universidad y Sociedad del Conocimiento (RUSC). Vol. 7, No 1. UOC.

  • FOX, S.; KARNAWATT, K.; MYDLAND, M. et al. (2005). “Evaluating Implicit Measures to Improve Web Search”. ACM Transactions on Information Systems. Vol. 23, No 2, pages 147–168.

    Article  Google Scholar 

  • GARCÍA-BARRIOCANAL, E.; SICILIA, Miguel A. (2009). “Preliminary Explorations on the Statistical Profiles of Highly-Rated Learning Objects”. Metadata and Semantic Research. Proceedings of Third International Conference. Milan: MTSR 2009. Pages 108–117.

    Google Scholar 

  • HAN, K. (2004). Quality Rating of Learning Objects using Bayesian Belief Networks. PhD thesis. Canada: Simon Fraser University.

    Google Scholar 

  • KAY, R.; KNAACK, L. (2009). “Assessing learning, quality and engagement in learning objects: the Learning Object Evaluation Scale for Students (LOES-S)”. Educational Technology Research and Development. Vol. 57, No 2, pages 147–168.

    Article  Google Scholar 

  • KAY, R.; KNAACK, L. (2007). “Evaluating the learning in learning objects”. Open Learning: The Journal of Open and Distance Learning. Vol. 22, No 1, pages. 5–28.

    Article  Google Scholar 

  • KELTY, C. M.; BURRUS, C. S.; BARANIUK, R. G. (2008). “Peer Review Anew: Three Principles and a Case Study in Postpublication Quality Assurance”. Proceedings of the IEEE. Vol. 96, No 6, pages 1000–1011.

    Article  Google Scholar 

  • KUMAR, V.; NESBIT, J.; HAN, K. (2005). “Rating Learning Object Quality with Distributed Bayesian Belief Networks: the why and the how”. Fifth IEEE International Conference on Advanced Learning Technologies. ICALT 2005, pages 685–687.

  • KURILOVAS, E.; DAGIENE, V. (2009). “Learning objects and virtual learning environments technical evaluation criteria”. EJEL — Electronic Journal of e-Learning. Vol. 7, No 2, pages 147–168.

    Google Scholar 

  • LI, J. Z.; NESBIT, J. C.; RICHARDS, G. (2006). “Evaluating learning objects across boundaries: The semantics of localization”. Journal of Distance Education Technologies. Vol. 4, No 1, pages 17–30.

    Article  Google Scholar 

  • MCGREAL, R.; ANDERSON, T.; BABIN, G. et al. (2004). “EduSource: Canada’s Learning Object Repository Network”. International Journal of Instructional Technology and Distance Learning. Vol. 1, No 3.

  • OCHOA, X.; DUVAL, E. (2006). “Quality Metrics for Learning Object Metadata”. Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications. ED-Media 2006. Pages 1004–1011.

  • OCHOA, X.; DUVAL, E. (2008). “Relevance Ranking Metrics for Learning Objects”. IEEE Transactions on Learning Technologies. Vol. 1, No 1, pages 34–48.

    Article  Google Scholar 

  • SANZ RODRÍGUEZ J.; DODERO, J. M.; SÁNCHEZ ALONSO, S. (2009). “A preliminary analysis of software engineering metrics-based criteria for the evaluation of learning objects reusability”. International Journal of Emerging Technologies in Learning. Vol. 4, No 1, pages 30–34.

    Google Scholar 

  • SCHAFFERT, S.; GESER, G. (2008). “Open Educational Resources and Practices”. eLearning Papers. No 7. Available at www.elearningpapers.eu. [Accessed: 26 October 2010].

  • TZIKOPOULOS, A.; MANOUSELIS, N.; VUORIKARI, R. (2007). “An Overview of Learning Object Repositories”. In: Learning Objects for Instruction: Design and Evaluation. Idea Group Publishing. Pages 44–64.

  • VUORIKARI, R. Tags and self-organisation: a metadata ecology for learning resources in a multilingual context. PhD thesis. Open Universiteit Nederland.

  • YEN, N. Y.; HOU, F. F.; CHAO, L. R. et al. (2009). “Weighting and Ranking the E-Learning Resources” Proceeding of the 9th IEEE International Conference on Advanced Learning Technologies. ICALT 2009. Pages 701–703.

  • ZIMMERMANN, B.; MEYER, M.; RENSING, C. et al. (2007). “Improving retrieval of re-usable learning resources by estimating adaptation effort backhouse”. First International Workshop on Learning Object Discovery and Exchange.

Download references

Author information

Authors and Affiliations

  1. Department of Informatics, Universidad Carlos III de Madrid, Av. Universidad, 30, 28911, Leganés, Madrid, Spain

    Javier Sanz Rodríguez (Temporary Part-time Lecturer)

  2. Department of Computer Languages and Systems, Universidad de Cádiz, C/ Chile, s/n, 1003, Cadiz, Spain

    Juan Manuel Dodero (Tenured University Lecturer)

  3. Department of Computer Science, Universidad de Alcalá de Henares, Ctra. Barcelona, km 33600, 28871, Alcalá de Henares, Madrid, Spain

    Salvador Sánchez Alonso (Tenured University Lecturer)

Authors
  1. Javier Sanz Rodríguez
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Juan Manuel Dodero
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Salvador Sánchez Alonso
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Javier Sanz Rodríguez.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (https://creativecommons.org/licenses/by/4.0), which permits use, duplication, adaptation, distribution, and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Reprints and permissions

About this article

Cite this article

Sanz Rodríguez, J., Dodero, J.M. & Sánchez Alonso, S. Ascertaining the Relevance of Open Educational Resources by Integrating Various Quality Indicators. Int J Educ Technol High Educ 8, 211–224 (2011). https://doi.org/10.7238/rusc.v8i2.1031

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.7238/rusc.v8i2.1031

Keywords

Palabras clave