Meta-analysis of research in e-learning published in Spanish journals

Abstract

E-learning is a way of teaching that presently makes greater sense in the process of democratization of education. This articles describes a meta-analysis performed on Spanish journals found in Social Sciences and Law areas (N = 118) indexed in JCR and SCOPUS and awarded with the FECYT seal, that have published articles on this topic in the last 5 years (2011–2015). The total number of articles analyzed was 108. The main results show that these have focused on studies on academic efficiency, learning, teaching and cognitive styles, and interaction and communication. The main conclusion reached was the need for a study on the reasoning behind why elements such assessments are not treated in these articles.

Introduction

E-learning is becoming one of the most-utilized teaching methods in different universities, institutions and businesses. Salinas, Darder, and De Benito (2015), when defining it, have pointed to the need to differentiate between the three different concepts: e-learning, b-learning and m-learning, which are conditioned not only by advances in telecommunications, but by transformations in the teaching fields as well.

Without wanting to do an exhaustive analysis of the three terms mentioned, we still agree with different authors (Castaño & Cabero, 2013; Sevillano & Vázquez, 2015; Thorne, 2012; Torres, Infante, & Torres, 2015), when they state that “e-learning” refers to virtual education supported by the presentation of content through the Internet and the use of different synchronous and asynchronous communication tools to establish the interaction between the teachers and the learners, “b-learning” is the combination of learning through the Internet as well as face-to-face, and “m-learning” is virtual learning supported by mobile devices, taking advantage of their mobility and ubiquity.

To these definitions, the term e-learning 2.0 has been added, which surpasses the simple addition of the distance learning platforms, also known as LMS with the Web 2.0 tools, resulting in a more critical and participative vision of the student in the process of learning, also implying a transformation of the virtual learning process and the roles played by the teachers and learners (Cabero, 2013).

Independently of modality, different authors give e-learning different advantages when used in learning processes (Gros, 2011; Roig, Mengual, & Rodríguez, 2013): it easily adapts to the student’s individual characteristics and needs, connects students that are located in different geographical areas, the learning pace is set by the students, updating of the content is fast, it offers spatial-temporal flexibility for learning, it facilitates the interaction between the people that participate in the teaching processes, the students themselves, and the students with the learning/teaching objects, resources in different formats (text, video,…) can be used, different types of synchronous and asynchronous communication tools are used for the interaction between participants, learning scenarios are broadened, and the delocalization of knowledge and where it is found takes place.

On his part, Bates (2011) points to a series of arguments with respect to the failure produced through its use: scarcely-motivating content; impossibility of downloading the content in printable version; lack of feedback by the instructor; lack of activities that are truly significant for the student’s needs; scarce personalization of the content; excessive time intervals lapsed between the application for taking the course and the course’s start date; lack of student’s motivation; low usability of the system, and technical failure and delay in resolving the incidents.

For Cabero (2006), in order to perform high-quality e-learning, it is not enough to focus exclusively on the technological aspects, but the focus should also be on the educational and organizational variables: this is how the content, methods employed, the e-activities performed by the students and the roles played by the teacher and tutors should be structured. These aspects are currently being performed in different research works (Andreu-Andrés & García-Casas, 2015; Gámiz, Montes, & Pérez, 2014; Islas, 2015).

Meta-analysis of research works are becoming traditions in the education domain, such as the ones performed by Cabero (2008) on e-learning research; Kinshuk, Huang, Sampson, and Chen (2013) on the tendencies of educational technologies; Bacca, Baldiris, Fabregat, Graf, and Kinshuk (2014), focused on augmented reality; Sangrá, González, and Anderson (2015) on the MOOCs, or Badia (2015), which analyzes research tendencies on technology-enriched learning.

Methods

The general objective of by this research work is: “to perform a meta-analysis of research performed in the last five years, referring to the analysis of the problems of performance and other learning variables of e-learning applied to teaching contexts that were published between 2011 and 2015”.

The criteria used was to only select research works published in Spanish journals that appeared in the JCR and/or SCOPUS databases, or had been awarded a FECYT Seal, and that also had an on-line version that could be used to more easily access them. Their inclusion within the area of Social Sciences was also taken into account.

The number of journals chosen was 118, out of which only 23 contained articles that dealt with the subject studied in the present work (View Tables 1 and 2).

Table 1 Presence of e-learning articles in the journal selected

Table 2 Shows the frequencies and percentages reached for the different subjects analyzed in the instrument

The instrument used for the analysis of the articles was the “Analysis sheet of the content found in documents related to e-learning and the results found”, created by Cabero (2008) (http://www.grupotecnologiaeducativa.es/images/LIBROS/elearning08.pdf), which was constructed through a Delphi study. A quantitative analysis was performed with the data extracted. The codification of the articles was performed by the authors of the present study, after previously agreeing on the meaning of the different categories in the instrument.

Results

The data on the year of publications show that similar values were found in the years analyzed (2011–2015), although there was a greater number of articles in 2012 (22.2 %) and a lesser number in 2011 (17.6 %), while the quantity was the same in the years 2013 and 2014 (19.4 %), with the articles published in 2015 contributing 21.3 % of the total.

The journals’ impact factor data shows that only one of the journals was indexed in JCR (f = 1, 0.9 %), the same as in JCR and SCOPUS combined (f = 1, 0.9 %), while most of the journals had the FECYT Seal (f = 42, 38.9 %), and only 24.1 % (f = 26) of the articles were found in journals that were both indexed by SCOPUS and had the FECYT Seal.

When classifying the articles by the numbers of authors, the articles with three authors totaled 33.3 %, followed by two (29.6 %) and more than 3 (19.4 %), with the least being the articles written by only one author (17.6 %).

The innovation and investigation experiments found in the articles were directed towards university surroundings (87.0 %); with a lesser number found for the institution-administration pair (0.9 %); those that had variable subjects of study (3.7 %), while those that were impossible to identify added up to 2.8 %.

Most of the studies were grouped under the label “various” (44.4 %), the descriptive studies added up to 27.8 % and the experimental studies to 23.1 %, with the case studies being the less addressed (4.6 %).

With respect to the research methodology employed, those that used quantitative methods were 51.9 %, only 6.6 % employed a mixed method of analysis; likewise, 21.7 % described projects, and 19.8 % used a qualitative methodology.

As for topics studied within the different subjects, we found that from greater to lesser frequency, the results found were: academic performance (f = 218; 22.45 %); learning, teaching and cognitive styles (f = 212; 21.83 %); interaction, communication (f = 165; 16.99 %); others (f = 1140; 14.42 %); degree of satisfaction (f = 57; 5.87 %); development of competencies (f = 55; 5.66 %); assessment (f = 48; 4.94 %); curricular integration into diverse areas (f = 41; 4.22 %); and attitudes/perception (f = 35; 3.60 %).

The topic studied results within the subject of “Design of materials and/or digital objects”, showed that studies that focused on the sub-category “academic performance” predominated (34.3 %), followed by “learning styles” (22.2 %). The least studied were those referring to “assessment” (1.9 %) (Fig. 1).

Fig. 1

Classification of the articles found on the subject “Design of materials and/or digital objects”

On the subject of “E-learning environments, distance learning platforms”, the topics related to “academic performance (20.4 %) and “learning styles” (25.0 %) were emphasized, with the topic of “assessment” being the least studied (3.7 %) (Fig. 2).

Fig. 2

E-learning environments, distance learning platforms

On the subject of “Tutorships and advice used in e-learning activities”, the topics “interaction and communication (39.3 %) predominated, with the “curricular integration into diverse areas” (1.9 %) being the least taken on (Fig. 3).

Fig. 3

Tutorship and advice used in e-learning activities

As for the subject “Didactic strategies and methodologies used in e-learning activities”, the topics “academic performance” (21.3 %) and “learning styles” (33.3 %) stood out (Fig. 4).

Fig. 4

Didactic strategies and methodologies used in e-learning activities

Within the subject “Use of synchronous and asynchronous communication tools in e-learning activities”, the effects of “interaction and communication” (28.7 %) were predominant, with those referring to “curricular integration into diverse areas” (1.9 %) being the less studied (Fig. 5).

Fig. 5

Use of synchronous and asynchronous communication tools in e-learning activities

On the subject “Use of assessment techniques and strategies in e-learning activities”, the studies on the topics of “academic performance” and “assessment” predominated (both 21.3 %), with those referring to “curricular integration to diverse areas” being the less attitudes and perceptions (1.9 %) (Fig. 6).

Fig. 6

Use of assessment techniques and strategies in e-learning activities

On the subject of “Organizational and institutional aspects related to e-learning activities”, research on “academic performance” (30.6 %) was highlighted, with those referring to “assessment” being less common (2.8 %) (Fig. 7).

Fig. 7

Organizational and institutional aspects related to e-learning activities

For the subject “Collaborative and cooperative activities used in e-learning activities”, the most investigated topics were “interaction and communication” (27.8 %) followed by “academic performance (22.2 %), while the least researched was “attitudes and perceptions” (0.9 %) (Fig. 8).

Fig. 8

Collaborative and cooperative activities used in e-learning activities

Lastly, the results on the articles referring to “Studies on accessibility and usability”, indicated that “academic performance (22.2 %) and “learning styles” (24.1 %) predominated, while “attitudes and perceptions” (2.8 %) was the least researched (Fig. 9).

Fig. 9

Studies on accessibility and usability and their effects on e-learning activities

The comparison between the journals’ indexation with the type of study the article investigated were found to be related, with a Chi-square = 44.056 (p < 0.001). As the contingency table for the statistical analysis was 6x4, other association statistics were needed, as pointed by Rodríguez and Mora (2001), in order to re-enforce results. Values such as Phi (0.639, p < 0.001), Cramer’s V (0.369 and p < 0.001), with the Coefficient of contingency (0.538 and p < 0.001) were calculated, and they all showed that there was a relationship between both variables.

To understand the proportional reduction in the error, when using the values of the independent variable (journal indexing) to forecast the values of the dependent variable (type of study), the nominal statistic Goodman-Kruskal Tau was applied, obtaining a value of 0.821, p < 0.001 (dependent), and a value for the Coefficient of uncertainty (Cu) of 0.812 (p = 0.021). This means that knowing one variable reduces the error 80 % when forecasting the values of the other.

As can be observed in Fig. 10, the experimental studies were found in those journals that had all the impact factors (44 %), while the descriptive studies were gathered primarily in the journals that had been awarded the FECYT Seal (30 %), just as those labeled various (56.3 %). On the other hand, the ones containing case studies were found in SCOPUS and FECYT journals combined (40 %).

Fig. 10

Type of study as a function of the impact factor of the journal where it was published

To unify and homogenize the criteria found in the articles selected, a comparison between the subject “Design of materials and/or digital objects used in e-learning activities”, with the type of method used in the article was performed, taking into account the existence of a relationship between both aspects. The Chi-square value was 58.714 (p < 0.001), with the Phi (0.744, p < 0.001), Cramer’s V (0.430, p < 0.001) and a Coefficient of contingency (0.597, p < 0.001), with these values indicating that there is a relationship among all the variables.

The Tau value was 0.831 (p < 0.001) for the dependent variable, with a Cu value of 0.702 (p < 0.001). This signifies that the knowledge of a variable reduces 70 % of the error when forecasting the values of the other.

The results of this comparison are shown in Fig. 11, indicating that the sub-category of academic performance studied within the subject “Design of materials and/or digital objects used in e-learning activities” and assessment were mainly studied a using quantitative methodology (50.9, and 3.6 %, respectively); attitudes and perceptions and the degree of satisfaction used mixed typologies (both 28.6 %). The interaction and communication used description of projects (21.7 %), in the same manner as that relative to curricular integration into diverse areas (8.7 %) and development of competencies (17.4 %) and others (17.4 %), while the styles of learning used qualitative methodologies (42.9 %).

Fig. 11

Methodology employed in the articles found under the “Design of materials and/or digital objects” subject

Likewise, when addressing the subject “E-learning environments, distance learning platforms” with the type of method used in the article, we observed the existence of a relationship between both aspects, with a Chi-square of 42.590, p = 0.011, with values of Phi (0.634, p = 0.011), Cramer’s V (0.456, p = 0.011) and a Coefficient of contingency (0.535, p = 0.011), signifying that there was a relationship between both variables.

The Tau value was 0.731 (p = 0.005) for the dependent variable; however, the symmetric “Cu” obtained was not significant (p = 0.186). Therefore, knowledge of one variable did not reduced the error when forecasting the value of the other.

Data gathered on this comparison are shown in Fig. 12, indicating that the sub-category “academic performance” (29.1 %) and “assessment” (7.3 %) within the subject “E-learning environments and distance learning platforms” were elaborated mostly using quantitative methods; the “attitudes and perceptions” (14.3 %), “degree of satisfaction” (28.6 %), and the “development of competencies” (28.6 %) utilied mixed typologies; the “interaction and communication” used description of projects (21.7 %), likewise the methods used relative to the “curricular integration into diverse areas” (13 %) and “others” (17.4 %); while “learning styles” used qualitative methodologies (47.6 %).

Fig. 12

Methodology used in the articles as a function of the subject “E-learning environments and distance learning platforms”

Fig. 13

Methodology used in the articles as a function of the subject “Tutorships and advice used in e-learning activities”

With respect to the subject “Tutorships and advice used in e-learning activities” with the comparison of the type of method used in the article, we observed that there was a relationship between both aspects, Chi-square = 47.774 (p = 0.003), with values of Phi (0.675, p = 0.003), Cramer’s V (0.423 and p = 0.003), with the “Coefficient of contingency (0.559, p = 0.003), meaning that there was a relationship between both variables.

The Tau values were 0.631 (p < 0.001) for the dependent variable; however, those (Fig. 13) obtained for the symmetric “Cu” values were not significant (p = 0.230), therefore the knowledge of one variable did not reduce the error when forecasting the values of the other.

The data showed that in the articles selected, for the subject “Tutorships and advice used in e-learning activities”, quantitative methodologies were mostly used in the sub-categories “academic performance” (20.0 %), “attitudes and perceptions” (1.8 %), “assessment” (5.5 %) and the “curricular integration into diverse areas” (3.6 %). On the other hand, those that had a more qualitative approach were observed in “learning styles” (47.6 %), as was already found with the rest of the subjects. The results related to “interaction and communication” (63.6 %) used description of projects, just as the development of competencies (13.6 %) and others (18.2 %) did. And lastly, the “degree of satisfaction” used mixed typologies (28.6 %).

However, when analyzing the subject “Didactic strategies and methodologies used in e-learning activities” with a comparison of the type of methodology used in the article, we observed that there was no relation between them, as the Chi-square values were 31.087, p = 0.151, which were not significant.

Likewise, when looking at the subject “Use of synchronous and asynchronous communication tools in e-learning activities” compared to the type of methodology utilized in the article, we observed that there was a relationship between both aspects, with a Chi-square equal to 62.806 (p < 0.001) with values of Phi (0.770, p < 0.001), Cramer’s V (0.444, p < 0.001) and a Coefficient of contingency (0.610, p < 0.001), which implied that both variables were related.

The Tau value obtained was 0.831 (p = 0.025) for the dependent variable; however, the values obtained for the symmetrical “Cu”, were not significant (p = 0.201). Therefore the knowledge of a variable did not reduce the error when forecasting the values of the other.

Figure 14 shows the data obtained in the comparison, which demonstrate that the sub-category “academic performance” within the subject “Use of synchronous and asynchronous communication tools in e-learning activities” mostly used quantitative methodologies (25.5 %), just as “assessment” (7.3 %). The “curricular integration into diverse areas” (3.6 %), “attitudes and perceptions” (28.6 %), “degree of satisfaction” (28.6 %) and the “development of competencies” (14.3 %) used mixed typologies; while “interaction and communication” utilized description of projects (60.9 %), while at the same time, “others” and “learning styles” used qualitative methods (17.4 and 42.9 %, respectively).

Fig. 14

Methodology employed in articles as a function of the subject “Use of synchronous and asynchronous tools used in e-learning activities”

In relation to the subject “Use of assessment techniques and strategies in e-learning activities” compared to the type of methodology used in the article, we observed that a relationship between both aspects existed, with values of Chi-square = 41.716, p = 0.014, Phi =0.627, p = 0.014), Cramer’s V (0.401, p = 0.014) and Coefficient of contingency (0.572, p = 0.014), indicating that there was a relationship between both variables.

The Tau value was 0.712 (p = 0.008) for the dependent variable, and the symmetrical “Cu” was significant at p < 0.001 with a value of 0.902. Therefore the knowledge of one variable reduced the error 90 % when forecasting the value of the other variable.

The data showed that in the articles selected for the subject “Use of assessment techniques and strategies in e-learning activities”, none of the topics studied greatly a quantitative methodology. However, the qualitative methodology was used in the sub-categories “academic performance” (33.3 %), “learning styles” (33.3 %). The “attitudes and perceptions” (14.3 %), the “degree of satisfaction” (28.6 %), the “curricular integration into diverse areas” (14.3 %) and “others” (14.3 %) mostly used mixed methodologies (Fig. 15).

Fig. 15

Methodology employed in the articles as a function of the subject “Use of assessment techniques and strategies utilized in e-learning activities

The subject “Organizational and institutional aspects related to e-learning activities”, along with the type of methods used in the articles, were found to be related, with a Chi-square = 48.850 (p = 0.002), with values of Phi (0.679 and p = 0.002), Cramer’s V (0.424, p = 0.002) and Coefficient of contingency (0.569, p = 0.002), all of which point to a relationship between both variables.

The Tau value was 0.724 (p < 0.001) for the dependent variable, with a symmetric “Cu” which was significant, as shown by p = 0.038, and a value of 0.623. Therefore the knowledge of a variable reduces the error 62 % when forecasting the values of the other.

The results of the comparison of the subject “Organizational and institutional aspects related to e-learning activities” indicated that the articles did not use a quantitative methodology to a great extent. For qualitative methodologies, on the other hand, we found the “academic performance” (42.9 %) and “learning styles” (28.6 %). “Attitudes and perceptions” (14.3 %), “degree of satisfaction” (28.6 %), “curricular integration into diverse areas” (28.6 %) and “development of competencies” (14.3 %) used mixed methodologies (Fig. 16).

Fig. 16

Methodology used in the articles as a function of the subject topic “Organizational and institutional aspects related to e-learning activities”

Fig. 17

Methodology employed in the articles as a function of the subject “Collaborative and cooperative actions used in e-learning activities”

As for the comparison results of the subject “Collaborative and cooperative actions used in the e-learning activities” with type of methods used in the articles, we observed that there was a relationship between both aspects with Chi-square =46.327 (p = 0.004), values of Phi (0.661, p = 0.004), Cramer’s V (0.452, p = 0.004), and a Coefficient of contingency (0.543, p = 0.004), indicating that there was a relationship between both variables.

The Tau value reached was 0.653 (p < 0.001) for the dependent variable, with a symmetric “Cu” value of 0.836 (p = 0.048), a value that implied that the knowledge of one variable reduced 83 % of the error when forecasting the values of the other.

The data showed that on the articles selected under the subject “Collaborative and cooperative actions used in e-learning activities”, the the sub-categories of “attitudes and perceptions” (1.8 %) and “assessment” (3.6 %) were found using quantitative methods, while for the qualitative methods, we found it to be used in the sub-category “academic performance” (38.1 %) and “learning styles” (28.6 %). The degree of satisfaction (28.6 %) used mixed methodologies.

Lastly, the subject “Specific studies on accessibility, usability and their effects on e-learning activities” with the comparison of the type of methodology used in the articles, we observed the existence of a relationship between both aspects as shown by a Chi-square of 43.734 (p = 0.008), with values of Phi (0.645, p = 0.008), Cramer’s V (0.413, p = 0.008) and a Coefficient of contingency (0.542, p = 0.008), with a relationship therefore present (Fig. 17).

The Tau value was 0.639 (p = 0.001) for the dependent variable; however, the results obtained in the symmetric “Cu” were not significant (p = 0.127), therefore the knowledge of one variable did not reduce the error when forecasting the values of the other.

The data showed that in the articles selected for the subject “Specific studies on accessibility, usability and their effects on e-learning activities”, the sub-category of “assessment” (3.6 %), were observed using quantitative methods. While qualitative methods were used on “academic performance” (33.3 %) and “learning styles” (33.3 %). The description of projects were used mainly on the topics of “interaction and communication” (30.4 %), “curricular integration into diverse areas” (4.3 %), and “others” (17.4 %). Lastly, the topics “attitudes and perceptions” (42.9 %), “degree of satisfaction” (28.6 %) and “development of competencies” (14.3 %) used mixed methodologies (Fig. 18).

Fig. 18

Methodology employed in the articles as a function of the subject “Specific studies on accessibility, usability and their repercussions on e-learning activities”

Conclusions

The first result that should be pointed out is that from 2011 to 2015, the number of publications linked to e-learning decreased when compared to the volume of work found in the meta-analysis from 2008 by Cabero. However, the presence of this subject has been steady in the different years analyzed, which suggests that it is still a field of interest in education research (Raghuram, 2013), where there are still a great number of problems to study (Conole & Oliver, 2007).

The study performed revealed, on the one hand, the tendency of the research to focus on the university context (Islam, Martin Beer, & Slack, 2015; Jirasak, 2014), possibly due to the university being the place where it is mostly used. On the other hand, there was not a single type of method that was predominantly used, although studies that used a quantitative methodology were numerous.

With respect to the subjects analyzed, we can highlight two groups, with the most analyzed being the articles that studied problems of: “Design of materials and/or digital objects used in e-learning activities”, “E-learning environments, distance-learning platforms”, “Didactic strategies and methodologies used in e-learning”, and “use of synchronous and asynchronous communication tools in e-learning activities”. The least subjects studied were related to “Tutorships and advice used in e-learning activities”, “Use of assessment techniques and strategies in e-learning activities”, “Organizational and institutional aspects related to e-learning activities”, “Collaborative and cooperative actions used in e-learning activities, and “Specific studies on accessibility, usability and their effects on e-learning activities”, in line with research by Borokhovski et al. (2011), Jirasak (2014), Anil and Batdi (2015), Bozkurt et al. (2015) or Islam et al. (2015). The results found coincide with those found by Cabero (2008), and point to the need for more research on deficient areas, even more so when some of them could explain some of the errors committed during their putting into practice, as suggested by Bates (2011).

At the same time that the areas investigated allowed us to point out that the proposal suggested by Conole and Oliver (2007), which stated that research in the area of e-learning could be performed with respect to its organizational, pedagogic and technical dimensions, at the individual level as well as in their interaction, without forgetting that these could be explained by sociocultural factors, could be a useful and interesting proposal for generating research proposals.

The present study has revealed the tendency to study the repercussions of e-learning on academic performance (Bagarukayo & Kalema, 2015). That said, it also points to the few works that study its effects on assessment (Noesgaard & Ørngreen, 2015; Sozcu, 2014), and its curricular integration into different areas. These studies are necessary, in agreement with the new directions of integration of the ICTs, and as pointed out by Conole and Oliver (2007), Kituyi and Tusubira (2013), Islam et al. (2015) or Mothibi (2015) in their research works.

Lastly, it is important to come to an agreement on a list of study subjects, to direct future research, in order to ease their significant integration to education and training (Koehler, Mishra, & Cain, 2013; Middle States Commission on Higher Education, 2011, Mothibi, 2015). But these studies should be routed to the areas of study found to be deficient in our study.