- Research article
- Open Access
Self-reported internet skills, previous knowledge and working memory in text comprehension in E-learning
International Journal of Educational Technology in Higher Education volume 15, Article number: 18 (2018)
We examined the contribution of Internet operational and navigation skills, previous knowledge, and working memory capacity to expository text comprehension as a lesson within an e-learning course. As different from previous studies in controlled settings; this study addressed students’ typical behavior in more ecological conditions. The first study tested self-reported Internet Skills Scale structure, reliability and concurrent validity, in a sample of 254 college students from a large Latin American public university. The second study addressed the contribution of self-reported Internet skills, previous domain knowledge, and working memory capacity to text comprehension in e-learning. Students (n = 125) read high or low previous knowledge expository science texts and answered questions about them, in an e-learning course specifically designed for research purposes, accessed remotely. They also completed working memory tests. Working memory and navigation were significantly associated with text comprehension: higher working memory, and lower scores in self-reported navigation behavior, led to better comprehension. These results have implications for instructional design and reading strategies interventions.
Effects of internet skills, previous knowledge and working memory on text comprehension in E-learning
E-learning frequently needs to deliver expository instruction based on the distribution of specific thematic material within a closed learning content management system (such as Moodle, Blackboard, or similar), resulting in a text or video based lesson embedded in an online course (Clark & Mayer, 2016). The purpose of many instructional lessons is to deliver expository content, facts and concepts which the student must understand and remember. Text based multimedia formats (text and graphics) is the most frequently employed method to support such instruction. A recent survey with a stratified sample of all college level teaching faculty in the U.S. (Allen & Seaman, 2016) found that 77.7% adopted a textbook based teaching methodology; in blended or fully online courses, around 80% employed “copyrighted” digital textbooks that could be embedded in their LMS system, in contrast to the scarce use of open and free resources.
Individual differences in text and digital text comprehension: Verbal ability, previous knowledge, and working memory
Comprehension of expository text entails constructing a coherent internal representation of what the text conveys, and requires active inferential and coherence building processing (Graesser, Leon, & Otero, 2002; Kintsch, 1998). Digital technologies first brought a categorization of texts as printed or digital; a more recent distinction considers whether texts are static (printed and similar digital counterparts such as in pdf or epub formats), or dynamic (hypertext or digital texts) (OECD, 2009).
Traditional research on printed expository text comprehension in college level readers has shown that individual differences in previous domain knowledge, verbal ability, and metacognitive strategies contribute to comprehension (Graesser et al., 2002). Digital texts, or hypertexts, share processing requirements with printed text, but present additional specific features needing processing. Studies of complex digital text comprehension have employed controlled digital tasks in closed, mock Internet environments Readers have to navigate several information nodes, evaluating and selecting or discarding content, integrating multimedia information from multiple sources, to seek answers to questions and construct a mental model of the content (Coiro, 2011a, 2011b; Leu, Kiili, & Forzani, 2014; OECD, 2009, 2011). Participants are required to search, identify, compare information, communicate results and solve problems, defining in advanced the optimal navigation path and the correct final information. For example, the ORCA Project (Coiro, 2011a, 2011b; Leu et al., 2014) presented a complex closed web environment including social networks, webpages, wikis, and e-mail, where children have to perform different information search and problem resolution tasks. A similarly complex digital text comprehension construct has been incorporated since 2009 in the Program for International Student Assessment (PISA; OECD, 2009; OECD, 2011). Digital reading is assessed by creating closed web sites with multimedia material (texts and images), such as a social media platform for learning a language, an official city website, or a chain of e-mails. Tasks vary from finding explicit information, to making global comprehension inferences about content and authors, to forming personal opinions.
Comprehension of hypertexts is, as in printed texts, also a function of general verbal ability, and/or previous knowledge of the subject matter (Amadieu & Salmerón, 2014; Coiro, 2011a; Leu et al., 2014; OECD, 2011). Verbal ability and previous knowledge enable semantic activation of relevant information, so that the reader can draw inferences and build an accurate model (OECD, 2011; Perfetti & Stafura, 2014). In addition, verbal ability and previous knowledge can have an indirect effect on comprehension, through navigation. Given that making sense of digital text entails navigation, coherence of the navigational path (whereas aided by interface design or guided by semantic knowledge) helps to build an integrated mental model of the content (Amadieu & Salmerón, 2014; Coiro, 2011b; Leu et al., 2014; Salmeron, Kintsch, & Cañas, 2006). Semantic knowledge helps in link and node selection and evaluation (Rouet, Ros, Goumi, Macedo-Rouet, & Dinet, 2011; Salmerón, Cerdán, & Naumann, 2015). For example, Salmerón et al. (2015) found that high school students’ search for answers to questions relied both on lexical matching between hyperlink labels and questions, and on semantic analysis; high verbal comprehension abilities were needed to discard an irrelevant link that matched literally and select a semantically relevant one.
Another contributing factor seems to be individual differences in working memory, the neurocognitive system for temporary retention and processing, limited in capacity (Baddeley, 2010). With printed materials, individual differences in working memory have been associated with differences in text comprehension (Daneman & Merikle, 1996), because working memory constrains inferential processing and model construction (Barreyro, Cevasco, Burin, & Molinari, 2012). In hypertext comprehension, more demanding interfaces affect particularly low working memory participants (DeStefano & LeFevre, 2007; Gwizdka, 2009; Lee & Tedder, 2003; Naumann, Richter, Christmann, & Groeben, 2008). For example, Naumann et al. (2008) taught students strategies for learning with hypertexts, and found different outcomes for high and low working memory participants. Whereas high working memory students benefitted from training, low capacity ones were negatively impacted both in comprehension and navigation, as shown by the number of visits to task relevant content pages. This suggested that learning a new strategy loaded working memory beyond comprehension and navigation requirements, and this increased load was difficult to handle for low working memory students.
Internet skills and digital text comprehension
Given that digital text requires navigation, link selection and evaluation, and integration from different nodes, individual differences in these type of skills might play a role in online reading. Constructs such as digital literacy (Bawden, 2008), digital skills (Fraillon, Ainley, Schulz, Friedman, & Gebhardt, 2014), and Internet skills (van Deursen & van Dijk, 2009) assume that new information technologies require specific knowledge and skills for their use, and an understanding of how to use them in context. The notion of digital skills can be related to, but distinguished from, Internet skills. Digital skills include the basic and advanced operation, knowledge, and strategic use, of computers and digital devices in general, including offline (e.g. using a word processor, a spreadsheet, presentation software, or coding). Internet skills refer to operational skills specific to the web, and its core would reside in navigation, integration and assessment of multiple sources, evaluation of content and sources, and communication. Van Deursen and van Dijk (2009, 2016) categorized Internet skills in four domains: operational, formal, information and strategic. Operational skills consist of the actions that allow a person to use digital tools, such as navigation bars, buttons or links, different ways to input information in fields or pages, possibilities to download or save files, and similar procedural knowledge. Formal skills refer to navigation and orientation in structures specific to digital media (hypertexts), such as knowing which website one is in, identifying how to navigate each site, orienting oneself or getting lost while performing a task, familiarity with different types of sites and applications. Information skills consist of knowledge about how to perform a search and select the most appropriate information to meet a specified purpose, and its core resides in knowing how to compare, assess, and integrate information from the Internet including different sources. Finally, strategic skills involve knowledge and use of Internet to achieve a personal, professional, or social goal.
As stated above, studies of complex digital text comprehension have employed complex digital tasks requiring search, evaluation and integration. This conflates Internet skills and digital comprehension, because the latter is defined in terms of complex search and comprehension scenarios. As a result, the digital text comprehension construct is defined by a mix of operational computer skills, knowledge of the Internet environment, search and navigation skills, text integration and comprehension abilities, information evaluation, communication, and problem solving.
To disentangle Internet skills and comprehension, in an extension of the large PISA study in Germany, Hahnel, Goldhammer, Naumann, and Kröhne (2016) assessed the effects of linear reading (e.g. traditional reading comprehension ability), basic computer skills, navigation skills, and online information evaluation, on reading digital text. Indexes for each construct were derived: performance in printed PISA items for linear reading; number of visits to relevant pages in the PISA digital text comprehension tasks for navigation; performance in the Basic Computer Skill Scale, a test simulating word processing software, web browser, and email, for basic computer skills; and performance in the Test for Evaluation of Online Information, which poses problem solving activities (e.g., preparing a talk for biology class or looking for information about how to change a bicycle chain) and requires identification of links most likely to provide credible and useful information. They found that digital reading comprehension was directly related to unique variance provided by linear reading ability, basic computer skills, and information evaluation, and that navigation mediated the influence of the first two abilities and skills. In a similar vein, Fajardo, Villalta, and Salmerón (2016) created a web site with multimedia digital texts followed by literal and inferential comprehension questions, and also an Internet skills test in a mock web environment (tasks such as saving favorite websites, selecting active links on a website, or copying someone on an email message). With primary and high school students, printed comprehension ability, basic Internet skills, and navigation behavior contributed to digital comprehension. For their part, Naumann and Salmerón (2016) found not only simple effects of navigation and offline comprehension, but also an interaction between them, in the sense that good navigation (e.g., selecting relevant pages) was beneficial for students high in verbal comprehension skills, but not so much for low skilled students. They argued that a minimal or threshold amount of general comprehension is needed to benefit from navigation; even if students find the relevant information through navigation, they might lack the ability to build a coherent model of the content.
The present study: Text comprehension in e-learning
In synthesis, verbal comprehension, previous knowledge, and operational and navigation Internet skills contribute, solely and in interaction, to hypertext and more complex digital text comprehension Working memory capacity might also have a limiting role. However, these studies have been conducted in controlled laboratory or classroom testing environments. They are test-like situations with the presence of instructors who can guide and monitor performance, and with computers specially prepared for the tasks. In contrast, e-learning at the college level is frequently carried out at the students’ home or habitual place of study, in their own computers, where they can open many other programs or apps. It relies more heavily on students’ self-organization and self-regulation (Dabbagh & Kitsantas, 2004).
In addition, in an e-learning course the student is generally presented with a lesson or a content in a page, with few links. This differs from complex multi-page web environments, and can be considered a static text. On the other hand, it usually presents navigational features, for example, a hierarchical menu with the content; it may have hyperlinks to other parts of the course where related information is present. Also, it is embedded in an e-learning course or web environment that must be understood, navigated and managed to achieve learning goals. For example, the student must understand how to inform that s/he has finished reading and access questions, how to continue to the following tasks, and whatever messages the learning system presents them. Therefore, even when in an e-learning lesson the student might not need to search and evaluate several pages to construct a mental representation of the lesson content, Internet skills might also be required for task completion.
So then, in text comprehension from a lesson embedded in an e-learning course solved at home, previous studies would suggest an association between performance and students’ Internet operational and navigational abilities, as well as previous knowledge and working memory capacity. The present research has examined the contribution of Internet skills, previous knowledge, and working memory capacity to text comprehension when understanding a lesson within an e-learning course.
The first study tested Internet skills construct validity and measurement with a self-report instrument, in college students from a large Latin American public university. Previous studies have tested Internet skills performance in controlled settings; this study addressed students’ typical behavior with self-report. Laboratory testing generally limits the navigation and does not incorporate other features of the Internet environment (such as commercial links and content, or other applications simultaneously open). Also, they do not capture the participants’ perception of their actions in everyday Internet use (Hargittai, 2005). The objective of the first study was to validate the Internet Skills Scale (van Deursen & van Dijk, 2016; van Deursen, van Dijk, & Peters, 2012) to be used in our sample of different age, culture and socio-economic characteristics than the original. We tested its structure, reliability and concurrent validity, expecting to replicate van Deursen et al. (van Deursen & van Dijk, 2016; van Deursen et al., 2012). The second study addressed the contribution of Internet skills, previous domain knowledge, and working memory capacity to text comprehension in e-learning. Students read expository science texts and answered questions about them, in an e-learning course specifically designed for research purposes (e.g. not related to their actual college course), and which the students accessed remotely. We expected that individual differences in Internet operational and navigational skills, working memory, and previous knowledge would contribute to text comprehension. Specifically, we expected better comprehension with higher Internet skills, working memory, and previous knowledge.
Van Deursen and van Dijk (2009) tested their model in a sample of 109 Dutch adults (18 to 79 years old). Participants performed a series of assignments related to online public services and captured their navigation, in a laboratory setting. They were required to locate a marriage license, locate and download a tax declaration, find the better rent subsidy and apply, and similar applied goals. Tasks were fact-based and had a specific correct action. They found that errors and problems could be specifically related to the four types of Internet skills presented above, operational, formal, information and strategic. In subsequent studies they sought to validate the four domains model (van Deursen & van Dijk, 2016; van Deursen et al., 2012). A 20 items self-assessment Likert-type scale was designed. An on-line study with 1008 Dutch adults, aged 18 to 88 found high reliability for the subscales (Cronbach’s α between .70 and .82), and confirmatory factor analyses validated the four-domain structure. Results also showed convergent validity with a traditional alphabetization self-report measure, and medium sized correlations (.34–.50) between information, formal and strategic skills and measures of Internet use for recreation, information search and occupational purposes.
The objective of this first study was to test the Internet Skills Scale structure and convergent validity (van Deursen & van Dijk, 2016; van Deursen et al., 2012) in a sample of college aged Latin American students.
A sample of 260 first-year undergraduate psychology students (age M = 23.91, SD = 6.47, range = 18–48 years old) from a large, public Argentinian university volunteered to participate anonymously. Due to missing responses, the final sample comprised 254 students. The study was approved by the institutional research ethics board.
Participants provided their sex, age, and their Internet habitual use. They were asked to categorize their time spent online in computer or tablet devices. The Internet Frequency categories were similar to those in the PISA 2009 study (OECD, 2011), adding a category for very frequent users given the participants’ demography (young college students). The Internet Frequency categories were: (1) less than once a week; (2) 1-5 hours per week, (3) almost every day, up to an hour; and (4) almost every day, several hours. They were also asked to complete a questionnaire about their habitual Internet activities, an adapted version of van Deursen and van Dijk (2016), to check which activities they carried out online. Given the participants’ age and context, two items referring to performing transactions with the government and looking for a job were replaced by buying or selling products online, and looking for practical, everyday information. The questionnaire has seven questions referring to Information/Academic uses, and seven concerning Recreation/Social purposes.
Internet skills scale
A first translation of van Deursen et al. (2012) was discussed by two research assistants and the principal investigator, looking for correct and contextually adapted linguistic expressions for the behaviors described by the items (International Test Commission, 2005). Consensus was difficult for the fourth domain, strategic Internet skills, and three items could not be clearly adapted (for example, “use reference websites” did not have a clear meaning in our context). Given the potential problems of these items in terms of sample and cultural appropriateness (International Test Commission, 2005), and that the objectives of this study concerned mainly operational and formal domains, the solution was to only adapt and study items excluding the strategic dimension. Translated and adapted items were then back-translated by a local native who had lived in the United States and so had mastery of both languages and contextual knowledge. The resulting Internet Skills Scale, then, consisted of 15 items, pertaining to the operational, formal and informational domains, translated and adapted from the original (van Deursen et al., 2012).
Participants were invited to fill the questionnaires in collective sessions, in paper and pencil format. Their participation was voluntary and anonymous, stating sex and age as personal information. They could refuse participation at any given moment. After completing the questionnaires, they were debriefed about the study.
To explore the factorial structure of the instrument, factor analyses (EFA) for ordinal data derived from Likert-type scales were performed using package FACTOR (Lorenzo-Seva & Ferrando, 2006). The implemented factor analyses were based on polychoric correlations, an extension of the tetrachoric correlation between two bivariate distributed continuous variables measured at an ordinal level. To identify the adequate number of factors to retain, parallel analysis with minimum rank factor analysis were employed (Baglin, 2014; Lorenzo-Seva & Ferrando, 2006, 2013). Parallel analysis works by generating many random parallel datasets, with the same number of variables and cases as the sample dataset; each of these parallel datasets, which are filled with independent randomly generated data, is analyzed using principal component analysis. The mean eigenvalues of the factors extracted from the random parallel datasets are compared to the samples’ eigenvalues. Minimum rank factor analysis estimates the percentage of common variance explained by an EFA model. In addition, root mean square of residual (RMSR) can be obtained to assess model fit. Overall, FACTOR has been characterized as a “semi-confirmatory” factorial analysis tool (Lorenzo-Seva & Ferrando, 2013).
As suggested by Baglin (2014) a first EFA was performed without factor constraints, to explore suggested number of factors for the database. After this first pass, additional one, two or three- factors models with Promin oblique rotation (factor loadings below .32 and dual loadings as criteria for elimination; Tabachnick & Fidell, 2001), were compared. Internal consistency of the resulting factors were determined with McDonald’s ω (Peters, 2014).
The matrix of intercorrelations was adequate to perform factorial analysis, KMO = .70, χ2(105) = 604.10, p < .001. A first EFA for ordinal data with parallel analysis with minimum rank factor analysis suggested three factors which accounted for 69.57% of explained variance. In successive analyses, one, two or three- factors models with Promin oblique rotation were carried out to identify which model best accounted for the scale. A one factor model obtained a standardized root mean residual RMSR = .12, greater than recommendation (RMSR < .06; Ferrando & Lorenzo-Seva, 2013), and a lower percentage of explained variance (34.65%) than the other models. Also, the two factor model obtained an unacceptable standardized root mean residual model fit (RMSR = .09), and also a lower percentage of explained variance (53.99%). The three factor model had a good model fit (RMSR = .05) and better percentage of explained variance (69.57%). Table 1 shows the communalities and factor loadings for each item. Item content analyses for the factors suggest that the first factor (seven items) corresponds to the formal or navigation skills, the second (three items) to the informational or evaluation skills and the third one (five items) to the operational skills.
Internal consistency was, for Factor 1 (Formal/Navigation), ω = .72; for Factor 2 (Informational/Evaluation), ω = .70; for Factor 3 (Operational), ω = .63.
Finally, to analyze convergent validity, standardized factorial scores were correlated with Internet frequency and number of online informational and recreational activities. Table 2 shows descriptive statistics for these variables, and Kolmogorov- Smirnov tests for normality. Variables included Internet Frequency (1 to 4), amount of Informational Activities online (1 to 7), amount of Recreational Activities online (1 to 7), standardized factorial scores on the Internet Skills factors derived before. Table 3 shows the correlation coefficients (Spearman for non-normal variables) between Internet skills and Internet habitual use. Given the number of contrasts, significance level was set at p < .01.
As seen in Table 3, Operational skills exhibit a significant, positive and moderate correlation with Internet habitual use; participants who spend more hours online, and use the Internet for more activities would possess more operational skills. On the other hand, Formal/Navigation skills and Informational/Evaluation skills only show significant, positive and weak correlations with the number of Informational activities online, whereas their correlation with the number of Recreational activities was not significant.
This study sought to validate the Internet Skills Scale proposed by van Deursen et al. (van Deursen & van Dijk, 2016; van Deursen et al., 2012). We have translated and adapted the Internet Skills Scale for operational, formal/navigation, and informational/evaluation skills, and administered the self-report instrument to a sample of young college adults. The factorial structure obtained replicated the three original factors (van Deursen & van Dijk, 2016; van Deursen et al., 2012), as well as showed acceptable reliability for the scales. As for the methodological approach, this study replicated and extended the work of van Deursen and colleagues (van Deursen & van Dijk, 2016; van Deursen et al., 2012), with a factor analytic approach more appropriate for an ordinal scale (Baglin, 2014).
Two items that were part of the Informational factor in van Deursen et al. (van Deursen & van Dijk, 2016; van Deursen et al., 2012) loaded instead in the Formal factor; they specifically referred to search skills (Find the information you were looking for, Use more than one search keyword). These items thus belong to the navigation set of skills more than to the examination of content and sources. The item asking if they went beyond the first three top search results belonged to the Evaluation factor; this suggests that this practice requires a critical examination of the search results (Salmerón et al., 2015).
Reliability indexes were lower than those originally obtained, although in the acceptable range, around .70. Reliability was lower for the Operational subscale, perhaps due to the fact that technological changes might have rendered items outdated (e.g. one asks for the frequency of “watching videos” which has become more prevalent in recent years). Future research could consider other operational aspects that render individual differences in this dimension.
In addition, results showed that frequency of Internet use, and number of informational and recreational activities were correlated with Operational skills, suggesting that more basic operational skills or “button knowledge” were associated with more frequent Internet use and more diverse Internet habits. In contrast, Formal/Navigation skills and Informational/Evaluation skills were only significantly associated with the number of Informational activities online. This pattern of results provides concurrent validation for self-assessment of skills.
This study examined the contribution of previous knowledge, working memory, and operational and navigation Internet skills, to text comprehension in a lesson embedded in an e-learning course. In an offline session, students completed the Internet Skills Scale and working memory tests. Afterwards, students accessed an e-learning course specifically designed for research purposes at their home or habitual place of study. Lessons consisted of static, scroll down scientific expository texts, but presented hierarchical navigation menus and were embedded in an e-learning platform. Comprehension was assessed with literal and inferential questions. Hypertext and digital text comprehension studies found that previous knowledge, working memory (Amadieu & Salmerón, 2014; Burin, Kahan, Irrazabal, & Saux, 2013) and operational and navigation skills (Fajardo et al., 2016; Hahnel et al., 2016; OECD, 2011) contribute to digital text comprehension, and the effect of Internet skills depends on level of previous knowledge and working memory capacity. This study sought to extend these results to a lesson embedded in an e-learning course.
A sub-sample of Study 1, 127 first-year undergraduate psychology students (age M = 23.23, SD = 5.78) from a large, public Argentinian university volunteered to participate in this study and completed all tasks. The study was approved by the institutional IRB.
In addition to the questionnaires and Internet Skills scale from Study 1, participants also completed working memory tests, and reading comprehension tasks online.
Verbal working memory tests
Digit Span Backwards and Letter-Number Sequencing WAIS III subtests (adapted versions, Wechsler, 2003). Scoring was computed as number correct, taking into account the discontinue rule, as described in the Manual (Wechsler, 2003).
Reading comprehension tasks
Four expository texts with a similar argumentative structure (a general concept, two subordinate concepts, details about them, a general conclusion relating both) and length, around 710–720 words were employed (see Burin et al., 2015). Two versed on high (Memory, Language) and two on low (Astronomy, Physics) previous knowledge topics according to prior testing (Burin et al., 2013). They were text-based lessons within a course, presented with a linked hierarchical overview, and decorative images. Each was followed by 12 comprehension questions: ten literal and inferential (T/F), and two (multiple-choice) about global inferences. Reading and comprehension questions were implemented within courses (i.e., each study condition was a course) on Moodle 2.5, hosted at a different institutional server than the university. Figures 1 and 2 show examples of a course and a lesson, respectively.
Participants completed the Internet Skills Scale, and then the working memory tasks, in small group sessions. At their house or habitual place of study they completed an e-learning course. Each course presented two texts, each one followed by questions (they could not return to the texts), and a questionnaire about their strategies for completing the tasks (not analyzed here). After completing all the tasks they were debriefed about the study.
The courses were hosted in an institutional server, different from the university, and were ostensibly not related to their college course. There were four courses, two for each level of previous knowledge (high or low previous knowledge texts), to counterbalance presentation order.
Two participants had missing scores in Internet skills and were eliminated for further analyses, rendering the total sample n = 125. A composite measure of working memory capacity (WM) was computed for each participant, averaging z scores over both working memory tests. Also for each participant, factorial scores in each Internet Skill factor derived from Study 1 was expressed as z scores. Regarding comprehension questions, reliability for comprehension accuracy in each text (number of correct responses) was: α Memory = .708; α Language = .786; α Astronomy = .777; α Physics = .796. Given that each participant saw two texts, for each participant, total correct answers in both texts was computed (Comprehension Total). Variables were distributed normally according to Kolmogorov-Smirnov’s Z (all p > .05). Table 4 shows descriptive statistics for these variables.
The contribution of previous knowledge, Internet skills and working memory on comprehension was analyzed with a generalized linear regression model for a normal probability distribution with identity link. Working memory, Operational and Navigational z scores were predictors, previous knowledge was entered as a factor, and z scores in Comprehension was the dependent variable. The model tested for main effects, and the interaction between previous knowledge and WM with Operational and Navigational skills. The model was significant compared to an intercept only one, Likelihood ratio test χ2 (8) = 17.04, p = .03. Parameters are shown in Table 5. Only WM, B = 0.214, 95% CI = [0.012, 0.415], Wald (1) = 4.321, p = .016, and Navigation, B = − 0.345, 95% CI = [− 0.625, − 0.064], Wald (1) = 5.795, p = .038, contributed significantly to comprehension. Higher scores in WM, and lower in Navigation, were significantly associated with better comprehension.
Hypertext and complex digital text comprehension studies found that general verbal ability, and/or previous knowledge of the subject matter (Amadieu & Salmerón, 2014; Coiro, 2011a; Leu et al., 2014; OECD, 2011) working memory (DeStefano & LeFevre, 2007; Gwizdka, 2009; Lee & Tedder, 2003; Naumann et al., 2008) and operational and navigation skills (Fajardo et al., 2016; Hahnel et al., 2016; Naumann & Salmerón, 2016; OECD, 2011) contribute, solely and in interaction, to digital text comprehension. This study tested whether these factors influenced comprehension of a text embedded in an e-learning course, solved remotely. Only working memory and navigation were significantly associated with text comprehension. As expected, increased working memory capacity led to better accuracy in answering comprehension questions.
Regarding previous knowledge, even though it has been associated with comprehension, it did not affect comprehension results in this study, neither directly nor moderating other variables. One possible explanation for this result is the task employed: answering literal and inferential questions immediately following the lesson could be more a function of memory than elaboration of the content. In this regard, the strongest predictor in this study has been working memory. Another possible explanation for this result arises from the remote testing situation: students could implement strategies, such as taking notes, making screenshots, or other means to increase accuracy. These possibilities should be considered in applied settings where e-learning lessons are followed by test questions.
Regarding Internet skills, navigation impacted performance in a negative way: higher navigation scores led to decreases in performance. As compared to previous studies (Fajardo et al., 2016; Hahnel et al., 2016; Naumann & Salmerón, 2016; OECD, 2011; Salmerón et al., 2015) employing complex dynamic digital texts, in this case the lesson required less navigation but was presented in a web environment. Complex dynamic digital text comprehension scenarios tested in previous studies comprise diverse websites (for example, social media, government websites, forums or e-mail chains), and tasks vary from finding explicit information, to making global comprehension inferences about content and authors, to expressing opinions. In comparison, the task presented here is more similar to e-learning lessons, or in the case of an open web task, reading a single Wikipedia result. Taking into account that these scores reflect self-reported search and navigation behavior, and is associated with the number of informational activities students usually perform online, this result would concur with the large PISA study (OECD, 2011) finding that intensive use of computers at home (as compared to moderate use) leads to worse comprehension. Possibly the students more accustomed to search and navigate are less able, or less predisposed to, reading thoroughly a single, attention demanding, scroll-down page.
General discussion and conclusions
This study examined the contribution of self-reported Internet operational and navigational skills, previous domain knowledge, and working memory capacity, to the comprehension of scientific texts embedded as lessons in an e-learning course, taken remotely by college aged students. Only working memory and navigation were significantly associated with text comprehension: higher working memory, and lower scores in self-reported navigation behavior, led to better comprehension.
This study has employed a self-report measure of Internet skills, as different from performance-based laboratory assessments. On one hand, this can be a limitation of the study, given that participants can over- or under report their real performance, and can be biased by participants beliefs and self perception. On the other hand, both types of measures have their benefits and shortcomings and complement each other (Hargittai, 2005). Controlled performance tasks show effective behavior, as opposed to the hypothetical questions of self-reports, but generally limit the navigation and do not incorporate other features of the Internet environment (such as commercial links and content, or other applications simultaneously open). Also, they do not capture the participants’ perception of their actions in everyday Internet use. In addition, they are more laborious to measure, both in terms of needed equipment and in testing time. Future research can compare both types of measures.
We found that working memory capacity increased accuracy in answering questions about a scientific text, but that previous knowledge did not have an effect. This could suggest that answering questions immediately following the lesson in an e-learning environment like the one implemented here could be tapping immediate memory rather than elaboration of the content, and thus is another limitation of this study. Furthermore, in this remote testing situation participants could appeal to strategies, such as taking notes, making screenshots, or other means to increase accuracy. So then, a lesson followed by closed questions tapping literal and inferential information in e-learning could be measuring memory, or test-taking strategies. Therefore, teachers and instructional design in e-learning should consider this before implementing the common practice of including closed questions after a lesson as a comprehension check.
This study also found that students who self-reported more skilled search and navigation performed worse. As different from other studies, where complex dynamic web scenarios are presented, and tasks vary from searching literal information, to construct global inferences comparing multiple sources (Fajardo et al., 2016; Hahnel et al., 2016; OECD, 2011; Salmerón et al., 2015), in this case students had to read a text based lesson (or Wikipedia-like page) and answer questions about it within an e-learning platform. In those previous studies, skilled navigation was required for successful performance: it is not only a question of intensive computer use (OECD, 2011) but knowing when to use navigation, interpreting search findings, and constructing a correct, integrated mental model of the content (Salmerón et al., 2015). In this regard, we employed a self-report measure of Internet skills; scores in this scale were associated with online frequency and number of Internet activities. In the PISA study (OECD, 2011), self-reported intensive use of computers at home also was associated with worse comprehension. Overall, then, perhaps students who are typical digital “power users” do not approach an attention demanding text lesson with the appropriate abilities and strategies.
These results could be taken into account in the design of instructional materials, given that the most frequent vehicle to deliver expository content in blended or fully online courses is still the copyrighted textbook (Allen & Seaman, 2016). Also, future research can focus on the strategies students adopt to understand and learn expository contents in e-learning.
Allen, I. E., & Seaman, J. (2016). Online Report Card: Tracking Online Education in the United States. Babson Survey Research Group.
Amadieu, F., & Salmerón, L. (2014). Concept maps for comprehension and navigation of hypertexts. In R. Hanewald, & D. Ifenthaler (Eds.), Digital knowledge maps in education, (pp. 41–59). New York: Springer.
Baddeley, A. D. (2010). Working memory. Current Biology, 20(4), 136–140. https://doi.org/10.1016/j.cub.2009.12.014.
Baglin, J. (2014). Improving your exploratory factor analysis for ordinal data: A demonstration using FACTOR. Practical Assessment, Research & Evaluation, 19(5), 1–15.
Barreyro, J. P., Cevasco, J., Burin, D. I., & Molinari, C. (2012). Working memory capacity and individual differences in the making of reinstatement and elaborative inferences. Spanish Journal of Psychology, 15(2), 471–479. https://doi.org/10.5209/rev_SJOP.2012.v15.n2.38857.
Bawden, D. (2008). Origins and concepts of digital literacy. In C. Lankshear, & M. Knobel (Eds.), Digital literacies: Concepts, policies and practices, (pp. 17–32). New York: Peter Lang.
Burin, D. I., Kahan, E., Irrazabal, N., & Saux, G. (2013). Comprensión de contenidos científicos enformato hipertextual: La estructura de navegación tiene efectos distintos según el conocimiento previo y lacapacidad de memoria de trabajo [Scientific hypertext comprehension: Navigation structure as a function ofprevious knowledge and working memory capacity] . In V. Jaichenco (Ed.), Psicolingüística en español.Homenaje a Juan Seguí [Spanish psycholinguistics: A tribute to Juan Segui]. Buenos Aires: Eudeba.
Burin, D. I., Barreyro, J.P., Saux, G., & Irrazabal, N. (2015). Navigation and comprehension of expository hypertext: Hypertext structure, previous domain knowledge, and working memory capacity. Electronic Journal of Research in Educational Psychology, 13(3), 529–550. https://doi.org/10.14204/ejrep.37.14136.
Clark, R., & Mayer, R. E. (2016). E-learning and the science of instruction, (4th ed., ). San Francisco: Pfeiffer.
Coiro, J. (2011a). Predicting reading comprehension on the internet: Contributions of offline reading skills, online reading skills, and prior knowledge. Journal of Literacy Research, 43(532–392). https://doi.org/10.1177/1086296X11421979.
Coiro, J. (2011b). Talking about reading: Modeling the hidden complexities of online reading comprehension. Theory Into Practice, 50(107–115). https://doi.org/10.1080/00405841.2011.558435.
Dabbagh, N., & Kitsantas, A. (2004). Supporting self-regulation in student-centered web-based learning environments. International Journal on E-Learning, 3(1), 40–47.
Daneman, M., & Merikle, P. M. (1996). Working memory and language comprehension: A meta-analysis. Psychonomic Bulletin & Review, 3, 422–433. https://doi.org/10.3758/BF03214546.
DeStefano, D., & LeFevre, J. A. (2007). Cognitive load in hypertext reading: A review. Computers in Human Behavior, 23, 1616–1641. https://doi.org/10.1016/j.chb.2005.08.012.
Fajardo, I., Villalta, E., & Salmerón, L. (2016). ¿Son realmente tan buenos los nativos digitales? Relación entre las habilidades digitales y la lectura digital [Are really digital natives so good? Relationship between digital skills and digital reading]. Anales de Psicología, 32, 89–97. https://doi.org/10.6018/analesps.32.1.185571.
Ferrando, P. J., & Lorenzo-Seva, U. (2013). Unrestricted item factor analysis and some relations with item response theory. In Technical Report. Tarragona: Department of Psychology, Universitat Rovira i Virgili.
Fraillon, J., Ainley, J., Schulz, W., Friedman, T., & Gebhardt, E. (2014). Preparing for life in a digital age. The IEA international computer and information literacy study international report. Amsterdam: Springer, IEA.
Graesser, A. C., Leon, J. A., & Otero, J. C. (2002). Introduction to the psychology of science text comprehension. In J. Otero, J. A. Leon, & A. C. Graesser (Eds.), The psychology of science text comprehension, (pp. 1–15). Mahwah: Erlbaum.
Gwizdka, J. (2009) Assessing Cognitive Load on Web Search Tasks, The Ergonomics Open Journal, 2 (1), 114–23. https://doi.org/10.2174/1875934300902010114.
Hahnel, C., Goldhammer, F., Naumann, J., & Kröhne, U. (2016). Effects of linear reading, basic computer skills, evaluating online information, and navigation on reading digital text. Computers in Human Behavior, 55, 486–500. https://doi.org/10.1016/j.chb.2015.09.042.
Hargittai, E. (2005). Survey measures of web-oriented digital literacy. Social Science Computer Review, 23(3), 371–379. https://doi.org/10.1177/0894439305275911.
IBMcorp (2012). IBM SPSS statistics 21 for windows. Armonk: IBM Corp.
International Test Commission, I. T. C. (2005). ITC guidelines for translating and adapting tests: Document ITC-G-TA 20140617.
Kintsch, W. (1998). Comprehension. A paradigm for cognition. Cambridge: Cambridge University Press.
Lee, M. J., & Tedder, M. C. (2003). The effects of three different computer texts on readers’ recall: Based on working memory capacity. Computers in Human Behavior, 19(6), 767–783. https://doi.org/10.1016/S0747-5632(03)00008-6.
Leu, D. J., Kiili, C., & Forzani, E. (2014). Individual differences in the new literacies of online research and comprehension. In P. Afflerbach (Ed.), Handbook of individual differences in reading: Reader, text, and context. New York: Routledge.
Lorenzo-Seva, U., & Ferrando, P. J. (2006). FACTOR: A computer program to fit the exploratory factor analysis model. Behavior Research Methods, 38(1), 88–91. https://doi.org/10.3758/BF03192753.
Lorenzo-Seva, U., & Ferrando, P. J. (2013). FACTOR 9.2: A comprehensive program for fitting exploratory and semiconfirmatory factor analysis and IRT models. Applied Psychological Measurement, 37(6), 497–498. https://doi.org/10.1177/0146621613487794.
Naumann, J., Richter, T., Christmann, U. & Groeben, N. (2008). Working memorycapacity and r eading skill moderate the effectiveness of strategy training in learning fromhypertext. Learning and Individual Differences, 18, 197–213. https://doi.org/10.1016/j.lindif.2007.08.007.
Naumann, J., & Salmerón, L. (2016). Does navigation always predict performance? Effects of relevant page selection on digital reading performance are moderated by offline comprehension skills. The International Review of Research in Open and Distributed Learning, 17, 42–59. https://doi.org/10.19173/irrodl.v17i1.2113.
OECD (2009). PISA 2009 Assessment Framework. Key competencies in reading, mathematics and science.
OECD (2011). PISA 2009 Results. Students on line: Digital technologies and performance.
Perfetti, C., & Stafura, J. (2014). Word knowledge in a theory of reading comprehension. Scientific Studies of Reading, 18(1), 22–37. https://doi.org/10.1080/10888438.2013.827687.
Peters, G. J. Y. (2014). The alpha and the omega of scale reliability and validity: Why and how to abandon Cronbach’s alpha and the route towards more comprehensive assessment of scale quality. The European Health Psychologist, 16(56–69).
Rouet, J.-F., Ros, C., Goumi, A., Macedo-Rouet, M., & Dinet, J. (2011). The influence of surface and deep cues on primary and secondary school students' assessment of relevance in Web menus. Learning and Instruction, 21(2), 205–219. https://doi.org/10.1016/j.learninstruc.2010.02.007.
Salmerón, L., Cerdán, R., & Naumann, J. (2015). How adolescents navigate Wikipedia to answer questions. Infancia y Aprendizaje: Journal for the Study of Education and Development, 38(435–471). https://doi.org/10.1080/02103702.2015.1016750.
Salmeron, L., Kintsch, W., & Cañas, J. (2006). Coherence or interest as basis for improving hypertext comprehension. Information Design Journal, 14(1), 45–55.
Tabachnick, B. G., & Fidell, L. S. (2001). Using multivariate statistics, (4th ed., ). Boston: Allyn & Bacon.
van Deursen, A. J. A. M., & van Dijk, J. A. G. M. (2009). Improving digital skills for the use of online public information and services. Government Information Quarterly, 26, 333–340. https://doi.org/10.1016/j.giq.2008.11.002.
van Deursen, A. J. A. M., & van Dijk, J. A. G. M. (2016). Modeling traditional literacy, internet skills and internet usage: An empirical study. Interacting with Computers, 28(1). https://doi.org/10.1093/iwc/iwu027.
van Deursen, A. J. A. M., van Dijk, J. A. G. M., & Peters, O. (2012). Proposing a survey instrument for measuring operational, formal, information and strategic internet skills. International Journal of Human-Computer Interaction, 28(12), 827–837. https://doi.org/10.1080/10447318.2012.670086.
Wechsler, D. (2003). WAIS III: Test de Inteligencia para Adultos [Wechsler Adult Intelligence Test]. Buenos Aires: Paidós.
A preliminary report of this research was accepted and presented at the 26th Annual Meeting of the Society for Text and Discourse (Kassel, Germany, 2016).
We thank Federico Martín González and Yamila Coccimiglio for their help with data collection.
This work was supported by Secretaría de Ciencia y Técnica, Universidad de Buenos Aires UBACYT 20020150100024BA and Agencia Nacional de Promoción Científica y Tecnológica PICT-2015-2706 grants.
Availability of data and materials
Data and materials will be provided by the corresponding autor upon request.
The authors declare that they have no competing interests.
About this article
Cite this article
Burin, D.I., Irrazabal, N., Ricle, I.I. et al. Self-reported internet skills, previous knowledge and working memory in text comprehension in E-learning. Int J Educ Technol High Educ 15, 18 (2018). https://doi.org/10.1186/s41239-018-0099-9
- Internet skills
- Working memory
- Text comprehension
- Previous knowledge
- Factor analysis