125 Studies in Second Language Learning and Teaching Department of English Studies, Faculty of Pedagogy and Fine Arts, Adam Mickiewicz University, Kalisz SSLLT 13 (1). 2023. 125-149 https://doi.org/10.14746/ssllt.31990 http://pressto.amu.edu.pl/index.php/ssllt Construct validation of the revised Metacognitive Awareness of Reading Strategies Inventory (MARSI-R) and its relation to learning effort and reading achievement Abdullah Alamer King Faisal University, Alhasa, Suadi Arabia https://orcid.org/0000-0003-4450-0931 aa.alamer@kfu.edu.sa Ahmad Alsagoafi King Faisal University, Alhasa, Suadi Arabia https://orcid.org/0000-0001-6788-3870 aalsagoafi@kfu.edu.sa Abstract Language students apply different strategies to learn a second language (L2), es- pecially when they want to attain proficiency in reading. The aim of the present study was to revisit the validity of the Metacognitive Awareness of Reading Strat- egies Inventory (MARSI-R) among Saudi students using a new statistical method of confirmatory composite analysis (CCA). Past studies modeled MARSI-R as a common factor and applied confirmatory factor analysis (CFA) to test its validity. However, studies struggled to provide support for the validity of the MASRI-R with each suggesting different model. Instead, we treat the inventory as a composite, meaning that the items in MARSI-R form and define the inventory and not the other way around. We use partial least squared structural equation modeling (PLS-SEM) to allow the composite model to be estimated. The results indicated that the constructs of MARSI-R are better operationalized as composites not com- mon factors as supported through CCA exclusively. After confirming the nature of the inventory, we evaluated the extent to which MARSI-R is related to reading proficiency through the mediational mechanism of motivational intensity (i.e., learning effort). Descriptive statistics illustrated that problem-solving strategies Abdullah Alamer, Ahmad Alsagoafi 126 are the most used strategy and that females used the strategies more frequently than their male counterparts. Most importantly, the structural model showed that metacognitive reading strategies only exert an indirect effect on reading profi- ciency, suggesting that the effect of strategies is mediated by motivational inten- sity (i.e., learning effort). Thus, motivational intensity seems to be mediator in the relationship between metacognitive reading strategies and reading proficiency. Finally, methodological and educational implications are provided. Keywords: metacognitive reading strategies; MARSI; MARSI-R; construct valid- ity; confirmatory composite analysis (CCA); confirmatory factor analysis (CFA); partial least squared structural equation modeling (PLS-SEM) 1. Introduction Metacognitive reading strategies are one of the variables that are related to sec- ond language (L2) learners’ success in reading comprehension and proficiency (Cai & Kunnan, 2020). Interest in the role of metacognitive reading strategies is observable in research studies and edited volumes in the field (see, for example, Cai & Kunnan, 2020; Cohen, 2011; Mokhtari & Reichard, 2002, Oxford, 2017; Phakiti, 2006; Purpura, 1997; among others). In L2 research, the concept of met- acognitive reading test-taking strategies is closely related to students’ cognition process regarding reading and the self-awareness underlying comprehension of the text (Mokhtari & Reichard, 2002). Although the importance of metacogni- tive reading strategies has been documented in the literature, the field agrees that they are multifaceted and not easy to assess (Cohen, 2011). A prominent measure of metacognitive reading strategies is the MARSI (Mokhtari & Reichard, 2002). The questionnaire was designed to evaluate learners’ awareness of meta- cognitive reading strategies when they read academic texts. The MARSI has three subscales: global reading strategies (GRS), problem-solving strategies (PSS), and sup- port reading strategies (SRS). In essence, the MARSI instrument (Mokhtari & Reich- ard, 2002) was originally designed to assess the metacognitive awareness of read- ing strategies among L1 students, not L2 students, although several authors have used it since its inception to measure the metacognitive awareness of reading strategies among L2 students as well. The modified version, the MARSI-R (Mokhtari et al., 2018), is a short version that the authors created by reducing the number of items in the original scale from 30 to 15 based on theoretical and empirical reasons. Mokhtari et al. (2018) provided preliminary evidence of the constructs’ validity and reliability. However, methodologically, psychometric properties of MARSI-R have yet to be established and further investigations are warranted to obtain the best model that mirrors Construct validation of the revised Metacognitive Awareness of Reading Strategies Inventory . . . 127 the theoretical underpinnings of MARSI-R considering recent advances in the field of structural equation modeling (SEM; Dirsehan & Henseler, 2022; Hair & Alamer, 2022; Henseler & Schuberth, 2021; Schuberth, 2021; Sparks & Alamer, 2023). Ac- cordingly, the present study attempts to answer three research questions: RQ1: How do L2 Saudi undergraduate students of English use the MARSI-R for reading learning? RQ2: What is the best model representation of the internal structure of MARSI-R? RQ3: Are metacognitive reading strategies directly or indirectly linked to L2 students’ reading proficiency through motivational intensity (criterion- related validity)? 2. Literature review Interest in investigating learning processes in L2 research and theory has in- creased over the past five decades. The shift from examining teaching methods to investigating learning processes has led researchers to focus on the relationship between the processes and products of second language acquisition (SLA) re- search and learners’ strategy use (Purpura, 1997). Similarly, language testing re- searchers have long been interested in examining the effect of candidates’ cogni- tive characteristics on test performance (Kunnan, 1995; Phakiti, 2003; Purpura, 1997). In test situations, test takers apply language use strategies to enhance their test scores (Cohen, 2011). More specifically, Bachman and Palmer (1996, 2010) proposed a theoretical framework for cognitive and metacognitive strategy use in language testing contexts. Phakiti (2008) classified metacognitive strategies into planning, monitoring, and evaluation strategies. Building on anxiety research in psychology, Phakiti (2008) characterized Bachman and Palmer’s (1996, 2010) “strategic competence” as consisting of trait metacognition (i.e., knowledge about cognition) and state metacognition (i.e., regulation of cognition). 2.1. Comprehension processes and strategies for reading texts Cohen and Upton (2007) maintained that readers consciously and purposefully “do exert a significant level of active control over their reading process through the use of strategies” (p. 211). Specifically, the employment of cognitive strate- gies to comprehend reading texts assists students with making sense of a text, while metacognitive strategies help to monitor and appraise reading processes (Griffiths, 2018; Schallert & Martin, 2003). Nassaji (2003) noted that cognitive and metacognitive comprehension strategies are used at multiple levels to distinguish Abdullah Alamer, Ahmad Alsagoafi 128 between skilled and less-skilled English as a second language (ESL) readers. For this reason, high-proficiency learners who frequently use reading strategies should not always be considered better readers (Prichard, 2014). That is, strat- egy use should be tailored to personal choices because not all individuals may find a given strategy beneficial in all contexts and for all purposes (Griffiths & Soruç, 2020). Carrell (1998) showed that successful strategy use and strategy instruction are context-dependent. In addition, Rivera-Mills and Plonsky (2007) argued that there is no recipe for learning strategies that would be suitable for everyone. In order for test takers to succeed in test situations and identify the right answer, “The test items should invoke the construct-relevant strategies L2 test developers intend to use” (Lin et al., 2019, p. 197). Language testing research has focused on identifying the aspects that cause variation in test takers’ performance, chief among which are processing strategies (Bachman, 1990). In language testing, Cohen and Upton (2007) clas- sified strategies into three main categories: language learner strategies (e.g., confirming final understanding of the passage based on the content and/or the discourse structure), test management strategies (e.g., making an educated guess using background or extra-textual knowledge), and test wiseness strate- gies (e.g., using clues from other items to answer the item under consideration). Cohen (2006) considered test management and test wiseness strategies to be test-taking strategies, and drew a clear distinction between those and language learner strategies. Test management strategies are used to respond meaning- fully to test items and tasks, whereas test wiseness strategies are used to re- spond to language tests without the use of L2 knowledge and performance abil- ity (Cohen, 2013). Test-taking strategies can assist test takers with overcoming the challenges they face in test situations. Gebril (2018) noted that giving test takers appropriate test-taking strategies can enhance their test performance. Based on this distinction, this study will investigate the metacognitive strategies students use when taking a reading test at the B2 test. 2.2. Studies on strategies used during reading tests and test performance Language testing researchers have recognized the crucial role of cognitive and metacognitive strategies in the variation in language test performance. For in- stance, Purpura (1998) examined the impact of metacognitive strategy use on candidates’ B2 test performance and found variation across the high- and low- ability groups in terms of lexico-grammatical and reading ability, based on the test takers’ use of retrieval and metacognitive strategies. Similarly, Phakiti (2006) investigated cognitive and metacognitive strategies in relation to 358 stu- dents’ reading test performance using the structural equation modelling (SEM) Construct validation of the revised Metacognitive Awareness of Reading Strategies Inventory . . . 129 approach. He found that (1) memory and retrieval strategies promoted the can- didates’ performance through comprehension strategies; (2) monitoring strate- gies had an “executive function” in relation to memory strategies, while evalu- ating strategy-controlled retrieval strategies; (3) planning strategies did not di- rectly control memory, retrieval, or comprehension strategies, yet they con- trolled such cognitive strategies through monitoring and evaluation strategies; and (4) only comprehension strategies had a direct impact on students’ perfor- mance in the EFL reading test (Phakiti, 2006). Similarly, Zhang et al. (2014) ex- plored the relationships between test takers’ metacognitive and cognitive strat- egy use and their test performance after sitting for an English as a foreign lan- guage (EFL) reading test. Using a multi-sample SEM approach, their study com- prised a total of 593 students. The findings of the study indicated that students’ strategy use significantly influenced their lexico-grammatical reading ability. In contrast, Lin et al. (2019) found in their study, which had a total sample of 552 test takers, that only cognitive strategies had a direct impact on test takers’ read- ing test performance, whereas metacognitive strategies indirectly influenced test takers’ performance. However, metacognitive strategies had a strong effect on cognitive strategies. Recently, Wallace et al. (2021) investigated EFL readers’ metacognitive strategy use in terms of gender and reading ability. Using the MARSI, they found that the participants in their study used PSS more frequently than any other strategy. GRS was used significantly more often than SRS. More- over, their findings showed that gender differences did not affect strategy use. 2.3. MARSI and MARSI-R The metacognitive awareness of reading strategies inventory (MARSI) was de- veloped by Mokhtari and Reichard (2002). It is a self-report scale constructed to measure students’ generalized use and awareness of different metacognitive reading strategies applied to the reading and comprehension of texts. The MARSI scale has been useful in the field; however, the authors identified issues pertaining to the original scale, such as its appropriateness for college and adult readers, time consumption, and the generalized nature of the directions (for an in-depth discussion, see Mokhtari et al., 2018). In an attempt to resolve these issues, the researchers made a few modifications and shortened the scale. In its modified version (i.e., the MARSI-R), the scale was reduced from 30 to 15 items, as some statements covered the same reading strategy constructs. The MARSI- R measures the same three constructs: GRS, PSS, and SRS. Mokhtari et al. (2018) validated the MARSI-R by using confirmatory factor analysis (CFA). In sum, based on the misspecifications and related substantive considerations in the original MARSI, the number of items was significantly reduced, and the validity of the Abdullah Alamer, Ahmad Alsagoafi 130 MARSI-R was claimed to be established. Nonetheless, empirical studies have been struggling with replicating this refined version in other socio-cultural contexts. Although the inventory has undergone different validity assessments (see for example Ondé et al., 2022), this was only done using CFA (by assuming MARSI-R as common factor). Nevertheless, CFA is not the most appropriate analytical tool for validating such composite constructs as we will explain later. 2.4. Motivational intensity as a mediator Learning an L2 is more likely to be successful when the language learners exert effort necessary to integrate more with the language tasks. Similarly, learners who are aware of a set of learning strategies are less likely to proceed well in the language learning process if they do not put effort necessary for the learning to happen (Alamer, 2022a, 2022b; Gardner, 2010; Griffiths, 2018). Motivational in- tensity is defined as the expended effort the language learners exert to learn the language (Gardner, 2010). As such, it can be hypothesized that for the metacog- nitive reading strategies to explain students’ reading achievement, learning effort should be considered as a means that translates into the awareness and use of met- acognitive reading strategies. That is, it is postulated that learners who recognize which metacognitive reading strategies to use and then expend the learning effort are likely to have higher scores on reading achievement. Thus, motivational inten- sity can be said to operate as a proximal variable that mediates the effect of met- acognitive reading strategies on reading achievement (Alrabai & Alamer, 2022; Griffiths, 2018). This indirect process has been less studied in the field; as such, the present study investigates the potential meditating role of motivational in- tensity between the predictor and the outcome. 2.5. Methodological issues related to the validity of MARSI-R and other related constructs The dominant type of model used in the field for testing the internal structure of language learning strategies in general and MARSI-R in specific is the common factor model. Common factor models are typically applied through exploratory factor anal- ysis (EFA), and CFA (Alamer, 2022c; Alamer et al., in press). In common factor models, the items (i.e., indicators) are conceptually viewed as interchangeable and similar in meaning; thus, the measurement model assumes that the items similarly reflect the construct (Schuberth, 2021; Schuberth et al., 2022). As such, items should be highly correlated because they are targeting one specific domain from slightly different an- gles. In this way, any item could be removed from the construct without changing its overall meaning (Hair & Alamer, 2022). When the construct follows this definition, Construct validation of the revised Metacognitive Awareness of Reading Strategies Inventory . . . 131 it should be referred to as a latent variable or common factor (Schuberth et al., 2018). In the common factor model, the relationship goes from the factor to the items, meaning that if the conceptional meaning of the latent variable changes, all items should be changed simultaneously. In contrast to the common factor, emergent variable or composite comes as another type of construct configuration. The composite postulates that items con- stitute (i.e., define) the construct in a linear combination. Each item in the compo- site holds unique information about the construct that is not expressed by the re- maining items in the scale/inventory (Alamer et al., in press). Therefore, removing any item from the model likely alters the construct meaning (Henseler & Schuberth, 2021). Accordingly, items in the composite model should not be highly correlated. When the construct follows this definition, it should be referred to as a composite (also called emergent variable and formative construct) (Dirsehan & Henseler, 2022; Schuberth et al., 2018). Hence, the relationship in the composite model goes from the items to the construct, meaning that the indictors independently form the con- struct. An example for a construct that follows composite definition from L2 litera- ture is language achievement (e.g., Sparks & Alamer, 2022, 2023). See Figure 1 for a visual representation of latent and emergent variables. Figure 1 Visual representation of CCA (on the left) and CFA (on the right) When using CFA, the researcher assumes the constructs as latent varia- bles. However, forcing emergent variables to work under the common factor approach (i.e., through CFA) is inappropriate. Nonetheless, metacognitive read- ing strategies perfectly fit the definition of emergent variable because each item on the inventory represents a specific (i.e., distinct) aspect of the construct that is conceptually different from other items in the inventory. Alternatively, confirma- tory composite analysis (CCA) has been developed to allow for the assessment of emergent variables (Henseler, & Schuberth, 2021; Schuberth et al., 2018). To apply CCA, researchers often rely on partial least squares structural equation modeling (PLS-SEM). PLS-SEM is a composite-based method that allows emergent variables Abdullah Alamer, Ahmad Alsagoafi 132 (i.e., composite) to be easily estimated (Alamer et al., 2022; Dirsehan & Henseler, 2022; Hair & Alamer, 2022). Among its benefits, PLS-SEM allows emergent varia- bles to be estimated without identification issues that CFA faces. It allows higher- order constructs made of (caused by) lower-order constructs to be easily specified without special requirements (see Alrabai & Alamer, 2022 for an empirical exam- ple in L2 research). We believe that applying PLS-SEM to assess the composite model as well as the structural model is an important methodological contribu- tion to the evaluation of MARSI-R and other similar measures in the field (see Hair & Alamer, 2022 for PLS-SEM guidelines in L2 domain). Based on the available lit- erature, which will be discussed next, the postulated relationships in the process model are depicted in Figure 2. Figure 2 The structural (conceptual) model of the relationships linking metacog- nitive reading strategies, motivational intensity, and reading proficiency (Con- structs represented in hexagon denote composite; those represented in oval de- note common factor; those represented in rectangle denote observable variable (sum/mean value)) 3. Method 3.1. Participants The participants in the present study were undergraduate students enrolled in one language level (intermediate level) at two Saudi public universities. The original invitation was sent to around 200 students enrolled in the two universi- ties. Convenient sampling was utilized, and 108 students completed the ques- tionnaire. All participants spoke Arabic as their first language and were enrolled in the Department of English studying English as an L2. Their ages ranged from 18 to 35 years (Mage = 22.73, SD = 2.72). The participants were students nested in their classes. Thus, the language level of the participants was largely similar (intermediate), which was also confirmed by their teachers. Regarding gender composition, 30.23% of the participants were male, while 69.77% were female. All students enrolled in the Department of English at the two universities were invited via a message sent using a Telegram channel dedicated to student announcements. Construct validation of the revised Metacognitive Awareness of Reading Strategies Inventory . . . 133 Those who were willing to participate simply clicked the link provided in the mes- sage to access an online questionnaire created using Google Forms. A total of 108 students completed the online questionnaire. The study procedure was reviewed and approved by the Department of English in the involved universities. The par- ticipants provided their written informed consent in the online questionnaire to participate in this study. The participants were not trained or prepared to take part in the reading test used this study. Also, they had not received any instruc- tions about reading strategies, though some textbooks from some language courses may have described some types of strategies. 3.2. Instruments 3.2.1. MARSI-R This study utilized the revised version of the MARSI questionnaire (MARSI-R), which consists of 15 items. Mokhtari et al. (2018) reduced the number of items measuring global strategies in the original version from 30 to 15 in the revised version. Partici- pants were asked to indicate the extent to which they use the following strategies when they read English texts on a 5-point scale ranging from “I have never heard of this strategy before” to “I know this strategy quite well, and I often use it when I read.” The three subscales comprising the MARSI-R are: (1) GRS, for which an exam- ple item is “Previewing the text to see what it is about before reading it,” (2) PSS, for which an example item is “Getting back on track when getting sidetracked or distracted,” and (3) SRS, for which an example item is “Using reference materials such as dictionaries to support my reading” (see Appendix A for full scale items). 3.2.2. Motivational intensity Motivational intensity (also called effort in the literature) was measured using Gardner’s (2010) scale. The scale has five items and is a self-reported measure that uses a 5-point Likert-type response format. The participants were asked to indicate the extent to which each statement represented their efforts. An exam- ple item is “I really work hard to learn English.” The internal consistency reliabil- ity of this variable in this study was .80. 3.2.3. Reading proficiency This study used the B2 First test, formerly known as Cambridge English: First (FCE). The B2 test is a test of the English language at level B2 based on the Common European Framework of Reference (upper-intermediate level). The test measures Abdullah Alamer, Ahmad Alsagoafi 134 four skills: listening, reading, writing, and speaking. Of the seven parts in the read- ing test, the researchers selected a free sample part which was a reading passage followed by eight questions from the Cambridge website (see an example in Ap- pendix B). The whole reading test lasts for one hour 15 minutes. The focus of this part of the test is on detail, opinion, attitude, tone, purpose, main idea, gist, meaning from context, implication and text organization features (e.g., exemplifi- cation, reference). It follows the format of a text followed by six 4-option multiple- choice questions. The reading part of the B2 test was given to 108 students who participated in the study right after they had filled out the general information part about their reading levels. This was an external measure which functioned as a self-perception measure of students’ reading ability (Mokhtari et al., 2018). The item reads as follows: “I consider myself (1) an excellent reader, (2) a good reader, (3) an average reader, or (4) a poor reader.” The aim of this item was to be included in the reading achievement construct as it hypothetically contributes to students achievement of the language. The test was administrated to the students, and they were given the choice to withdraw if they have changed their mind about participation. Because this is a one item scale, no reliability test was reported. 3.3. Statistical analyses 3.3.1. Data screening and assumptions Before we analyzed the data, a preliminary check was conducted for missing data, normality, and outliers. Although normality of the data is not assumed by PLS-SEM (Hair & Alamer, 2022), it was assessed graphically and statically for extreme values by checking the Q-Q plots as well as the skewness and kurtosis values to check. Outliers are data points that depart from the rest of the data points. In addition, a check was performed to ensure that the dataset was free from carelessness and intentional idiosyncrasies (e.g., answering all items with one response). 3.3.2. Construct validity of the composite model To evaluate PLS-SEM models, researchers are required to assess the composite model first before assessing the structural model. The evaluation of the compo- site differs from common factor as it relies on three different indices (Hair & Al- amer, 2022; Dirsehan & Henseler, 2022): (1) convergent validity, which is estab- lished by evaluating the relationship between the composite (also called forma- tive measure) with a measure that generally reflect the phenomenon. In our case, we evaluate the extent to which the three strategies are positively associated with a general statement dictating the extent to which students use metacognitive reading Construct validation of the revised Metacognitive Awareness of Reading Strategies Inventory . . . 135 strategies as follows: “Overall, I use different metacognitive reading strategies” a scale out of five are presented from “quite frequently” to “quite rarely;” (2) lack of multicollinearity, which is assessed by variance inflation factor (VIF) with values of 5 or greater indicating serious collinearity; and (3) the size and significance of the indicators weights and loadings of composite constructs, which should be positive and significant. We also report the SRMR of the composite model with SRMR < .08 as the suggested cut-off value (Dirsehan & Henseler, 2022). 3.3.3. Evaluating the structural model The structural model was examined using two measures: (1) the coefficient of determination (R2) in the outcome variable and (2) PLSpredict which assess the out-of-sample predictive power (Hair & Alamer, 2022; Sparks & Alamer, 2023). In PLSpredict, we compare the value of root mean squared error (RMSE) in the PLS model with the naïve linear regression model (LM). The model has good predic- tive power when it generates lower RMSE values in the PLS model compete to the LM model (Hair & Alamer, 2022). The structural model should also be free from collinearity issues by inspecting the VIF value in the path coefficients. Note that PLS-SEM can handle latent and emergent variables in one structural model (Alamer et al., 2022; Hair & Alamer, 2022). 4. Results 4.1. Preliminary analysis Data screening revealed no concerns regarding missing data, carelessness, or outliers. However, visual inspection of the data through Q-Q plots suggested that the data depart slightly from normality. The skewness and kurtosis values confirmed this, as seven variables slightly violated the cut-off values. Accord- ingly, normality was not ideally established; thus, we used tests and estimations that are robust to non-normal and ordinal data. 4.2. Main analyses To answer RQ1 (How do L2 Saudi undergraduate students of English use the MARSI-R for reading learning?), in general, PSS was the most reported type of strategy (M = 4.07, SD = .71), followed by SRS (M = 3.83, SD = .84) and GRS (M = 3.64, SD = .75). To investigate whether male and female students hold similar endorsements of these strategies, a t-test (Welch’s version) was conducted. Levene’s homogeneity test indicated that the equal variances assumption was Abdullah Alamer, Ahmad Alsagoafi 136 met (p values > .05). Results of the Welch’s test showed that gender was signifi- cantly different for GRS (Welch’s t = -4.59, df = 76.83, p < .001, d = -.92, d 95% CI [-1.36, -.47]) and SRS (Welch’s t = -5.03, df = 70.37, p < .001, d = -1.03, d 95% CI [- 1.51, -.59]) in favor of females (see Figure 3), while PSS did not show significant differences (Welch’s t = -1.67, df = 82.36, p = .09, d = -.33, d 95% CI [-.73, .07]). Figure 3 Descriptive plots of the mean differences in SRS, GRS, and PSS with a 95% CI To answer RQ2 (What is the best model representation of the internal struc- ture of MARSI-R?), we evaluate the composite model through CCA (Alamer et al., in press; Henseler & Schuberth, 2021; Schuberth, 2021) to establish the construct va- lidity of MARSI-R which is a precedent of assessing the structural model. First, con- vergent validity was tested by regressing each construct of MARSI-R on the single- item measure of the general use of metacognitive reading strategies. The results indicated that GRS, PSS, and SRS predicted the general metacognitive reading strat- egies use item sufficiently (β = .70, .59 and .50, respectively); all regression paths were significant at p < .001. Next, an assessment of multicollinearity was considered using VIF measure. Our analysis showed all items were below 3 (i.e., VIF < 3) indi- cating lack of collinearity in the indicators. Finally, the sizes and significance of the indicators weights and loadings as shown in Table 1 illustrated that all weights are positive and all were significant at p < .01, with an exception to one item on PSS Construct validation of the revised Metacognitive Awareness of Reading Strategies Inventory . . . 137 (item 5) whose weight was relatively weak. However, following Hair and Alamer (2022) guideline, items on composite measure should not be removed automatically based on empirical suggestions but must be informed by theory. Hair and Alamer (2022) explain that an item in composite may be considered for removal if both weight and loading are significantly negative which is not the case here. In addition, we believe that retaining this item “Guessing the meaning of unknown words or phrases” is conceptually justified to maintain the fuller understanding of the PSS. Note that having the loading of this item significant (i.e., p value < .05) we have further reason to retain this item. Table 1 Item weights and loadings of GRS, PSS, SRS in the composite (CCA) and common factor models (CFA) Items Composite weights in CCA model Composite loadings in CCA model Factor loadings in CFA model GRS items purpose .21 .45 .51 preview .26 .52 .84 check .31 .67 .77 typographical .34 .72 .37 analyzing .37 .65 .40 PSS items get back .34 .70 .82 adjust .36 .61 .39 stopping .40 .77 .52 re-reading .29 .65 .73 guessing .06 .23 .30 SRS items notes .39 .65 .66 aloud .21 .35 .35 discuss .24 .51 .29 underly .52 .91 .73 reference .14 .44 .28 Correlation GRS with PSS .55 .43 PSS with SRS .55 .63 GRS with SRS .56 .52 Note. All items are significant at p < .01 To ensure that composite model is a more appropriate representation of the inventory than the common factor model, we compare the results of CCA to CFA. A CFA model with three factors has resulted in a poor fitted solution (i.e., x2 = 333.73, df = 87, p < .001, CFI = .53, TLI = .44, SRMR = .12, RMSEA = .16). More importantly, there are several factor loadings in the model that are weak in magnitude (i.e., < .50) which, when assessed through common factor model, are candidate for removal to achieve better model fit. In contrast, the model fit Abdullah Alamer, Ahmad Alsagoafi 138 in the CCA showed that SRMR was .08 and that the composite weights and load- ings were in the expected directions. Therefore, our analysis supports MARSI-R through the composite model exclusively, thus we continue with this type of model specification when evaluating the structural model. Figure 4 The structural model linking metacognitive reading strategies, motiva- tional intensity, and reading achievement (Italicized and gray values represent 95% bias-corrected confidence intervals (CI). Dashed lines indicate non-signifi- cant paths. Constructs represented in hexagon denote composite constructs) Table 2 Standardized indirect and total effects in the structural model Paths β p CI 95%* Metacognitive reading strategies -> motivational intensity -> reading proficiency .12 .02 .05, .26 Metacognitive reading strategies (total effect) .32 <.01 .39, .68 Note. * Based on the bias-corrected confidence intervals To answer RQ3 (Are metacognitive reading strategies directly or indirectly linked to L2 students’ reading proficiency through motivational intensity?), we run the structural model within PLS-SEM. The assessment of the structural model as shown in Figure 4 starts with inspecting the R2 value on the outcome variable, which was found to be medium in size (R2 = .20) (Hair & Alamer, 2022). The direct effects are presented in Figure 4, along with their 95% confidence intervals (CIs), while the indirect and total effects are provided in Table 2. We ran PLSpredict analy- sis to assess the model’s out-of-sample prediction ability, following Shmueli et al.’s (2019) and Hair and Alamer’s (2022) recommendation. Our results showed that the model had good out-of-sample predictive power, that is, the PLS model showed lower errors (i.e., RMSE = .935) compared to the linear regression (LM) model (i.e., RMSE = .983). This implies that the hypothesized structural model has Construct validation of the revised Metacognitive Awareness of Reading Strategies Inventory . . . 139 predicted scores that are unused when executing the PLS model. Thus, the analy- sis illustrates evidence of the external validity of our results. As is evident in Figure 4, metacognitive reading strategies only exerted a direct effect on motivational intensity, and that effect was moderate in size (β = .29, 95% CI: [.05, .49]); metacognitive strategies had no direct effect on reading proficiency. Only motivational intensity directly affected reading proficiency, and that effect was moderate in magnitude (β = .40, 95% CI: [.22, .55]). Thus, it can be said that motiva- tional intensity functioned as a mediator in the relationship between metacognitive reading strategies and reading proficiency. This was substantiated by inspecting the indirect effect, which was significant (β = .07, 95% CI: [.01, .21]). Regarding the total effect, a strong link was found between metacognitive reading strategies and read- ing proficiency (β = .52, 95% CI: [.39, .68]). Overall, the structural model substanti- ated the nature of the association between metacognitive reading strategies and reading proficiency. Most importantly, although the effect of metacognitive reading strategies on reading proficiency was not significant, it was substantial and meaning- ful when considering the mediator (i.e., motivational intensity). 5. Discussion The present study aimed to replicate the findings pertaining to the MARSI-R by assessing its construct and criterion validity using the advanced method of PLS- SEM in the Saudi context. As for the RQ1, the present study suggested that a model for the MARSI-R, where SRS, PSS, and GRS are operationalized as common factor model through CFA is infeasible, and thus not supported. As psychometric research explained (Schuberth et al., 2018), CFA are typically used to assess com- mon factor models that assume items to be interchangeable, representing a uni- dimensional construct, and share very similar meaning. Common factor models conducted through CFA requires the items to be highly correlated as well as inter- changeable, such that item removal does not affect the overall meaning of the construct (Alamer & Marsh, 2022; Alamer et al., in press). However, items on met- acognitive reading strategies inventory (among other constructs with a similar structure) hold unique details about the construct such that any removal likely changes the conceptual meaning of the construct (Alrabai & Alamer, 2022; Henseler & Schuberth, 2021; Sparks & Alamer, 2022, 2023). Our selection of CCA to assess the composite model excellently reflected this empirical observation and dis- played that MARSI-R should be operationalized as emergent variable, not latent variable (Henseler & Schuberth, 2021; Schuberth, 2021). In this regard, the findings of this research add important details about the psychometric properties of certain types of scales in the field (Mokhtari et al., 2018; Mokhtari & Reichard, 2002; Phakiti, 2008; Purpura, 1997). In fact, in their Abdullah Alamer, Ahmad Alsagoafi 140 MARSI-R validation study, Mokhtari et al. (2018) reported that they encountered “numerous cross-loadings for items and correlated errors between items” while fitting the standard CFA model (p. 227). Empirically, it is vital to inform the field of the expected challenges when forcing the constructs to work according to the other construction domain (e.g., forcing composite to operate in a common factor model). If such misspecified model is assumed, researchers are often obliged to apply data-driven modifications to compensate for ill-fitted models (Alamer, 2022c; Alamer & Marsh, 2022; Alrabai & Alamer, 2022). Nonetheless, procedures such as multiple removal of items and multiple correlation of error terms nega- tively affect the quality of the internal structure of the models, which conse- quently provide questionable evidence of the construct validity of the instrument under assessment (Shao et al., 2022). We encourage researchers in metacognitive reading strategies in general to consider composite models in their analysis. With respect to RQ2, the results of this study showed that PSS was the most reported type of strategy. This may indicate that these students were stra- tegic when they faced reading difficulties reading English texts. This finding is consistent with Wallace et al. (2021), who found that their study participants used PSS more frequently than any other strategy. In contrast, our study found that SRS was used more frequently than GRS. This may suggest that the students in this study are independent and autonomous in their learning and thus need less help from their teachers. On the other hand, this study identified significant gendered differences in GRS and SRS in favor of females, while PSS did not show significant differences. This gendered difference may suggest that females were actively engaged during the reading process, which, in turn, would be related to positive reading comprehension. This is consistent with Chambers-Cantrell and Carter (2009), who found that females used PSS, SRS, and GRS more than males. With respects to RQ3, metacognitive reading strategies appeared to be only indirectly related to reading proficiency through motivational intensity (i.e., learn- ing effort). This is consistent with Cohen and Upton (2007), who found that readers consciously and purposefully tend to be persistent in their reading process through the use of strategies. Only motivational intensity exerted a direct effect on reading proficiency, and that effect was moderate in magnitude. This finding supports the conclusion that higher levels of reading engagement, can lead to more proficient reading comprehension (Guthrie et al., 2004). This conclusion is similar to what L2 motivation literature often reports (see Alamer, 2022a, 2022b; Oxford, 2017). Thus, it can be said that motivational intensity can be a mediator in the relationship be- tween metacognitive reading strategies and reading proficiency. Regarding the total effect, it was found that metacognitive reading strategies were moderately linked to reading proficiency, considering the two paths. Overall, the structural model sub- stantiated the nature of the association between metacognitive reading strategies Construct validation of the revised Metacognitive Awareness of Reading Strategies Inventory . . . 141 and outcomes, which is consistent with mainstream L2 metacognitive research (Co- hen, 2011; Cohen & Upton, 2007; Phakiti, 2008). 6. Pedagogical implications The present study has pedagogical implications for language learning and teach- ing. The findings of this research showed that male students reported less use of the three reading metacognitive strategies, with PSS and SRS being significantly different across the two genders. Therefore, L2 teachers can promote the use of these strategies among their students, with a particular focus on male students. Although females are often seen as more active in applying these strategies (see Griffiths, 2018) teachers may keep this in mind to remind their male students about these strategies’ usefulness for improving their reading proficiency. Teach- ers should also highlight metacognitive strategies’ conditional effect of increasing students’ reading attainment. As the present study has shown the indirect effect from reading metacognitive strategies to reading achievement, this might indicate that the effect of reading metacognitive strategies is better understood through effort (motivational intensity). Therefore, teachers should ensure that students not only beware of the available reading metacognitive strategies to employ but also, students need to be reminded that exerting effort and showing persistence to achieve the reading proficiency is key for the effect of strategies to be observed. 7. Limitations Although the present study provided insight into the validity of the MARSI-R model and the effects of reading metacognitive strategies on reading proficiency, it has limitations. First, although the study achieved the minimum sample size re- quired to run PLS-SEM (Hair & Alamer, 2022), the sample size was not sufficiently large to generalize the results to the Saudi population. However, we can hypoth- esize that students who share similar characteristics as the study participants may exhibit similar patterns regarding the use of reading metacognitive strategies, thus leading to a similar conclusion. This is supported by the results of the PLSpredict analysis as it showed that our model predicted unseen scores in the original anal- ysis, thus supporting the external validity of the results (Alamer et al., 2022). In addition, the present study was limited by the number of variables in- cluded in reading metacognitive strategies’ predictive literature. It is known from the literature that learning strategies do not operate in isolation of other individual difference variables (Oxford, 2017). Our study involved the evaluation of motivational intensity as a possible mediator, and the results showed that its inclu- sion was rather meaningful; thus, including other antecedents and consequences Abdullah Alamer, Ahmad Alsagoafi 142 of metacognitive strategy use can be beneficial for theoretical and empirical re- search in this area. 8. Conclusion In conclusion, our study achieved three main objectives related to deepening the understanding of metacognitive strategies. The first aim was to replicate the findings of previous studies on the MARSI-R in the context of Saudi Arabia. We investigated validity of the MARSI-R considering a composite model, not a com- mon factor model. To do so, we applied CCA first and compared it to CFA. We found that MARSI-R is only supported through CCA which suggested that MARSI- R is made of emergent variables, not latent variables. Moreover, we evaluated the extent to which Saudi students use the three strategies and whether there were significant gendered differences in their use of the GRS, PSS, and SRS sub- scales. Finally, we tested the explanatory power of the MARSI-R in a mediational model in which motivational intensity was positioned as a mediator in the rela- tionship between reading metacognitive strategies and reading proficiency. Our findings illustrated the ways in which reading metacognitive strategies are con- nected to reading performance. This research offers theoretical, methodologi- cal, and pedagogical implications for the field of language learning. Acknowledgment This work was supported by the Deanship of Scientific Research, Vice Presidency for Graduate Studies and Scientific Research, King Faisal University, Saudi Arabia [Grant No. 2670]. Construct validation of the revised Metacognitive Awareness of Reading Strategies Inventory . . . 143 References Alamer, A. (2022a). Basic psychological needs, motivational orientations, effort, and vocabulary knowledge: A comprehensive model. Studies in Second Language Acquisition, 44, 164-184. https://doi.org/10.1017/S027226312100005X Alamer, A. (2022b). Having a single language interest autonomously predicts L2 achievement: Addressing the predictive validity of L2 grit. System, 108, 102850. https://doi.org/10.1016/j.system.2022.102850 Alamer, A. (2022c). Exploratory structural equation modeling (ESEM) and bifac- tor ESEM for construct validation purposes: Guidelines and applied exam- ple. Research Methods in Applied Linguistics, 1, 100005. https://doi.org/10. 1016/j.rmal.2022.100005 Alamer, A., Al Khateeb, A., & Jeno, L. (2022). Using WhatsApp increases L2 stu- dents’ self-motivation and achievement, and decreases learning anxiety: A self-determination theory approach. Journal of Computer Assisted Learn- ing. https://doi.org/10.1111/jcal12753 Alamer, A., & Marsh, H. (2022). Exploratory structural equation modeling in second lan- guage research: An applied example using the dualistic model of passion. Studies in Second Language Acquisition. https://doi.org/10.1017/S0272263121000863 Alamer, A., Schuberth, F., & Henseler, J. (in press). Confirmatory composite anal- ysis: A tool for assessing inventories and questionnaires in language learn- ing research. Language Learning. Alrabai, F., & Alamer, A. (2022). The role of learner character strengths and class- room emotions in L2 resilience. Frontiers in Psychology. 13, 956216. https:// doi.org/10.3389/fpsyg.2022.956216 Bachman, L. F. (1990). Fundamental considerations in language testing. Oxford University Press. Bachman, L. F., & Palmer, A. (1996). Language testing in practice: Designing and developing useful language tests. Oxford University Press. Bachman, L. F., & Palmer, A. S. (2010). Language testing in practice (2nd ed.). Oxford University Press. Cai, Y., & Kunnan, A. J. (2020). Mapping the fluctuating effect of strategy use ability on English reading performance for nursing students: A multi-lay- ered moderation analysis approach. Language Testing, 37, 280-304. https:// doi.org/10.1177/0265 532219893384 Carrell, P. L. (1998). Can reading strategies be successfully taught. Australian Re- view of Applied Linguistics, 21, 1-20. Chambers Cantrell, S. & Carter, J. (2009). Relationships among learner character- istics and adolescents’ perceptions about reading strategy use. Reading Psychology, 30, 195-224. https://doi.org/10.1080/02702710802275397 Abdullah Alamer, Ahmad Alsagoafi 144 Cohen, A. D. (2006). The coming of age of research on test-taking strategies. Language Assessment Quarterly, 3, 307-331. Cohen, A. D. (2011). L2 learner strategies. In E. Hinkel (Ed.), Handbook of re- search in second language learning and teaching (Part V). Methods and instruction in second language teaching: Vol. II (pp. 681-698). Routledge. Cohen, A. D. (2013). Using test-wiseness strategy research in task development. In A. J. Kunnan (Ed.), The companion to language assessment – Vol. 2: Approaches and development, Part 7: Assessment development. Wiley and Sons. Cohen, A. D., & Upton, T. A. (2007) “I want to go back to the text:” Response strat- egies on the reading subtest of the New TOEFL. Language Testing, 24(2), 209-250. https://doi.org/10.1177/0265532207076364 Dirsehan, T., & Henseler, J. (2022). Modeling indices using partial least squares: How to determine the optimum weights? Quality & Quantity, 1-15. https://doi.org/10.1007/s11135-022-01515-5 Gardner, R. (2010). Motivation and second language acquisition: The socio-edu- cational model. Peter Lang. Gebril, A. (2018). Test preparation in the accountability era: Toward a learning-ori- ented approach. TESOL Journal, 9, 4-16. https://doi.org/10.1002/tesj.302 Griffiths, C. (2018). The strategy factor in successful language learning: The tor- nado effect. Multilingual Matters. Griffiths, C., & Soruç, A. (2020). Individual differences in language learning. Palgrave. Guthrie, J. T., Wigfield, A., Barbosa, P., Perencevich, K. C., Taboada, A., Davis, M. H., Sca- fiddi, N. T., & Tonks, S. (2004). Increasing reading comprehension and engage- ment through concept-oriented reading instruction. Journal of Educational Psy- chology, 96(3), 403-423. https://doi.org/10.1037/0022-0663.96.3.403 Hair, J., & Alamer, A. (2022). Partial least squares structural equation modeling (PLS-SEM) in second language and education research: Guidelines using an applied example. Research Methods in Applied Linguistics, 1(3), 100027. https:// doi.org/10.1016/j.rmal.2022.100027 Henseler, J., & Schuberth, F. (2021). Confirmatory composite analysis. In J. Henseler (Ed.), Composite-based structural equation modeling: Analyzing latent and emergent variables (pp. 179-201). The Guilford Press. Kunnan, A. J. (1995). Test taker characteristics and test performance: A structural modeling approach. Cambridge University Press. Lin, L., Lam, J. W.-I., & Tse, S. K. (2019). Test takers’ strategy use and L2 Chinese reading test performance in mainland China: A structural equation ap- proach. Studies in Educational Evaluation, 60, 189-198. https://doi.org/10. 1016/j.stueduc.2019.01.002 Mokhtari, K., Dimitrov, D. M., & Reichard, C. A. (2018). Revising the Metacogni- tive Awareness of Reading Strategies Inventory (MARSI) and testing for Construct validation of the revised Metacognitive Awareness of Reading Strategies Inventory . . . 145 factorial invariance. Studies in Second Language Learning and Teaching, 8(2), 219-246. https://doi.org/10.14746/ssllt.2018.8.2.3 Mokhtari, K., & Reichard, C. A. (2002). Assessing students’ metacognitive aware- ness of reading strategies. Journal of Educational Psychology, 94(2), 249- 259. https://doi.org/10.1037/0022-0663.94.2.249 Nassaji, H. (2003). Higher-level and lower-level text processing skills in advanced ESL reading comprehension. Modern Language Journal, 87, 261-276. Ondé, D., Jiménez, V., Alvarado, JM., & Gràcia, M. (2022). Analysis of the struc- tural validity of the reduced version of metacognitive awareness of read- ing strategies inventory. Frontiers in Psychology, 13, 894327. https://doi.org/ 10.3389/fpsyg.2022.894327 Oxford, R. (2017). Teaching and researching language learning strategies: Self- regulation in context (2nd ed.). Routledge. Phakiti, A. (2003). A closer look at the relationship of cognitive and metacogni- tive strategy use to EFL reading achievement test performance. Language Testing, 20, 26-56. https://doi.org/10.1191/0265532203lt243oa Phakiti, A. (2006). Modeling cognitive and metacognitive strategies and their re- lationships to EFL reading test performance. Melbourne Papers in Lan- guage Testing, 11, 53-95. Phakiti, A. (2008). Construct validation of Bachman and Palmer’s (1996) strate- gic competence model over time in EFL reading tests. Language Testing, 25(2), 237-272. Prichard, C. (2014). Reading strategy use of low- and high-proficiency learners and the effect of reading instruction. Bulletin of Higher Education Oka- yama University, 10, 115-122. Purpura, J. E. (1997). An analysis of the relationships between test takers’ cog- nitive and metacognitive strategy use and second language test perfor- mance. Language Learning, 47, 289-325. Purpura, J. E. (1998). Investigating the effects of strategy use and second lan- guage test performance with high- and low-ability test takers: A structural equation modeling approach. Language Testing, 15(3), 333-379. https://doi. org/10.1177/026553229801500303 Rivera-Mills, S. V., & Plonsky, L. (2007). Empowering students with language learning strategies: A critical review of current issues. Foreign Language Annals, 40, 535-548. Schallert, D. L., & Martin, D. B. (2003). A psychological analysis of what teachers and students do in the language arts classroom. In J. Flood, D. Lapp, J. R. Squire, & J. M. Jensen (Eds.), Handbook of research on teaching the Eng- lish language arts (pp. 31-45). Erlbaum. Abdullah Alamer, Ahmad Alsagoafi 146 Schuberth, F. (2021). Confirmatory composite analysis using partial least squares: Setting the record straight. Review of Managerial Science, 15, 1311-1345. https://doi.org/10.1007/s11846-020-00405-0 Schuberth, F., Henseler, J., & Dijkstra, T. K. (2018). Confirmatory composite analysis. Frontiers in Psychology, 9(2541). https://doi.org/10.3389/fpsyg.2018.02541 Schuberth, F., Rademaker, M. E., & Henseler, J. (2022). Assessing the overall fit of composite models estimated by partial least squares path modeling. Euro- pean Journal of Marketing. https://doi.org/10.1108/EJM-08-2020-0586 Shao, K., Elahi Shirvan, M., & Alamer, A. (2022). How accurate is your correlation? Different methods derive different results and different interpretations. Fron- tiers in Psychology, 13, 901412. https://doi.org/10.3389/fpsyg.2022.901412 Shmueli, G., Sarstedt, M., Hair, J., Cheah, J., Ting, H., Vaithilingam, S., & Ringle, C. (2019). Predictive model assessment in PLS-SEM: Guidelines for using PLSpredict. European Journal of Marketing, 53(11), 2322-2347. https://doi. org/10.1108/EJM-02-2019-0189 Sparks, R., & Alamer, A. (2022). Long-term impacts of L1 language skills on L2 anx- iety: The mediating role of language aptitude and L2 achievement. Language Teaching Research. https://doi.org/10.1177/13621688221104392 Sparks, R., & Alamer, A. (2023). How does first language achievement impact sec- ond language reading anxiety? Exploration of mediator variables. Reading and Writing. https://doi.org/10.1007/s11145-022-10410-2 Wallace, M., Li, V., Huang, T., & He, N. (2021). Metacognitive strategy use for EFL readers: Differences in gender and reading ability. TESL-EJ, 25(2), 1-22. Zhang, L. M., Goh, C. M. C., & Kunnan, A. J. (2014). Analysis of test takers’ meta- cognitive and cognitive strategy use and EFL reading test performance: A multi-sample SEM approach. Language Assessment Quarterly, 11, 76-120. Construct validation of the revised Metacognitive Awareness of Reading Strategies Inventory . . . 147 APPENDIX A MARSI-R scale items (Mokhtari et al., 2018) Strategy scale 1. I have never heard of this strategy before. 2. I have heard of this strategy, but I don’t know what it means. 3. I have heard of this strategy, and I think I know what it means. 4. I know this strategy, and I can explain how and when to use it. 5. I know this strategy quite well, and I often use it when I read. Strategies 1-15 01. Having a purpose in mind when I read. 02. Taking notes while reading. 03. Previewing the text to see what it is about before reading it. 04. Reading aloud to help me understand what I’m reading. 05. Checking to see if the content of the text fits my purpose for reading. 06. Discussing what I read with others to check my understanding. 07. Getting back on track when getting side tracked or distracted. 08. Underlining or circling important information in the text. 09. Adjusting my reading pace or speed based on what I’m reading. 10. Using reference materials such as dictionaries to support my reading. 11. Stopping from time to time to think about what I’m reading. 12. Using typographical aids like bold face and italics to pick out key information. 13. Critically analyzing and evaluating the information read. 14. Re-reading to make sure I understand what I’m reading. 15. Guessing the meaning of unknown words or phrases. Abdullah Alamer, Ahmad Alsagoafi 148 APPENDIX B An example of a B2 reading test Construct validation of the revised Metacognitive Awareness of Reading Strategies Inventory . . . 149