Microsoft Word - Dandotkar_etal_final.doc Journal of Effective Teaching in Higher Education, vol. 5, no. 1 https://doi.org/10.36021/jethe.v5i1.251 Knowing About Knowing: Assessing How Reflective Writing Influences Undergraduate Students’ Epistemic Beliefs Srikanth Dandotkar, University of Southern Indiana, sri.dandotkar@gmail.com Laura E. Cruz, Pennsylvania State University Jeffrey R. Stowell, Eastern Illinois University M. Anne Britt, Northern Illinois University Abstract. Epistemic beliefs are one’s assumptions about knowledge and knowing. Given the research in educational psychology that established epistemic beliefs as reliable predictors of student success, we devised a pedagogical intervention to improve students’ epistemic beliefs. In this study, we examined the effectiveness of the reflective writing task as a means of changing how students think about what is known. Students from two upper-level psychology classes (Cognitive and Research Methods) took a general epistemic belief survey by rating their agreements with 38 items at three different times in a semester (first-day, pre-reflective-writing task, and post-reflective-writing task). Day 1 responses were utilized to validate the survey items using principal component analysis—three variables (Knowledge Construction and Modification-KCM, Structure of Knowledge-KST, and Meaning of Successful Students-SS) emerged. The intervention successfully improved students’ beliefs specific to Structure of Knowledge-KST and Meaning of Successful Students- SS, beliefs that predict student learning. This study suggests that even short interventions have the potential to influence students’ beliefs about knowledge, which have been shown to have demonstrable effects on their academic success. Keywords: epistemic beliefs; epistemic reflection; non-cognitive factors; reflective writing In his 2020 essay entitled “What’s the Problem Now?,” Randy Bass (2020) argues that learning is a sufficiently complex construct and that we (as scholars and instructors) should shift from viewing it as a problem to be solved to viewing it as an opportunity to exercise our collective imagination to find novel pathways through it. His approach was affirmed by a recent meta-analysis of published scholarship of teaching and learning (SoTL) articles that concluded that a majority of the studies under-theorized learning, focusing primarily on assessing cognitive outcomes related to specific course material (Manarin et al., 2021). This disconnect has not occurred because of a lack of theory from which to draw. Educational theorists have long recognized that learning has multiple dimensions beyond the cognitive (e.g., behavioral, affective), and the more recent scholarship has articulated additional layers, with particular interest in the metacognitive and so- called non-cognitive factors that significantly influence learning (Kautz et al, 2014). The term “non-cognitive” is perhaps a misnomer as the components (e.g., grit, growth mindset, curiosity, self-concept, goal orientation) are not completely distinct Knowing About Knowing Journal of Effective Teaching in Higher Education, vol. 5, no. 1 2 from cognitive or other thinking processes, but they do share a common positionality in that they often foreground how students learn in the classroom (Barrett, 2014). Several high-profile studies have established strong theoretical and applied linkages between the cultivation of positive non-cognitive traits and improved student outcomes, especially in, but not limited to, STEM fields (Caviglia- Harris & Maier, 2020). Carol Dweck’s (Dweck, 2008; Yeager et al., 2019) studies of fixed versus growth mindset in middle school mathematics students and Duckworth et al.’s (2007) work with grit in high-achieving college students are perhaps two of the best-known examples. That said, recently both grit and growth mindset have been criticized as weak predictors of academic success (Maddi et al., 2017), and the search continues for a fuller understanding of the interplay between non-cognitive factors and student learning (Stankov & Lee, 2014). Several scholars have posited the significance of epistemic beliefs as non-cognitive factors that influence student learning. Epistemic beliefs are defined as a student’s assumptions about knowledge (Hofer & Pintrich, 1997; Schommer, 1990) and knowing (Schommer, 1990). According to Schommer (1990), epistemic beliefs are a system of independent beliefs, and these beliefs vary on a continuum of sophistication (see Table 1 for the list of beliefs and the unsophisticated and sophisticated views about these beliefs). As the table reflects, epistemic beliefs fall into five major categories: speed (how long will it take to learn), structure (how complex the subject matter is), construction (how new knowledge is created), success (what they need to do to master the subject), and objectivity (how they view the objectivity of the subject matter). Table 1 General Epistemic Beliefs on a Continuum of Sophistication Beliefs Unsophisticated View Sophisticated View B1. Speed of Knowledge- Acquisition-KSP Quick learning Slow learning B2. Structure of Knowledge-KST Simple knowledge Complex-knowledge B3. Knowledge Construction & Modification-KCM Passive-learning Active learning by questioning B4. Meaning of Successful Students-SS Innate Fixed ability Acquired Incremental ability B5. Obtaining Objective Truth-OT Certainty-knowledge Probabilistic knowledge Note: Taken from Schommer (1990). A number of seminal studies in educational psychology have established the validity of epistemic belief as a construct. Schommer-Aikins et al. (2005) found that middle Knowing About Knowing Journal of Effective Teaching in Higher Education, vol. 5, no. 1 3 school students’ epistemic beliefs directly predicted their overall GPA and their mathematics performance. In a series of studies, Schommer (1990; 1993) and Schommer et al. (1992) demonstrated that high sophistication in each of the Speed of Knowledge Acquisition-KSP, Structure of Knowledge-KST, and Obtaining Objective Truth-OT subscales (see Table 1) was significantly and independently correlated to secondary student learning across multiple disciplines, as measured either by mastery tests or student artifact analysis. More recently, scholars have engaged in secondary analysis of large education datasets (e.g., the Program for International Student Assessment or PISA) to identify statistically significant linkages between epistemic beliefs and student learning in a variety of national contexts (Khine et al., 2020; She et al., 2019). Karakolidis et al. (2019), for example, identified epistemic beliefs as significant mediators between high motivation and low achievement for 5,532 Greek students. Taken collectively, these studies of secondary students have served not only to establish the viability of epistemic belief as a construct but also to link it to student success. To date, however, there have been few, if any, commensurate studies of college students’ general epistemic beliefs. Rather, the scholarly emphasis had fallen largely on domain-specific ways of knowing, and studies of disciplinary epistemologies (e.g., beliefs about science or history) are prevalent in some fields (Donald, 1990; Park & Lee, 2004; Wiley et al., 2020). This perspective follows the disciplinary orientation of much work in SoTL (Huber, 2013), but the recent shift in emphasis towards integrated, cross-disciplinary learning as a critical component of general education has challenged the efficacy of this compartmentalization. In this context, scholars have worked to identify schemas that incorporate broad epistemic beliefs along with one, or more, domain-specific ways of knowing. The Theory of Integrated Domains in Epistemology (TIDE) model, for example, classifies beliefs as either domain-specific, general, personal, or all three (Muis et al., 2006; Muis et al., 2016), leading some scholars to argue that the respective constructs may serve different, though perhaps complementary, functions (Buehl & Alexander, 2006; Schraw, 2001). Accordingly, the current study explores the domain-general nature of epistemic beliefs, particularly in the less studied population of college students. In addition to debates over the relative salience of general versus domain-specific beliefs, researchers also question the extent to which such beliefs can be learned, and, if so, what appropriate developmental models may apply. The well-known Perry taxonomy (1970), for example, suggests levels of epistemic development across the four-year undergraduate experience. Schommer’s work (1993) affirms the latter stages of Perry’s model, indicating that beliefs characterized as sophisticated were prevalent among students graduating from four-year institutions. Further, King and Kitchener (1994) found that epistemic assumptions underlying reflective judgments of participants varied as a function of their number of years in college. Similarly, Schommer (1992) found that participants’ educational levels predicted their beliefs about the nature of knowledge: the higher the level of one’s education the less likely that the person would believe in simple (Structure of Knowledge-KST) and certain knowledge (Obtaining Objective Truth-OT). The logical corollary to the proposition that epistemic beliefs can be learned, at Knowing About Knowing Journal of Effective Teaching in Higher Education, vol. 5, no. 1 4 least while attending college, would be the assumption they can also be taught. However, it remains unclear what specific practices in college life provide students with the opportunity to realign their beliefs about knowledge and knowing. Numerous course- and program-based studies of domain-specific interventions exist (e.g., Kienhues et al., 2008), but these remain limited in replicability, especially with the emergence of studies emphasizing significant differences in epistemic belief development across both disciplines and levels of educational attainment (Rosman et al., 2017). Perhaps the most promising practice is epistemic reflection, a practice first introduced by Baxter Magolda (1992) in her longitudinal study of the epistemic beliefs of 101 incoming freshmen. Through their reflective writing exercises, Baxter Magolda argues, the students demonstrated the underlying connection between the development of their personal and academic epistemic beliefs. That said, her study follows students over a significant period (14 years), which invites questions regarding the appropriate timespan for achieving and, by extension, measuring changes in epistemic beliefs. The latter point about measuring is salient to the present study. Several researchers have noted persistent challenges in assessing deep-set beliefs as potential constructs are subject to variations in epistemic belief at the meta-level. Most studies to date have relied on a variety of validated self-report assessment instruments, with the Epistemic Belief Survey (EBS) used in the present study (Wood & Kardash, 2002) joining two others, the Epistemological Questionnaire (Schommer, 1990) and the Epistemic Beliefs Inventory (Schraw et al., 2002), as the predominant scales (DeBacker et al., 2008). Each of these tools are subject to the increasingly pointed criticisms of student self-report instruments in educational research, but they have been effective in demonstrating changes in epistemic belief over time, at least in the sense that these are articulated in the respective instruments. To both isolate when changes in epistemic beliefs occur and by what means, the present study utilizes a pre/post design with the EBS to assess changes in how psychology students view general ways of knowing after engaging in reflective writing exercises specifically designed to influence those beliefs. The Study The purpose of the current study was to explore the effectiveness of a reflective writing task in changing college students’ domain-general epistemic beliefs pertinent to speed (how long will it take to learn: KSP), structure (how complex the subject matter is: KST), construction (how new knowledge is created: KCM), success (what they think one needs to master the subject: SS), and objectivity (how they view the objectivity of the subject matter: OT) of knowledge or knowing. The effectiveness of the intervention was compared across two upper-level psychology classes to examine whether knowledge of research methods plays any role in ascertaining the effectiveness of the intervention. Participants’ domain-general epistemic beliefs specific to five dimensions (KSP, KST, KCM, SS, & OT) were captured by an epistemic belief survey consisting of 38 items (Wood & Kardash, 2002). While the specifics of this survey are provided in the “Instrument” section, we would like to briefly identify the variables in the study, Knowing About Knowing Journal of Effective Teaching in Higher Education, vol. 5, no. 1 5 how they were measured, and the purpose of using multiple analyses. The intervention, along with the pre- and post-assessments, served as a within-subjects factor, that is, the same participants took the survey before and after the intervention (reflective writing) while class (Research Methods & Cognitive) served as a between-subjects factor, that is, as two different groups. Participants’ epistemic scores, specific to five dimensions, served as the critical dependent measures. Before calculating the dependent measure for each dimension (i.e., average epistemic score), we conducted a Principal Component Analysis (PCA) to identify and isolate those items that were reliable and unique in capturing the assumed dimension. As will be discussed in the “Preliminary Results” section, this yielded 11 items that uniquely measured only three (KST, SS, KCM) of the original five dimensions. This resulted in three dependent measures, that is, average epistemic scores specific to KST, SS, & KCM dimensions. Consequently, we ran (as reported in the “Main Results”) three individual ANOVAs, one on each dependent measure (specific to KST, SS, & KCM). Appendix A lists all the identified 11 items under their respective dimensions. Additionally, Appendix B lists the original 38 items and the dimension to which each belongs. Further, we have reported these analyses under two different headers to underscore the different purposes these two analyses serve: PCA under Preliminary Analyses and ANOVAs under Main Results. We hypothesized that the epistemic belief scores for all five dimensions (or the number of factors the factor-analytic method yields, which you already know to be three factors) post-reflective task would be lower compared with pre-reflective task. A lower score indicates a more sophisticated view. This prediction is based on King and Kitchener’s (1994) findings specific to the reflective judgment task. We also hypothesized the effect of the intervention would be larger for the Research Methods class. This is because students in Research Methods classes learn about knowledge and knowing, which may give them an advantage in utilizing the intervention over others. Participants For this Institutional Review Board approved study, our initial sample consisted of 53 undergraduate students enrolled in two upper-division psychology courses (Cognitive Processes: n = 32; Research Methods: n = 21) at a medium-sized, state comprehensive university located in the midwestern region of the U.S. The two classes are comparable in that mostly second-semester juniors and seniors take these classes. Also, the background knowledge and characteristics of these students, though not measured, was comparable. The difference in sample size between the groups has to do with the limited class size of the Research Methods class. This should not pose an issue statistically as we tested the assumptions of the statistical analyses before interpreting their results. Also, there were four students who were a part of both the classes, so their data were excluded from the analyses, resulting in the sample size of 45. Although no demographic data were collected, the program-level data suggest that students majoring in psychology identify predominantly as a female and either white or mixed-race. Knowing About Knowing Journal of Effective Teaching in Higher Education, vol. 5, no. 1 6 Instrument We used a condensed version of the Epistemic Belief Survey (EBS) (Wood & Kardash, 2002) to measure epistemic beliefs about the nature and acquisition of knowledge. It has 38 items (e.g., “You can believe most things you read”) anchored on a 5-point Likert-type scale ranging from 1 (strongly-disagree) to 5 (strongly- agree). The EBS has five dimensions: Speed of Knowledge Acquisition (KSP), Structure of Knowledge (KST), Knowledge Construction and Modification (KCM), Meaning of Successful Students (SS), and the Attainability of Objective Truth (OT). Please see Appendix B for a complete list of these items. A total score for each dimension is computed by averaging its respective items. The EBS instrument itself is widely viewed as reliable and consistently demonstrates sound psychometric properties (Wood & Kardash, 2002). In a study validating this instrument, Wood & Kardash (2002) reported reliable Cronbach’s alpha values for KSP, KST, KCM, SS, and OT as .74, .72, .66, .58, and. 54, respectively. Procedure A paper-based EBS was administered on the first day of class (baseline measure), after week eight (pre-reflection), and after week nine (post-reflection). Students completed the survey during regular class time. No identifiable information was collected. Between the second and third administration, students read a review article by Jung (2011) that summarized the literature on epistemic beliefs and explained their role in education. Afterwards, they wrote a reflective essay in which they identified the nature of their own epistemic beliefs, explained the reason for their identification, and described what could be done to refine their epistemic beliefs. The purpose of this task was to trigger epistemic doubt (Bendixen, 2002), leading to dissonance and ultimately a resolution of the dissonance to establish more refined epistemic beliefs. Students participated in this study as part of a course assignment for which they received credit. Design and Analyses We adopted a mixed-methods design with the intervention as the within-subjects factor and class (Research Methods or Cognitive Processes) as the between- subjects factor. To determine which EBS dimensions were relevant to our sample, we used principal component analysis (PCA), which yielded three dimensions (reported in the Preliminary Results section below). Therefore, the average epistemic scores specific to each of these three epistemic dimensions alone served as the critical dependent measures (i.e., average epistemic scores pertinent to Structure of Knowledge-KST, Meaning of Successful Students-SS, and Knowledge Construction and Modification-KCM). We performed three separate 2 X 2 mixed- subjects ANOVAs, one on each identified dimension (reported in the Main Results subsection). Knowing About Knowing Journal of Effective Teaching in Higher Education, vol. 5, no. 1 7 Results Preliminary Results Given that the epistemic questionnaires such as the Epistemic Belief Survey (EBS) produce sample-specific results (Bråten, et al., 2019; Ferguson et al., 2013), we scored it using a principal component analysis (PCA). PCA is a dimensionality reduction technique that allows researchers to determine which items uniquely capture a latent-construct (e.g., Structure of Knowledge-KST) and no other dimensions. This suggests that the identified items for a given dimension would be reliable in measuring the supposed (sub-)construct (i.e., construct validity). Because only uniquely contributing items are identified to extract dimensions, two important consequences emerge: 1) not all items in a dimension are necessarily selected, and 2) not all dimensions (five in this survey) are necessarily captured. The EBS responses from the first day were used to extract latent variables (epistemic dimensions). As recommended by Ferguson et al. (2013), we removed items that were either unrelated (factor loading <1.5) or negatively related to other items in that dimension. This resulted in 15 items. An additional four items were removed because they loaded on more than one dimension. An initial principal component analysis with oblique rotation yielded three factors–structure: Structure of Knowledge-KST, Knowledge Construction and Modification-KCM, and Meaning of Successful Students-SS. Eleven items met the Kaisser-Guttman retention criteria of Eigenvalues greater than unity and explained 53.62% of sample-variation. These items were used to calculate average epistemic belief scores (dependent measure) for each dimension in both pre-reflection and post-reflection conditions. Please see Appendix A for the extracted dimensions and factor loadings related to each dimension. Main Results Three 2 X 2 mixed-subjects ANOVAs, one on each PCA-identified belief (KST, SS, & KCM) were conducted. It is a common practice to use quantitative analysis such as ANOVA when the dependent measures are calculated using student responses on a Likert scale as a recent review makes clear (Willits et al., 2016). The following assumptions were tested each time: a) independence of observations, b) normality, c) homogeneity of variance, d) sphericity, and e) equality of covariances. In each case, all these assumptions were met. A 2 X 2 mixed-subjects ANOVA suggested no significant interaction between intervention and class on the average epistemic score specific to Structure of Knowledge dimension (p = 0.056). However, there was a significant main effect of intervention, F(1, 43) = 7.65, p = .008, ηp2 = .15. As such, the average scores pertinent to the Structure of Knowledge belief decreased from pre-reflection (M = 3.56; SE = .09) to post-reflection (M = 3.4; SE = .11). As stated previously, a lower number indicates a more refined view in each dimension. This means, the intervention led to a refining of epistemic beliefs about the Structure of Knowledge. Knowing About Knowing Journal of Effective Teaching in Higher Education, vol. 5, no. 1 8 In other words, students perceived the Structure of Knowledge as more complex after the intervention. Similarly, A 2 X 2 mixed-subjects ANOVA suggested no significant interaction between intervention and class on the average epistemic score specific to Successful Students dimension (p = .08). However, there was a significant main effect of the intervention for the Meaning of Successful Students-SS belief, F(1, 43) = 6.28, p = .016, ηp2 = .13. As such, the average belief scores about the Meaning of Successful Students-SS decreased—refined from pre-reflection (M = 2.68; SE = .12) to post-reflection (M = 2.44; SE = .11). In other words, the belief that successful or smart students are inborn decreased after they participated in the intervention. Finally, A 2 X 2 mixed-subjects ANOVA on the average epistemic score specific to Knowledge Construction and Modification did not yield any significant interaction or main effects (All Fs<1). Discussion The possibility of changing general epistemic beliefs through short-term teaching interventions is tantalizing as it opens the door for the identification of other interventions that could form the basis for an emerging body of evidence-based pedagogical practice. As a reminder, however, this study emphasized general, rather than domain-specific, beliefs, but the trend in differentiated results between the two classes (as seen in marginal interaction effects) hints at the potential interplay between the two levels. It certainly seems possible that students engaged in practices regarding the construction of disciplinary-based knowledge, as would be expected in a Research Methods class, may be predisposed towards greater openness regarding general epistemic beliefs. This hypothesis does not, however, explain why these students registered gains in the Structure of Knowledge-KST and Meaning of Successful Students-SS beliefs and not the Knowledge Construction and Modification-KCM belief. A partial explanation may lie in the nature of the knowledge construction process associated with the field of psychology. Previous experimental studies have emphasized that general epistemic beliefs are not universal and show moderate degrees of sensitivity to disciplinary context (Faber et al., 2016; Muis et al., 2016). A number of studies of differences in general epistemic beliefs along the Biglan (1973) classification of disciplines, for example, found that students majoring in disciplines classified as “hard,” i.e., paradigmatic, often registered as less sophisticated on measures of Knowledge Structure-KST and Objectivity-OT (Hofer, 2000; King et al., 1990; Paulsen & Wells, 1998). One study specifically compared epistemic beliefs in psychology, classified as a “soft” discipline, and the “hard” sciences, noting that psychology students were more likely to recognize complexity (i.e., Knowledge Structure-KST) and hard science students register stronger beliefs that truth is obtainable (i.e., Obtaining Objective Truth-OT) (Hofer, 2000). The patterns revealed in this study (i.e., gains in structure-KST and not in obtaining objectivity-OT) are consistent with these domain-specific findings, but the differences between the two courses of the same domain challenge this hypothesis. Knowing About Knowing Journal of Effective Teaching in Higher Education, vol. 5, no. 1 9 A future study that assesses similar interventions specifically across Research Methods courses in a variety of disciplinary contexts might prove to be illuminating. Not all disciplines have Research Methods courses, and they may be positioned at different stages of the curriculum. As noted in the literature review above, previous studies have emphasized that the epistemic beliefs of students change as they move through their undergraduate coursework. That means that another possible explanation for the results of this study could be found in the pre- and post-test designed throughout the four years of college. Remember, the majority of students who participated in this study were already third- or fourth-year students, and the pre-tests in both classes indicate relatively high degrees of sophistication even prior to the intervention, making it less likely that gains could accrue. By its nature, the study of psychology emphasizes the complexity of learning and, by extension, knowledge, and the signature pedagogy of the field emphasizes the engagement of undergraduate students in the conduct of research or knowledge creation (Peden & Wilson, 2009). Constructs such as growth mindset, as reflected in the successful students-SS subscale, would likely be familiar to many psychology majors. Rather than asking why the students in the Cognitive class did not register similar gains, we might ask ourselves why the students in the Research Methods class had more space for gains to register, a question that could be answered by extending the scope of this study to encompass previous coursework in the major. In other contexts, it might be tempting to interpret these findings through the lens of gender, as epistemic beliefs have been shown to be mildly gender-specific in previous studies (Wang et al., 2015), but the sample used appeared to be predominantly female, with an insufficient number of males to allow for statistical comparisons. This may have turned out for the best, however, as gender constructs have become the subject of intense scholarly scrutiny, with recent advances in neuropsychology serving largely to debunk many prior assumptions regarding cognitive differences between men and women, now referred to as neurosexism (Eliot, 2019). At the same time, however, an increasing number of comparative studies have suggested that epistemic beliefs are sensitive to cultural context, extending from the conditions present in the family home to the broader geopolitical region the student inhabits (Bernholt et al., 2021; Chan & Elliott, 2004). This suggests that future studies should include demographic questions related to cultural background, or perhaps not, as this finding also suggests that the long-standing constructs used in the EBS may need to be critically evaluated for its suitability in measuring epistemic beliefs in cross-cultural contexts. Previous studies utilizing the EBS have often incorporated qualitative components to provide deeper insight, a practice which becomes even more salient for capturing likely increasingly diverse constructs of epistemic beliefs. The discerning reader may be slightly disappointed to reach this point in the discussion and realize that this review of potential explanatory factors has not resulted in a clear alternative hypothesis for the findings of this study. What it reveals instead is that the development of epistemic beliefs in college students is a highly complex process, subject to a large and expanding range of confounding variables, and one that does not necessarily fit neatly into a single construct, Knowing About Knowing Journal of Effective Teaching in Higher Education, vol. 5, no. 1 10 assessment instrument, discipline, or set of students. Rather than despair at this result, this study joins others in providing a glimmer of hope that epistemic beliefs can be influenced through teaching interventions, and that realization will hopefully serve to inspire others to try other interventions that might bring these deep- seated and often implicit beliefs, held by both faculty and students alike, to light. The sheer complexity of studying epistemic beliefs challenges us to find novel means for enhancing our own beliefs about how knowledge is developed, applied, and fostered in higher education—and perhaps beyond. As Bass (2020) himself implores, perhaps we “should balance our well-placed dedication to evidence with professional competency in imagination” (p. 3). Conflicts of Interest The authors declare that there are no conflicts of interest regarding the publication of this article. References Barrett, L. (2014). What counts as (non)cognitive? A comment on Rowe and Healy. Behavioral Ecology, 25(6), 1293–1294. Bass, R. (2020). What’s the problem now? To Improve the Academy: A Journal of Educational Development, 39(1). Baxter Magolda, M. B. (1992). Knowing and reasoning in college: Gender-related patterns in students' intellectual development. Jossey-Bass. Bendixen, L. D. (2002). A process model of epistemic belief change. In B. K. Hofer & P. R. Pintrich (Eds.), Personal epistemology: The psychology of beliefs about knowledge and knowing (pp. 191–208). Lawrence Erlbaum Associates. Bernholt, A., Lindfors, M., & Winberg, M. (2021). Students’ epistemic beliefs in Sweden and Germany and their interrelations with classroom characteristics. Scandinavian Journal of Educational Research, 65(1), 54–70. Biglan, A. (1973). The characteristics of subject matter in different academic areas. Journal of applied Psychology, 57(3), 195–203. Bråten, I., Brandmo, C., & Kammerer, Y. (2019). A validation study of the Internet- specific Epistemic Justification Inventory with Norwegian preservice teachers. Journal of Educational Computing Research, 57(4), 877–900. Buehl, M. M., & Alexander, P. A. (2006). Examining the dual nature of epistemological beliefs. International Journal of Educational Research, 45, 28–42. Caviglia-Harris, J., & Maier, K. (2020). It's not all in their heads: The differing role of cognitive factors and non-cognitive traits in undergraduate Knowing About Knowing Journal of Effective Teaching in Higher Education, vol. 5, no. 1 11 success. Education Economics, 28(3), 245–262. Chan, K.-W., & Elliott, R. G. (2004). Epistemological beliefs across cultures: Critique and analysis of beliefs structure studies. Educational Psychology, 24(2), 123–142. doi:10.1080/0144341032000160100 DeBacker, T. K., Crowson, H. M., Beesley, A. D., Thoma, S. J., & Hestevold, N. L. (2008). The challenge of measuring epistemic beliefs: An analysis of three self-report instruments. Journal of Experimental Education, 76(3), 281–312. doi:10.3200/JEXE.76.3.281-314 Donald, J. G. (1990). University professors’ views of knowledge and validation processes. Journal of Educational Psychology, 82(2), 242–249. Duckworth, A. L., Peterson, C., Matthews, M. D., & Kelly, D. R. (2007). Grit: Perseverance and passion for long-term goals. Journal of Personality and Social Psychology, 92(6), 1087–1101. Dweck, C. S. (2008). Mindset: The new psychology of success. Random House Digital. Eliot, L. (2019). Neurosexism: The myth that men and women have different brains. Nature, 566(7745), 453–455. Faber, C., Vargas, P., & Benson, L. (2016, October). Measuring engineering epistemic beliefs in undergraduate engineering students. 2016 IEEE Frontiers in Education Conference (FIE), 16505079, 1–7. doi:10.1109/FIE.2016.7757461 Ferguson, L.E., Bråten, I., Strømsø, H. I., & Anmarkrud, Ø. (2013). Epistemic beliefs and comprehension in the context of reading multiple documents: Examining the role of conflict. International Journal of Educational Research, 62, 100–114. Hofer, B. K. (2000). Dimensionality and disciplinary differences in personal epistemology. Contemporary Educational Psychology, 25, 378–405. Hofer, B. K., & Pintrich, P. R. (1997). The development of epistemic theories: Beliefs about knowledge and knowing and their relation to learning. Review of Educational Research, 67(1), 88–140. Huber, M. T. (2013). The scholarship of teaching and learning in and across disciplines. Indiana University Press. Jung, E. J. (2011). Literature review: The importance of epistemology in education: The role of epistemology. Department of Educational Psychology and Instructional Technology University of Georgia. Knowing About Knowing Journal of Effective Teaching in Higher Education, vol. 5, no. 1 12 https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.518.8409&rep=r ep1&type=pdf. Karakolidis, A., Pitsia, V., & Emvalotis, A. (2019). The case of high motivation and low achievement in science: What is the role of students’ epistemic beliefs? International Journal of Science Education, 41(11), 1457–1474. Kautz, T., Heckman, J. J., Diris, R., ter Weel, B., & Borghans, L. (2014). Fostering and measuring skills: Improving cognitive and non-cognitive skills to promote lifetime success. OECD Education Working Papers, No. 110, OECD Publishing, Paris. https://doi.org/10.1787/5jxsr7vr78f7-en Khine, M. S., Fraser, B. J., & Afari, E. (2020). Structural relationships between learning environments and students’ non-cognitive outcomes: Secondary analysis of PISA data. Learning Environments Research, 23, 395–412. Kienhues, D., Bromme, R., & Stahl, E. (2008). Changing epistemological beliefs: The unexpected impact of a short-term intervention. British Journal of Educational Psychology, 78(4), 545–565. King, P. M., & Kitchener, K. S. (1994). Developing reflective judgment: Understanding and promoting intellectual growth and critical thinking in adolescents and adults. Jossey-Bass. King, P. M., Wood, P. K., & Mines, R. A. (1990). Critical thinking among college and graduate students. The Review of Higher Education, 13, 167–186. Maddi, S. R., Matthews, M. D., Kelly, D. R., Villarreal, B. J., Gundersen, K. K., & Savino, S. C. (2017). The continuing role of hardiness and grit on performance and retention in West Point cadets. Military Psychology, 29(5), 355–358. Manarin, K., Adams, C., Fendler, R., Marsh, H., Pohl, E., Porath, S., & Thomas, A. (2021). Examining the focus of SoTL literature—teaching and learning? Teaching & Learning Inquiry, 9(1), 349–364. Muis, K.R., Bendixen, L.D., & Haerle, F. (2006). Domain-generality and domain- specificity in personal epistemology research: Philosophical and empirical reflections in the development of a theoretical framework. Educational Psychology Review, 18, 3–54. Muis, K. R., Trevors, G., Duffy, M., Ranellucci, J., & Foy, M. J. (2016). Testing the TIDE: Examining the nature of students’ epistemic beliefs using a multiple methods approach. The Journal of Experimental Education, 84(2), 264–288. Park, J., & Lee, L. (2004). Analysing cognitive or non-cognitive factors involved in the process of physics problem-solving in an everyday context. International Journal of Science Education, 26(13), 1577–1595. Knowing About Knowing Journal of Effective Teaching in Higher Education, vol. 5, no. 1 13 Paulsen, M. B., & Wells, C. T. (1998). Domain differences in the epistemological beliefs of college students. Research in Higher Education, 39(4), 365–384. Peden, B. F., & Wilson VanVoorhis, C. R. (2009). Developing habits of the mind, hand, and heart in psychology undergraduates. In R. Gurung & N. Chick (Eds.), Exploring signature pedagogies: Approaches to teaching disciplinary habits of mind (pp. 161–182). Stylus. Perry, W. G., Jr. (1970). Forms of intellectual and ethical development in the college years: A scheme. Jossey-Bass. Rosman, T., Mayer, A. K., Kerwer, M., & Krampen, G. (2017). The differential development of epistemic beliefs in psychology and computer science students: A four-wave longitudinal study. Learning and Instruction, 49, 166– 177. Schommer-Aikins, M., Duell, O. K., & Hutter, R. (2005). Epistemic beliefs, mathematical problem-solving beliefs, and academic performance of medical school students. The Elementary School Journal, 105(3), 289–304. Schommer, M. (1990). Effects of beliefs about the nature of knowledge on comprehension. Journal of Educational Psychology, 82(3), 498–504. Schommer, M. (1992, October 14–17). Predictors of epistemic beliefs: Comparing adults with only a secondary education to adults with post-secondary education [Conference presentation]. Fourteenth Annual Meeting of the Mid- Western American Educational Research Association, Chicago, IL, United States. Schommer, M. (1993). Comparisons of beliefs about the nature of knowledge and learning among postsecondary students. Research in Higher Education, 34(3), 355–370. Schraw, G. (2001). Current themes and future directions in epistemological research: A commentary. Educational Psychology Review, 13, 451–465. Schraw, G., Bendixen, L. D., & Dunkle, M. E. (2002). Development and validation of the Epistemic Belief Inventory (EBI). In B. K. Hofer & P. R. Pintrich (Eds.), Personal epistemology: The psychology of beliefs about knowledge and knowing (pp. 261–275). Lawrence Erlbaum Associates. She, H. C., Lin, H. S., & Huang, L. Y. (2019). Reflections on and implications of the Programme for International Student Assessment 2015 (PISA 2015) performance of students in Taiwan: The role of epistemic beliefs about science in scientific literacy. Journal of Research in Science Teaching, 56(10), 1309–1340. Knowing About Knowing Journal of Effective Teaching in Higher Education, vol. 5, no. 1 14 Stankov, L., & Lee, J. (2014). Quest for the best non-cognitive predictor of academic achievement. Educational Psychology, 34(1), 1–8. Wang, M.-T., Degol, J., & Ye, F. (2015). Math achievement is important, but task values are critical, too: Examining the intellectual and motivational factors leading to gender disparities in STEM careers. Frontiers in Psychology, 6, 36, doi:10.3389/fpsyg.2015.00036 Wiley, J., Griffin, T. D., Steffens, B., & Britt, M. A. (2020). Epistemic beliefs about the value of integrating information across multiple documents in history. Learning and Instruction, 65, 101266. https://doi.org/10.1016/j.learninstruc.2019.101266 Willits, F. K., Theodori, G. L., & Luloff, A. E. (2016). Another look at Likert scales. Journal of Rural Social Sciences, 31(3), 126–139. Wood, P. K., & Kardash, C. A. (2002). Critical elements in the design and analysis of studies of epistemology. In B. K. Hofer & P. R. Pintrich (Eds.), Personal epistemology: The psychology of beliefs about knowledge and knowing (pp. 231–260). Lawrence Erlbaum Associates. Yeager, D. S., Hanselman, P., Walton, G. M., Murray, J. S., Crosnoe, R., Muller, C., Tipton, E., Schneider, B., Hulleman, C. S., Hinojosa, C. P., Paunesku, D., & Dweck, C. S. (2019). A national experiment reveals where a growth mindset improves achievement. Nature, 573(7774), 364–369. Knowing About Knowing Journal of Effective Teaching in Higher Education, vol. 5, no. 1 15 Appendix A List of Identified Items Under the Corresponding Extracted Dimensions with Factor Loadings for Each Item That Resulted from the Final Principal Component Analysis Variables & Items Factor loadings KCM KST SS Q25. I try my best to combine information across chapters or even across classes. .81 Q20. Today’s facts may be tomorrow’s fiction. .65 Q10. You should evaluate the accuracy of information in textbooks if you are familiar with the topic. .47 Knowledge construction & modification Eigenvalue = 2.50 Q21. I really appreciate instructors who organize their lectures carefully and then stick to their plan. .76 Q30. It is annoying to listen to lecturers who cannot seem to make their mind up as to what they really believe. .73 Q13. If professors would stick more to the facts and do less theorizing, one could get more out of college. .65 Q26.I don’t like movies that don’t have a clear-cut ending. .57 Knowledge structure Eigenvalue = 1.80 Q19. Successful students understand things quickly. .8 2 Q29. Understanding main ideas is easy for good students. .7 2 Q35. The really smart students don’t have to work hard to do well in school. .6 1 Q14. Being a good student generally involves memorizing a lot of facts. .5 2 Successful students Eigenvalue = 1.60 Knowing About Knowing Journal of Effective Teaching in Higher Education, vol. 5, no. 1 16 Appendix B Items From the Epistemic Belief Survey Listed Under the Intended Epistemic Belief Category Epistemic Belief (Latent Factor) Item # Item Speed of Knowledge Acquisition (KSP) 3 If something can be learned, it will be learned immediately. 7 Almost all the information you can understand form a textbook you will get during the first reading. 11 You will just get confused, if you try to integrate new ideas in a textbook with knowledge you already have about a topic. 16 Working on a difficult problem for an extended period of time only pays off for really smart students. 18 Usually, if you are ever going to understand something, it will make sense to you the first time. 24 If I can't understand something quickly, it usually means I will never understand it. 34 Most words have one clear meaning. 38 The information we learn in school is certain and unchanging. Structure of Knowledge (KST) 4 I like information to be presented in a straightforward fashion; I don't like having to read between the lines. 5 It is difficult to learn from textbook unless you start at the beginning and master one section at a time. 12 When I study, I look for specific facts. 13 If professors would stick more to the facts and do less theorizing, one could get more out of college. 21 I really appreciate instructors who organize their lectures carefully and then stick to their plan. 26 I don't like movies that don't have a clear-cut ending. 28 It's waste of time to work on problems that have no possibility of coming out with a clear-cut answer. 30 It is annoying to listen to lecturers who cannot seem to make their mind up as to what they really believe. 31 A good teacher's job is to keep students from wandering from the right track. Knowing About Knowing Journal of Effective Teaching in Higher Education, vol. 5, no. 1 17 33 The best thing about science courses is that most problems have only one right answer. 36 When I learn, I prefer to make things, as simple as possible. Knowledge Construction & Modification (KCM) 2 The only thing that is certain is uncertainty itself. 6 Forming you own ideas is more important than learning what the textbooks say. 8 A really good way to understand a textbook is to reorganize the information according to your own personal scheme. 10 You should evaluate the accuracy of information in textbooks if you are familiar with the topic. 15 Wisdom is not knowing the answers but knowing how to find answers. 20 Today's facts may be tomorrow's fiction. 22 The most important part of scientific work is original thinking. 23 Even advice from experts should be questioned. 25 I try my best to combine information across chapters or even classes. 32 A sentence has little meaning unless you know the situation in which it was spoken. 37 I find it refreshing to think about issues that experts can't agree on. Characteristi cs of Successful Students (SS) 14 Being a good student generally involves memorizing a lot of facts. 17 Some people are born good learners; others are just stuck with a limited ability. 19 Successful students understand things quickly. 29 Understanding main ideas is easy for good students. 35 The really smart students don't have to work hard to do well in school. Attainment of Objective Truth (OT) 1 You can believe most things you read. 9 If scientists try hard enough, they can find the answer to almost every question. 27 Scientists can ultimately get to the truth.