February 2021 C&RL News75 Information literacy is one of the fundamental skills to acquire to be able to navigate today’s complex information ecosystem, succeed as a lifelong learner, and make critical decisions as an active and informed citizen. Today’s information environment—saturated with misinformation, fake news, misleading informa- tion, propaganda, etc.—poses a great challenge to form unbiased views of the world and make sound judgment and decisions. In spring 2020, two management professors from the School of Business at West Chester University shared their interest in developing an information literacy module for their students in an upper-level management class. The course required students to find quality articles on management topics and present the topic to the class. What concerned the faculty was the students’ ability to distinguish high- quality information from low-quality information as they described. They hoped the information literacy module would improve students’ ability to critically evaluate online sources. Considering students’ needs and the faculty’s re- quests, I created a four-step Source Assessment Strategy infographic guide. One problem I encountered at the very beginning was whether to call it an “evaluation strategy” or an “assessment strategy.” Librarians often use evaluation and assessment interchangeably. How- ever, from an educational context, evaluation focuses on making a judgment or determination concerning the quality of performance, work product, or skill, while assessment is not judgmental and focuses on the process of measuring performance, work product, or skill to seek further improvement.1 In this sense, source assessment is more accurate. The four-step source assessment strategy encour- ages students to assess sources through a progressively in-depth process: from looking at the source’s appear- ance, doing investigations, and using lateral reading and critical thinking skills to reflecting on their own biases and interpretations. This strategy intends to expand the CRAAP test with critical thinking and metacognition. It responds to the affective learning and metacognition concepts in ACRL’s Framework for Information Literacy for Higher Education.2 This article will share some of my thoughts behind the development of the infographic guide and hopefully inspire others to further explore the ways to enhance source assessment strategies. The ladder of inference and information evaluation The idea of describing source assessment as mov- ing up a ladder was inspired by the concept of the “ladder of inference.” The ladder of inference, as depicted in the image, was initially developed by the late Chris Argyris, former professor at Har- vard Business School and pioneer in the field of organizational learning. The concept is used to describe a common mental pathway of increas- ing abstraction, which often leads to misguided beliefs.3 Grace Liu Moving up the ladder of source assessment Expanding the CRAAP test with critical thinking and metacognition Grace Liu is assistant professor of business and faculty services librarian at West Chester University’s FHG Library, email: yliu@wcupa.edu © 2021 Grace Liu mailto:yliu%40wcupa.edu?subject= mailto:emilyporterfyke%40gmail.com%20?subject= C&RL News February 2021 76 The ladder of inference reveals our natural mental process when encountering new information. From observing the information available to forming beliefs, we may not be consciously aware that we have taken several leaps of abstrac- tion. We have filtered out some relevant infor- mation, added personal or cultural meanings to interpret the selected information, have made our assumptions based on interpreted meanings, and have drawn conclu- sions based on these as- sumptions. The possibly distorted conclusions may eventually lead us to form misguided beliefs and take wrong actions.4 The ladder of inference reminds us that instead of quickly jumping to conclusions, we need to analyze and test the assumptions, meanings, and selected data and observations that created them. Information source assessment creates a new mental pathway that intervenes in the ladder of infer- ence process and prevents us from forming misguided beliefs. Over the years, scholars have developed many source assessment methods. Sarah Blakeslee, library faculty from California State University-Chico, developed the CRAAP test, which stands for Cur- rency, Relevance, Authority, Accuracy, and Purpose,5 and is widely used by librarians to teach information evaluation. John McManus, former journalist and communication professor, developed the SMELL test, which stands for Source, Motivation, Evidence, Logic, and Left-out.6 More recently, with the proliferation of fake news online, many researchers have developed checklists for spotting fake news. Researchers started to reflect the traditional ways of source evaluation and compare them with fact-checkers’ practices. A 2017 Stanford working paper assessed the web source evaluation skills between students, faculty, and professional fact-checkers and found what made some fact-checkers perform better than others was their habit of “lateral reading” along with their robust knowledge on sources, search engines, and content structures.7 Learning from the Stanford study, Jennifer A. Fielding, in the article “Rethinking CRAAP,” encouraged librarians to go beyond the CRAAP test and bring lateral searching and reading to the library classroom.8 But still, the process of source evaluation is not complete until we con- sider the source content with our critical thinking and reflection. Critical thinking and information evaluation Librarians often con- sider critical thinking and information lit- eracy are closely related and information evalu- ation is at the intersec- tion.9 Critical thinking scholars to some extent also recognized such correlation. As defined by the National Council for Excellence in Critical Thinking, “Critical thinking is the intellectually disciplined process of actively and skill- fully conceptualizing, applying, analyzing, synthesizing, and/or evaluating information gathered from, or generated by, observation, experience, reflection, reasoning, or commu- nication, as a guide to belief and action. In its exemplary form, it is based on universal intellectual values that transcend subject matter divisions: clarity, accuracy, precision, consistency, relevance, sound evidence, good reasons, depth, breadth, and fairness.”10 However, critical thinking and information lit- eracy take different paths in information evaluation. Critical thinking focuses on recognizing basic logic Chris Argyris’s “Ladder of Inference.” Source: Wiki- media Commons. February 2021 C&RL News77 concepts, evaluating arguments and logical fallacies, and examining deductive and inductive reasoning. In- formation literacy focuses on evaluating the authority, motivations, currency, relevancy, accuracy, consistency, and transparency of an information source. We cannot conduct a real information evaluation until we look deeply into the source content and assess the arguments. Recently, we made progress in integrat- ing the fact checker’s practices into the information evaluation process. But that is not enough. We must take a step further to merge information evaluation with more fundamental knowledge in logic and critical thinking. We cannot ignore this fundamental knowledge of information evaluation to reach our goal to educate an informed citizen. It may not be a librarian’s responsibility to teach hardcore knowledge of critical thinking, but helping students bridge information literacy with the knowl- edge they learn in other disciplines, and encouraging students to take a deep dive into the information assessment is what librarians could do and should be doing. Metacognition and information evaluation The ACRL Framework has incorporated the ideas of metaliteracy, with a special focus on metacog- nition or critical self-reflection. It recognizes that “metacognition is crucial to becoming more self- directed in [a] rapidly changing ecosystem.”11 As described in the Framework, “metacognition is an awareness and understanding of one’s own thought processes. It focuses on how people learn and process information, taking into consider- ation people’s awareness of how they learn.”12 Metacognition or self-reflection is a critical step for information evaluation. It leads us to reflect our ob- servations, interpretations, assumptions, conclusions, beliefs, and actions in the information assessment process. It prevents us from jumping quickly through the ladder of inference and going into a vicious cycle of the reflective loop. In the Framework, metacognition is embedded in many knowledge practices and dispositions: Authority Is Constructed and Contextual Knowledge Practices • acknowledge they are developing their own authoritative voices in a particular area and recog- nize the responsibilities this entails. Dispositions • develop and maintain an open mind when encountering varied and sometimes conflicting perspectives, • develop an awareness of the importance of assessing content with a skeptical stance and with a self-awareness of their own biases and worldview, • are conscious that maintaining these atti- tudes and actions requires frequent self-evaluation. Information Creation as a Process Knowledge Practices • develop, in their own creation processes, an understanding that their choices impact the purposes for which the information product will be used and the message it conveys. Information Has Value Knowledge Practices • make informed choices regarding their online actions in full awareness of issues related to privacy and the commodification of personal information. Dispositions • see themselves as contributors to the infor- mation marketplace rather than only consumers of it, • are inclined to examine their own informa- tion privilege. Research as Inquiry Dispositions • maintain an open mind and a critical stance; • value persistence, adaptability, and flex- ibility and recognize that ambiguity can benefit the research process; • demonstrate intellectual humility (i.e., recognize their own intellectual or experiential limitations). Scholarship as Conversation Dispositions • see themselves as contributors to scholarship rather than only consumers of it, • suspend judgment on the value of a particu- C&RL News February 2021 78 lar piece of scholarship until the larger context for the scholarly conversation is better understood, • understand the responsibility that comes with entering the conversation through participa- tory channels. Searching as Strategic Exploration Dispositions • exhibit mental flexibility and creativity; • understand that first attempts at searching do not always produce adequate results; • persist in the face of search challenges, and know when they have enough information to complete the information task. Integrating the concept of metacognition into information assessment and information literacy instruction will build students’ self-awareness, nurture their habits of mind, and help them to develop mental flexibility, creativity, and intellectual humility. A four-step source assessment strategy The four-step source assessment strategy is an ini- tial effort to enrich our current source evaluation practices with critical thinking and metacogni- tion. It brings in some popular concepts related to information source, such as content farm, lat- eral reading, and filter bubble. Below is a summarized structure of the four steps: Step 1. Assess sources by its appearances including URL, format, ads, errors, and references. Step 2. Assess sources by investigation with CRAAP test including currency, relevance, authority, accuracy, and purpose. Step 3. Assess sources by lateral reading and criti- cal thinking, asking questions related to single versus diverse sources, supported versus unsupported claims, logical reasoning versus fallacies, and anecdotes versus research. Step 4. Assess sources by reflection, asking stu- dents to pierce the filter bubbles, examine their own biases, question their own interpretations, keep an open mind, and suspend their judgment. This strategy guides students to go beyond ap- pearance check and the CRAAP test, and use lateral reading skills to explore diverse or competing facts and opinions. It asks students to differentiate supported claims from unsupported claims, and evaluate if the statements are supported by evidence and if the opinions are supported by arguments. It encourages students to distinguish logical reasoning from fallacies. It has students ask if the source is based on personal accounts or serious research, if the research method is valid and reliable, and what the research funder’s perspectives are on the issue. With an effort to integrate metacognition, the strategy guides students to pierce the filter bubble and seek different perspectives. It encourages students to examine their own biases and weigh the reasons from different sides to form unbiased views. It asks students to question their own interpretations, prior knowledge, and assumptions. It motivates students to keep an open mind and seek other possible explana- tions and reasoning. It helps students to recognize the disruptive nature of a simple fact and encourages them to suspend their judgment until they see the big picture. The infographic was designed using Canva. It has two pages and can be printed double-sided on cardstock. The front side lays out the four-step source assessment strategy, and the backside lists common cognitive biases, logical fallacies, and further readings on logical reasoning and critical thinking. I made some changes to the prior business-focused version and, hopefully, it can accommodate different needs. The PDF version of the infographic is available at the ACRL Sandbox13 and can be used and distributed under a Creative Commons License. Summary The ladder of inference inspires us to make the best use of the information sources assessment strategies to help learners resist the natural men- tal tendency of leaps of abstraction. To reach to the core of source assessment, we need to expand the CRAAP test with critical thinking and meta- cognition. I am hoping my four-step sources as- sessment strategy can inspire other librarians to integrate critical thinking and reflection in source evaluation and further explore the ways to en- hance our current source assessment approaches. Notes 1. Susan Starr, “Moving from evaluation to assessment,” Journal of the Medical Library Associa- February 2021 C&RL News79 tion: JMLA vol. 102,4 (2014): 227-9, https://doi. org/10.3163/1536-5050.102.4.001. 2. ACRL, “Framework for Information Lit- eracy for Higher Education,” (2016), www.ala. org/acrl/standards/ilframework. 3. Peter M. Senge, The fifth discipline fieldbook: Strategies and tools for building a learning organiza- tion (New York: Crown Business, 2014). 4. Jonathan Levene, “Solving the problem with problem-solving meetings,” https://www. extension.harvard.edu/professional-development /blog/solving-problem-problem-solving-meetings (accessed April 14, 2020). 5. Sarah Blakeslee, “The CRAAP Test,” LOEX Quarterly 31, no. 3 (2004), https://commons. emich.edu/cgi/viewcontent.cgi?article=1009&context =loexquarterly. 6. John McManus, “Don’t Be Fooled: Use the SMELL Test To Separate Fact from Fiction Online,” (2013), http://mediashift.org/2013/02 /dont-be-fooled-use-the-smell-test-to-separate -fact-from-fiction-online038/. 7. Sam Wineburg and Sarah McGrew, “Lateral Reading: Reading Less and Learning More When Evaluating Digital Information,” Stanford His- tory Education Group-Working Paper (Septem- ber 2017), https://papers.ssrn.com/sol3/papers. cfm?abstract_id=3048994. 8. Jennifer A. Fielding. “Rethinking CRAAP Test: Getting students thinking like fact-checkers in evaluating web sources,” (2019), https://crln.acrl. org/index.php/crlnews/article/view/24195/32005. 9. Helena Hollis, “Information literacy and critical thinking: different concepts, shared con- ceptions,” (2019), http://www.informationr.net /ir/24-4/colis/colis1921.html#author. 10. The National Council for Excellence in Critical Thinking, “Defining Critical Thinking,” (1987), http://www.criticalthinking.org/pages/ defining-critical-thinking/766. 11. ACRL, “Framework for Information Literacy for Higher Education,” (2016), www.ala. org/acrl/standards/ilframework. 12. Jennifer A. Livingston, “Metacognition: An Overview,” online paper, State University of New York at Buffalo, Graduate School of Education (1997). 13. h t t p s : / / s a n d b ox . a c r l . o r g / l i b r a r y - collection/4-step-source-assessment-strategy. certainly applicable to many areas of life, study, and work, and absolutely integral to achieving inclusivity within societies. Notes 1. Steve H. Ching, “Turning a Service Learn- ing Experience into a Model of Student En- gagement: The Lighthouse Heritage Research Connections (LHRC) Project in Hong Kong,” Journal of Academic Librarianship 44, no. 2 (2018): 196-206, https://doi.org/10.1016/j. acalib.2018.02.007. 2. Anthony K. H. Leung, Stephen Davies, and Steve H. Ching, “When new technology joins old documents and east meets west: vir- tually reconstructing the Fisher Island pagoda lighthouse (China),” Virtual Archaeology Review 18, no.9 (2018). 3. Richard W. L. Wong, Anthony K. H. Leung, Brad New, and Steve Ching, “Digital Forensic Investigation of the Xiyu Pagoda Lighthouse: A Library-Led Interdisciplinary Research Project,” International Journal of Art, Culture and Design Technologies 9, no. 2 (2020). 4. Hong Kong Blind Union, “About VI,” https://www.hkbu.org.hk/en/knowledge/statistics /index (accessed August 13, 2020). 5. See https://www.hkmaritimemuseum.org /eng/learn/hkimd2020-lighthouse-memories -diversity-inclusion/chasing-lighthouse-heritage - h o n g - k o n g - l i g h t h o u s e - h e r i t a g e - r e s e a r c h - c o n e c t i o n s - c i t y - u n i ve r s i t y - o f - h o n g - k o n g /135/254/. 6. See https://www.cityu.edu.hk/cityvod /video/play/LIB/lighthouse_memories_13.aspx. 7. See https://www.cityu.edu.hk/cityvod /video/play/LIB/lighthouse_memories_14.aspx. (“Touching history,” continued from page 69) https://doi.org/10.3163/1536-5050.102.4.001 https://doi.org/10.3163/1536-5050.102.4.001 http://www.ala.org/acrl/standards/ilframework http://www.ala.org/acrl/standards/ilframework https://www.extension.harvard.edu/professional-development/blog/solving-problem-problem-solving-meetings https://www.extension.harvard.edu/professional-development/blog/solving-problem-problem-solving-meetings https://www.extension.harvard.edu/professional-development/blog/solving-problem-problem-solving-meetings https://commons.emich.edu/cgi/viewcontent.cgi?article=1009&context=loexquarterly https://commons.emich.edu/cgi/viewcontent.cgi?article=1009&context=loexquarterly https://commons.emich.edu/cgi/viewcontent.cgi?article=1009&context=loexquarterly http://mediashift.org/2013/02 /dont-be-fooled-use-the-smell-test-to-separate-fact-from-fiction-online038/ http://mediashift.org/2013/02 /dont-be-fooled-use-the-smell-test-to-separate-fact-from-fiction-online038/ http://mediashift.org/2013/02 /dont-be-fooled-use-the-smell-test-to-separate-fact-from-fiction-online038/ https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3048994 https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3048994 https://crln.acrl.org/index.php/crlnews/article/view/24195/32005. https://crln.acrl.org/index.php/crlnews/article/view/24195/32005. http://www.informationr.net/ir/24-4/colis/colis1921.html#author http://www.informationr.net/ir/24-4/colis/colis1921.html#author http://www.criticalthinking.org/pages/defining-critical-thinking/766 http://www.criticalthinking.org/pages/defining-critical-thinking/766 http://www.ala.org/acrl/standards/ilframework http://www.ala.org/acrl/standards/ilframework https://sandbox.acrl.org/library-collection/4-step-source-assessment-strategy https://sandbox.acrl.org/library-collection/4-step-source-assessment-strategy https://doi.org/10.1016/j.acalib.2018.02.007 https://doi.org/10.1016/j.acalib.2018.02.007 https://www.hkbu.org.hk/en/knowledge/statistics/index https://www.hkbu.org.hk/en/knowledge/statistics/index https://www.hkmaritimemuseum.org/eng/learn/hkimd2020-lighthouse-memories -diversity-inclusion/chasing-lighthouse-heritage -hong-kong-lighthouse-heritage-research -connections-city-university-of-hong-kong/135/254/ https://www.hkmaritimemuseum.org/eng/learn/hkimd2020-lighthouse-memories -diversity-inclusion/chasing-lighthouse-heritage -hong-kong-lighthouse-heritage-research -connections-city-university-of-hong-kong/135/254/ https://www.hkmaritimemuseum.org/eng/learn/hkimd2020-lighthouse-memories -diversity-inclusion/chasing-lighthouse-heritage -hong-kong-lighthouse-heritage-research -connections-city-university-of-hong-kong/135/254/ https://www.hkmaritimemuseum.org/eng/learn/hkimd2020-lighthouse-memories -diversity-inclusion/chasing-lighthouse-heritage -hong-kong-lighthouse-heritage-research -connections-city-university-of-hong-kong/135/254/ https://www.hkmaritimemuseum.org/eng/learn/hkimd2020-lighthouse-memories -diversity-inclusion/chasing-lighthouse-heritage -hong-kong-lighthouse-heritage-research -connections-city-university-of-hong-kong/135/254/ https://www.hkmaritimemuseum.org/eng/learn/hkimd2020-lighthouse-memories -diversity-inclusion/chasing-lighthouse-heritage -hong-kong-lighthouse-heritage-research -connections-city-university-of-hong-kong/135/254/ https://www.cityu.edu.hk/cityvod/video/play/LIB/lighthouse_memories_13.aspx. https://www.cityu.edu.hk/cityvod/video/play/LIB/lighthouse_memories_13.aspx. https://www.cityu.edu.hk/cityvod/video/play/LIB/lighthouse_memories_14.aspx https://www.cityu.edu.hk/cityvod/video/play/LIB/lighthouse_memories_14.aspx