English Language Teaching and Linguistics Studies ISSN 2640-9836 (Print) ISSN 2640-9844 (Online) Vol. 7, No. 2, 2025 www.scholink.org/ojs/index.php/eltls 111 Original Paper A Study on the Content Validity of Cloze Test in 2023 Zhejiang Senior High School Entrance Examination English Paper Tong Li1 1 Zhejiang Normal University, China Received: February 28, 2025 Accepted: April 08, 2025 Online Published: April 22, 2025 doi:10.22158/eltls.v7n2p111 URL: http://dx.doi.org/10.22158/eltls.v7n2p111 Abstract This study analyzes the content validity of five cloze test passages from the 2023 Zhejiang Senior High School Entrance Examination English papers, focusing on two dimensions: Text Features and Test Validity. Through both textual and quantitative analysis, the results show that the selected passages are diverse in topic, educational in value, and of moderate difficulty. In terms of test level, the items mainly target discourse and sentence levels; in terms of focus factors, meaning is prioritized, with grammar also considered. The distribution of word classes emphasizes content words, indicating a high level of content validity. However, certain limitations remain, including a lack of genre diversity, imbalanced distribution within test levels and focus factors, and limited coverage of word classes. Therefore, future test design should promote greater genre variety, more balanced allocation of test levels and focus factors, and broader word class coverage. In classroom teaching, students should be guided to engage with a wide range of topics and genres, strengthen their foundational language knowledge, and develop logical thinking and discourse awareness. Keywords Zhejiang Senior High School Entrance Examination, cloze test, content validity 1. Introduction Cloze tests are an important item type for assessing students’ integrated language proficiency. They primarily examine students’ reading comprehension at the passage level and their ability to identify contextual constraints. Given their inherent difficulty in discourse analysis and sentence structure interpretation, cloze tests can effectively reflect learners’ overall language competence in real contexts, including their use of vocabulary, grammar, and semantics  (Xia & Luo, 2015). This item type is widely used in major standardized exams in China, such as the Senior High School Entrance Examination, the National College Entrance Examination, the College English Test (CET-4 www.scholink.org/ojs/index.php/eltls English Language Teaching and Linguistics Studies Vol. 7, No. 2, 2025 Published by SCHOLINK INC. 112 and CET-6), and the Test for English Majors (TEM-4 and TEM-8). The quality of cloze test items directly influences the overall validity of the exam. A review of the China National Knowledge Infrastructure (CNKI) database shows that while numerous scholars have explored the validity of cloze tests in exams like the College Entrance Exam, CET-4/6, and TEM-4/8 (Xing, 2022; Huang, 2018; Dong, 2014; Mu, 2011; Chen & Li, 2006), there is relatively limited research on the content validity of cloze tests in the Senior High School Entrance Examination (Zheng & Zhu, 2019; Ding, 2014). Therefore, this study investigates five cloze test items from the 2023 Zhejiang Senior High School Entrance Examination English papers, analyzing them from two dimensions—Text Features and Test Validity—in order to provide references for future test design and English teaching in junior high schools. 2. Theoretical Framework Content validity refers to the extent to which a test accurately represents the content it intends to measure. This study analyzes the content validity of cloze tests based on two key theoretical foundations: the discourse input features from the task characteristics framework proposed by Bachman and Palmer, and the test validity framework for cloze tests developed by Li Xiaoju. According to Bachman and Palmer (1999), discourse input refers to the test materials that candidates must process and respond to. It includes the mode of presentation (such as the language used and text length) and linguistic features (such as topic structure and discourse characteristics). In the analysis of cloze test text features, text difficulty is also considered a critical factor in material selection (Dong, 2014). Li (1997) proposed that the validity of cloze test items can be examined from two dimensions: test levels and focus factors. Test levels refer to the scope of contextual information needed to determine the correct word choice in a blank. Li categorizes these into four levels, from lower to higher: word level, phrase level, sentence level, and discourse level. At the word level, test-takers can determine the correct option based solely on the word itself, without relying on contextual clues. At the phrase level, the correct answer depends on fixed collocational relationships between the target word and adjacent words, typically requiring attention to the word immediately before or after the blank. The sentence level demands an understanding of the entire sentence in which the blank appears; the correct choice can be identified using intra-sentential context. Finally, the discourse level involves the highest degree of contextual integration, requiring test-takers to comprehend information across multiple sentences or the entire passage in order to choose the appropriate word. Higher-level items are more effective in assessing test-takers’ integrated language abilities, aligning with the intended function of cloze tests. Focus factors are classified into grammatical, collocational, and semantic categories. In general, lower-level items tend to emphasize grammatical focus, while higher-level items are more concerned with semantic understanding. www.scholink.org/ojs/index.php/eltls English Language Teaching and Linguistics Studies Vol. 7, No. 2, 2025 Published by SCHOLINK INC. 113 In addition, part of speech distribution also affects test validity (Huang, 2018), and should be taken into account when designing test items. The major English parts of speech include verbs, nouns, numerals, adjectives, adverbs, pronouns, articles, prepositions, conjunctions, and interjections. Drawing on these theoretical foundations, this study analyzes five cloze tests from the 2023 Zhejiang Senior High School Entrance Examination in terms of Text Features and Test Validity. The analysis of Text Features includes genre, topic, and text readability, while Test Validity includes test levels, focus factors, and part of speech distribution. The analytical framework is shown in Table 1. Table 1. Analysis Framework of Content Validity in Cloze Test Item Description Text Features Genre Narrative, Expository, Argumentative, Practical Writing Topic Self and Individuals, Society and Individuals, Nature and Individuals Text Readability Flesch Readability Test Validity Test Level Word Level, Phrase Level, Sentence Level, Discourse Level Focus Factors Grammar, Collocations, Meaning Part of Speech Distribution Verbs, Nouns, Numerals, Adjectives, Adverbs, Pronouns, Articles, Prepositions, Conjunctions, Interjections 3. Method 3.1 Research Subjects This study randomly selected five sets of English test papers from the 2023 Zhejiang Senior High School Entrance Examination. The five cloze passages and a total of 75 items extracted from these papers were used as the research corpus to analyze the content validity of the cloze test items in the English examination. (See Table 2) Table 2. Overview of Cloze Test in Five 2023 Zhejiang Papers PaperA Paper B Paper C Paper D Paper E Number of Passages 1 1 1 1 1 Number of 15 15 15 15 15 www.scholink.org/ojs/index.php/eltls English Language Teaching and Linguistics Studies Vol. 7, No. 2, 2025 Published by SCHOLINK INC. 114 Items Total Score 15 15 15 15 15 3.2 Research Methods Based on the analytical framework presented in Table 1, this study adopts a combination of textual analysis and quantitative analysis to evaluate the content validity of the cloze test items from two dimensions: text features and test validity. The analysis of genre and topic under text features, as well as test levels, focus factors, and part of speech distribution under test validity, is conducted with reference to the English Curriculum Standards for Compulsory Education (2022 Edition) (hereafter referred to as the Curriculum Standards). The text readability is assessed using the widely adopted Flesch readability formula (Dong, 2014), calculated via Microsoft Office Word 2019, and interpreted in light of the Flesch readability scale. To enhance the reliability of the study, the genre, topic, test level, focus factor, and part of speech distribution were independently coded twice. For any discrepancies, the researcher consulted relevant theoretical literature and previous studies to determine the appropriate classification. 4. Result 4.1 Text Features Table 3. Overview of Genre, Topic, and Text Readability in Cloze Test Passage in Five 2023 Zhejiang Papers PaperA Paper B Paper C Paper D Paper E Genre Narrative Narrative Narrative Narrative Narrative Topic Self and Individuals Nature and Individuals Society and Individuals Society and Individuals Society and Individuals Readability Score 83.6 73.9 92.5 89.1 83.3 As shown in Table 3, the genres of all five cloze test passages from the 2023 English exam papers in Zhejiang Province are narrative. These texts are mainly characterized by narration and description, focusing on storytelling and depicting people or events. The topics cover three major dimensions: self and individuals, society and individuals, and nature and individuals. Among them, one passage focuses on the self and individuals (20% of the total), three on society and individuals (60%), and one on nature and individuals (20%), indicating a greater emphasis on the theme of “society and individuals”. The readability scores of the cloze test texts range between 70 and 90. According to the Flesch www.scholink.org/ojs/index.php/eltls English Language Teaching and Linguistics Studies Vol. 7, No. 2, 2025 Published by SCHOLINK INC. 115 readability index, a higher score indicates greater ease of reading, while a lower score reflects higher difficulty. Based on the Flesch readability scale (Liu, 2015) (see Table 4), all texts fall within the range of “Fairly Easy” to “Very Easy,” corresponding to the reading levels of American students in Grades 4 to 6. This suggests that the texts are generally appropriate for the cognitive and language development level of junior secondary graduates. Table 4. Flesch Readability Scale Readability Score 0–30 31–50 51–60 61–70 71–80 81–90 91–100 Text Difficulty Very Difficult Difficult Fairly Difficult Standard Fairly Easy Easy Very Easy Intended Grade Level College Senior High Senior High Grades 7–8 Grade 6 Grade 5 Grade 4 4.2 Test Validity 4.2.1 Test Levels Table 5. Quantitative Statistics of Test Levels in Cloze Test Items in Five 2023 Zhejiang Papers Discourse Level (%) Sentence Level (%) Phrase Level (%) Word Level (%) Paper A 11 (73.3%) 4 (26.7%) 0 (0%) 0 (0%) Paper B 12 (80.0%) 3 (20.0%) 0 (0%) 0 (0%) Paper C 12 (80.0%) 3 (20.0%) 0 (0%) 0 (0%) Paper D 14 (93.3%) 1 (6.7%) 0 (0%) 0 (0%) Paper E 11 (73.3%) 4 (26.7%) 0 (0%) 0 (0%) Total (%) 60 (80.0%) 15 (20.0%) 0 (0%) 0 (0%) As shown in the total data of Table 5, the cloze test items in the 2023 English exam papers in Zhejiang Province mainly focus on the discourse level (80%), followed by the sentence level (20%), with no items targeting the phrase or word levels. Overall, these papers emphasize higher-level comprehension tasks, aiming to assess students’ abilities to understand and process information at the sentence and discourse levels, thereby testing their integrated language use. 4.2.2 Focus Factors www.scholink.org/ojs/index.php/eltls English Language Teaching and Linguistics Studies Vol. 7, No. 2, 2025 Published by SCHOLINK INC. 116 Table 6. Quantitative Statistics of Focus Factors in Cloze Test Items in Five 2023 Zhejiang Papers Meaning (%) Collocation (%) Grammar (%) Paper A 12 (80.0%) 0 (0%) 3 (20.0%) Paper B 15 (100%) 0 (0%) 0 (0%) Paper C 14 (93.3%) 0 (0%) 1 (6.7%) Paper D 13 (86.7%) 0 (0%) 2 (13.3%) Paper E 14 (93.3%) 0 (0%) 1 (6.7%) Total (%) 68 (90.7%) 0 (0%) 7 (9.3%) As shown in Table 6, 90.7% of the cloze test items across the five papers focus on meaning, while only 9.3% target grammar points, and none focus on collocations. Notably, Paper B includes only meaning-based items, accounting for 100%. In general, the test items primarily emphasize semantic understanding, with grammar playing a supplementary role, aiming to assess students’ ability to construct meaning from texts. 4.2.3 Distribution of Part of Speech Table 7. Quantitative Statistics of Part of Speech in Cloze Test Items in Five 2023 Zhejiang Papers Verb Noun Adj. Adv. Pron. Num. Prep. Conj. Paper A 4 3 3 3 1 0 0 1 Paper B 7 4 2 1 0 0 1 0 Paper C 4 3 4 1 1 0 1 1 Paper D 5 2 3 2 1 1 0 1 Paper E 5 4 2 2 1 0 0 1 Total (%) 25 (33.3%) 16 (21.3%) 14 (18.7%) 9 (12.0%) 4 (5.3%) 1 (1.3%) 2 (2.7%) 4 (5.3%) According to Table 7, verbs are the most frequently tested part of speech in the cloze test items, followed by nouns, adjectives, and adverbs. Pronouns, numerals, prepositions, and conjunctions are tested less frequently, with some papers not covering these parts of speech at all. Overall, the test items primarily focus on content words that carry substantial meaning and convey the main ideas of the texts. 5. Discussion From the perspectives of text features and test validity, the cloze tests in the five sets of 2023 Zhejiang English examination papers demonstrate relatively high content validity. However, there is still room for improvement in aspects such as genre, test level, focus factors, and part-of-speech distribution. www.scholink.org/ojs/index.php/eltls English Language Teaching and Linguistics Studies Vol. 7, No. 2, 2025 Published by SCHOLINK INC. 117 5.1 Text Features 5.1.1 Genre All five cloze test passages in the 2023 Zhejiang provincial English exam papers are narratives, which does not align with the principle of genre diversity in test design. Common text genres include narrative, expository, argumentative, and practical writing. The exclusive use of narrative texts in all five cloze tests may be attributed to two reasons: first, narrative texts often reflect real-life experiences, resonate with students’ daily lives, and are more likely to arouse their interest and facilitate engagement with the reading material; second, narratives tend to be coherent in structure, making them effective for assessing students’ discourse-level reading comprehension. However, “different genres serve different social functions and possess distinct structural and linguistic features” (Liu, 2019). For example, narrative texts recount events; expository texts aim to explain phenomena or concepts, and their inclusion can broaden students’ knowledge and perspectives; argumentative texts rely on reasoning and evidence to persuade readers of a particular viewpoint, and their inclusion helps foster students’ critical thinking and logical reasoning skills (Dong, 2014). Incorporating expository, argumentative, and other genres into cloze test materials would better reflect real-world reading contexts, in which individuals engage with various types of texts. It would also bring about positive washback effects on junior high school English instruction, discouraging the overemphasis on narratives and promoting more balanced genre-based teaching that supports the development of students’ overall language proficiency. 5.1.2 Topic The cloze test passages in the 2023 English exams from five cities in Zhejiang Province cover a wide range of themes and strongly reflect educational values. The five texts focus on themes such as personal development, social interaction, and the natural environment, which closely align with the thematic clusters outlined in China’s Curriculum Standards, including “Self-awareness, self-management, and self-improvement” under the theme of Self and Individuals; “Interpersonal relationships and communication,” “Cooperation and teamwork” under the theme of Society and Individuals; and “Love and respect for nature, living in harmony with nature” under the theme of Nature and Individuals. This demonstrates that the selection of materials for cloze tests in these five cities adheres closely to curriculum standards while reflecting broader social and historical contexts. Moreover, the chosen themes carry strong educational significance for students. For example, the passage from Paper A tells the story of a protagonist who overcomes prejudice against his sister with mental disorder after hearing her sing beautifully. This promotes the value of viewing others objectively and appreciating their strengths. The article conveys profound meaning and highlights the importance of moral education. 5.1.3 Text Readability The overall difficulty level of the cloze test texts in the 2023 Zhejiang Province English exams is www.scholink.org/ojs/index.php/eltls English Language Teaching and Linguistics Studies Vol. 7, No. 2, 2025 Published by SCHOLINK INC. 118 appropriately set. Text difficulty directly influences the challenge level of cloze tests. If a passage is too easy, most students can answer correctly, reducing the discriminatory power of the test. Conversely, if a passage is too difficult, it may lead to widespread failure, also diminishing the test’s ability to differentiate student ability (Yang et al., 2022). Therefore, for lower secondary exams, the text difficulty should be matched with the expected proficiency level of students. According to Dong Manxia’s research (2014), the readability scores of national college entrance exam cloze test passages from 2004 to 2013 mostly fluctuated around 70, with a minimum of 63.4 and a maximum of 80.6, indicating an appropriate level of difficulty. Based on this, the readability of the 2023 Zhejiang cloze passages, which ranges from 73.9 to 92.5, appears to be reasonable. This can be attributed to two factors. First, although junior secondary students generally have a lower proficiency level than senior secondary students, the overall English competence of Chinese students has improved in recent years. Thus, setting the difficulty about one readability tier below the college entrance exam is appropriate. Second, the Paper B, which comes from a region with relatively more advanced educational development, is slightly more difficult than those from the other cities, which aligns with the view of Liu Chao (2019) that in more developed regions, competition is fiercer and school selection criteria are more rigorous, resulting in higher exam difficulty. In general, the difficulty level of these five cloze tests is well aligned with the current English proficiency of junior secondary students and takes into account regional differences in talent selection, making the design overall appropriate. 5.2 Test Validity 5.2.1 Test Level The 2023 cloze tests from the five sets of Zhejiang Province English exams generally feature a well-balanced distribution of test levels. However, items at the word and phrase levels should also be included in appropriate proportions to ensure a reasonable range of difficulty. All five sets of cloze tests focus primarily on sentence-level and discourse-level comprehension, with discourse-level items accounting for up to 80%. According to the English Curriculum Standards, students in Grades 7–9 should be able to “identify the development and changes of events in a discourse, recognize the relevance of information, and grasp the overall meaning of a passage,” as well as “recognize cohesive devices and determine the logical relationships between sentences and paragraphs.” Unlike items that can be answered by understanding individual words or phrases, all blanks in the five tests require students to rely on contextual or even global discourse understanding. These items emphasize students’ ability to grasp the overall content and logical flow of the text, effectively assessing whether they have achieved the expected learning outcomes outlined in the curriculum standards. Nevertheless, items at lower levels (i.e., word or phrase level) should not be overlooked. The higher the www.scholink.org/ojs/index.php/eltls English Language Teaching and Linguistics Studies Vol. 7, No. 2, 2025 Published by SCHOLINK INC. 119 level of language focus, the greater the information load and cognitive demand, and hence the higher the difficulty (Dong, 2014). A reasonable mix of item levels can provide a more comprehensive assessment of students’ integrated language competence while enhancing the tests’ discriminatory power. 5.2.2 Focus Factors The 2023 cloze tests in Zhejiang’s five papers place significant emphasis on semantic understanding, while grammatical structures and collocational knowledge are underrepresented or even neglected. All five tests prioritize meaning-based processing, requiring students to choose the most contextually appropriate word based on a solid understanding of the passage. Grammatical aspects such as subordinate clauses are minimally tested. However, there is a lack of attention to fixed or idiomatic collocations, which weakens the function of cloze tests in assessing students’ overall language use. 5.2.3 Distribution of Part of Speech The 2023 cloze tests from the five sets of Zhejiang English exams overall feature a wide range of lexical categories, but some papers fail to assess function words such as prepositions or conjunctions, which should be further emphasized. Across the five sets, content words—verbs, nouns, adjectives, adverbs, pronouns, and numerals—account for 92% of the tested items, while function words—prepositions and conjunctions—account for only 8%. The focus on content words stems from their concrete meanings, which are useful for evaluating reading comprehension. However, function words are essential for constructing grammatical relationships and understanding textual cohesion. For example, conjunctions are critical for assessing students’ ability to interpret logical connections within a text. The Paper B does not include any conjunction-related items, thereby failing to exploit the cloze test’s potential for testing logical cohesion. Moreover, scholars have noted that “function words are generally more difficult to learn and master than content words due to their abstract meanings and higher demands for deep understanding” (Dong, 2014). Therefore, appropriately incorporating function words can serve as an effective means of adjusting test difficulty. 5.3 Concluding Implications for Practice 5.3.1 Implications for Cloze Test Design Firstly, the text genres used in cloze tests should be diversified rather than limited to narrative passages. Expository and argumentative texts, such as those presenting debates or explanations, can also be adopted to enhance the authenticity and communicative relevance of the test. Secondly, while discourse-level and sentence-level items should remain the primary focus, proper attention should also be given to phrase-level and word-level items. The design of higher-level items aims to assess students’ ability to comprehend the overall meaning of the text, contributing to higher content validity. However, incorporating lower-level items ensures a more balanced level of difficulty and provides a more comprehensive assessment of students’ integrated language competence. Thirdly, meaning should remain the core focus of test items, but greater attention should be paid to www.scholink.org/ojs/index.php/eltls English Language Teaching and Linguistics Studies Vol. 7, No. 2, 2025 Published by SCHOLINK INC. 120 grammar and collocations. A balanced distribution of these focus factors will enhance the construct validity and overall quality of test design. Finally, while emphasizing content words, cloze tests should also incorporate function words such as conjunctions and prepositions. Content words primarily assess semantic understanding and contextual comprehension, while function words play a critical role in grammatical cohesion and logical coherence. Maintaining the principle of “content words as primary, function words as supplementary” contributes to a more complete evaluation of students’ overall language proficiency. 5.3.2 Implications for Junior High School English Teaching Firstly, students should be guided to engage with a wide range of topics and text genres. The inclusion of a wide range of topics and various text types in junior high school English cloze tests aligns with the developmental trend of enhancing the content validity of exam. Therefore, teachers should guide students to engage with texts covering diverse topics, thereby exploring the underlying ideas, enriching their knowledge base, and fostering sound moral and ethical character. In addition, teachers could incorporate a variety of text genres into instruction, to help students understand the characteristics of different genres and enhances their ability to receive and process diverse types of information. Secondly, a solid foundation in basic English language knowledge must be built. As cloze tests assess students’ comprehensive abilities based on their language knowledge, skills such as listening, speaking, reading, and writing—as well as vocabulary and grammar—are fundamental. Teachers should follow the Curriculum Standards by helping students “learn to interpret and express meaning using vocabulary in context” and “understand that grammatical choices depend on specific contexts based on the comprehension of the main idea.” This lays the groundwork for integrated language use. Finally, students should be trained to develop logical thinking and discourse awareness. Since the majority of test items in cloze tests are set at the discourse level, they primarily examine students’ understanding of logical relationships and discourse meaning. Therefore, teaching should emphasize skills such as analyzing referential links, identifying cohesive devices, inferring logical relations between sentences and paragraphs, and summarizing main ideas of paragraphs. These abilities help students better understand the structure and main message of a text. References Bachman, L. F., & Palmer, A. S. (1999). Language Testing in Practice. Shanghai: Shanghai Foreign Language Education Press. Chen, X., & Li, S. (2006). A Study of the Construct Validity of TEM-4 Cloze Test: A Process Approach. Modern Foreign Languages, (01), 71-77+110. Ding, R. (2014). Validity Study on the Cloze Test of Shanxi Senior High School English Entrance Exam from 2009 to 2013 (Master’s thesis). Northwest University. Dong, M. (2014). A Study on the Text Features and Test Validity of the Cloze Test in the NMET English Papers. Educational Measurement and Evaluation, (08), 57–64+47. www.scholink.org/ojs/index.php/eltls English Language Teaching and Linguistics Studies Vol. 7, No. 2, 2025 Published by SCHOLINK INC. 121 Huang, J. (2018). Study on the Content Validity of Cloze in 2018 NMET. Educational Measurement and Evaluation, (08), 18-25. Li, X. (1997). The Science and Art of Language Testing. Changsha: Hunan Education Press. Liu, C. (2020). Research on Content Validity of Rational Cloze of English National Test Paper in 2019 National Matriculation Entrance Test. Educational Measurement and Evaluation, (03), 33-40. Liu, X. (2015). A Review of Readability Studies. Journal of Hubei University (Philosophy and Social Science Edition), 42(03), 141-146. Ministry of Education of the People’s Republic of China. (2022). English Curriculum Standards for Compulsory Education (2022 Edition). Beijing: Beijing Normal University Press. Mu, H. (2011). Corpus-Based Research on Content Validity of CET-4 Cloze. Technology Enhanced Foreign Language Education, (04), 66-70. Xia, C., & Luo, X. (2015) An Analysis of the Content Validity of English Cloze Tests in the Zhejiang Gaokao Papers (2010-2014). Educational Measurement and Evaluation, (03), 50-54. Xing, Y. (2022). A Study on the Content Validity of the Cloze Test of NMET Ⅰ (Master’s thesis). Jilin Normal University. Yang, Z., Chen, Y., & Zhu, S. (2022). English Writing Item Discrimination Analysis and Its Implications for the Chinese College Entrance Examination (CCEE): Using the CCEE Data from One Province as Examples. Educational Measurement and Evaluation, (05), 3-10. Zheng, L., & Zhu, Y. (2019). A Corpus-Based Study on Content Validity of Cloze in Senior High School Entrance Examination: A Case Study of Cloze Test in Ningbo from 2012 to 2019. Educational Measurement and Evaluation, (10), 49-55. A Study on the Content Validity of Cloze Test in 2 Tong Li1 1 Zhejiang Normal University, China Abstract This study analyzes the content validity of five c Keywords References