42 © 2023 by the author; licensee Asian Online Journal Publishing Group Asian Journal of Education and Training Vol. 9, No. 2, 42-47, 2023 ISSN(E) 2519-5387 DOI: 10.20448/edu.v9i2.4657 © 2023 by the author; licensee Asian Online Journal Publishing Group Investigation of middle school teachers' open-ended question writing skills Hatice Kumandas Ozturk Artvin Coruh University, Faculty of Education, Artvin, Turkey. Email: haticekumandas@yahoo.com Abstract In this study, the skills of middle school teachers to write open-ended questions that measure higher-order mental skills were examined. For this purpose, 18 middle school teachers who worked in Artvin, Turkey and volunteered to participate in the study were asked to write questions. Teachers wrote open-ended questions for a total of 4 courses (Sciences, Mathematics, Turkish, and Social Studies) in accordance with the specified outcomes. In the study designed in the document review pattern of qualitative research methods, questions were analyzed using an evaluation form. As a result of the evaluations, it was observed that teachers were more concerned with the content dimension of the outcomes while writing questions but disregarded the cognitive level of the outcome. It was also found that teachers were inadequate in terms of language and expression in all branches, showed a tendency to copy questions from a source, or could not move away from the question writing style they adopted. However, it was determined that there were especially expression errors, subject-verb agreement problems, and punctuation marks were not used correctly in the questions written. Keywords: Higher order thinking skills, In-class measurement and evaluation, Open-ended question, Assessment instruments, Question type appropriateness. Citation | Ozturk, H. K. (2023). Investigation of middle school teachers’ open-ended question writing skills. Asian Journal of Education and Training, 9(2), 42–47. 10.20448/edu.v9i2.4657 History: Received: 13 March 2023 Revised: 27 April 2023 Accepted: 4 May 2023 Published: 11 May 2023 Licensed: This work is licensed under a Creative Commons Attribution 4.0 License Publisher: Asian Online Journal Publishing Group Funding: This study received no specific financial support. Competing Interests: The author declares that there are no conflicts of interests regarding the publication of this paper. Transparency: The author confirms that the manuscript is an honest, accurate, and transparent account of the study; that no vital features of the study have been omitted; and that any discrepancies from the study as planned have been explained. Ethical: This study followed all ethical practices during writing. Contents 1. Introduction ...................................................................................................................................................................................... 43 2. Method ............................................................................................................................................................................................... 44 3. Results/Findings .............................................................................................................................................................................. 44 4. Conclusion and Discussion ............................................................................................................................................................. 46 References .............................................................................................................................................................................................. 46 mailto:haticekumandas@yahoo.com https://creativecommons.org/licenses/by/4.0/ https://creativecommons.org/licenses/by/4.0/ https://www.doi.org/10.20448/edu.v9i2.4657 https://orcid.org/0000-0003-3636-3690 Asian Journal of Education and Training, 2023, 9(2): 42-47 43 © 2023 by the author; licensee Asian Online Journal Publishing Group Contribution of this paper to the literature In Turkey, there are not many studies examining teachers' ability to write open-ended question and higher-order thinking skills. In addition, there is no study related to the review titles discussed in this study. In this respect, it is thought that the study will contribute to the field. 1. Introduction In the current information era, societies have switched from a teacher-centered to a student-centered approach to education. In this regard, many countries have updated their curriculum and embraced methodologies like the constructivist approach, which places a strong emphasis on learning and the student. In line with the updates and innovations, it is now crucial to link learning outcomes with life skills in the curriculum and to emphasize higher order thinking skills such as reading comprehension, research, questioning, critical thinking and problem solving. As a result of globalization, education programs in Turkey have also been revised to adapt to the changing world and raise qualified individuals. These changes in the curricula have led to changes in measurement and evaluation activities, which are the controlling elements of education. The changes reflected in the measurement and evaluation process have gained importance in points such as using various evaluation approaches (written exams, portfolios, performance evaluations with open-ended questions, self- assessment, etc.) that focus more on the process rather than the outcome and vary according to the purpose. In this context, a prominent feature of the renewed primary education programs when evaluated in these two dimensions is the development and measurement of higher-order thinking skills by establishing a connection with daily life (Demirtasli, 2010). Higher-order thinking skills, which is one of the concepts emphasized in the association of questions with life skills is defined as the individual's use of multiple skills in relation to their individual abilities (Kutlu, Doğan, & Karakaya, 2010). Higher-order thinking skills are defined in the new taxonomy, conducted by Bloom and updated under the leadership of Anderson and Krathwohl (2001) as behaviors at the analysis, evaluation, and creation stages. Moreover, Miri, David, and Uri (2007) characterize questions that examine higher-order thinking skills as those involving real-life situations, discussion, problem-solving, critical thinking, and content based on experiences (Zoller et al., 2000). In relation to this concept, one of the emphasized points in the new program is the examination of cognitive levels of questions used in the measurement and evaluation process with appropriate question types. In the literature, it is emphasized that the use of open-ended questions in measuring higher-order cognitive skills and using information (Badger & Thomas, 1991) is more effective than other question types. Open-ended questions are defined as a question style where students freely answer questions conveyed in a question mode or command mode according to their cognitive levels and writing skills (Erkus, 2006). Brookhart (2015) states that open-ended questions help individuals apply higher-order cognitive skills and provide an environment for expressing their thoughts freely. Some researchers (Friborg & Rosenvinge, 2013; Hadenfeldt, Bernholt, Liu, Neumann, & Parchmann, 2013) emphasize that one of the most effective assessment and evaluation methods for determining students' mathematical literacy and verbal expression skills, in situations where the desired product needs to be presented by the student, like as problem solving as well as measuring qualities such as creative thinking power, the ability to organize information, and evaluating the internal and external consistency of thoughts is through open-ended questions. It has been observed that students who were examined by using open-ended questions, have higher awareness about their own learning situations (Cooney, Sanchez, Leatham, & Mewborn, 2004) and that many real-world problems can be measured with this question type (Boaler, 1998). According to O'Neil Jr and Brown (1998), the use of an open-ended question format requires the use of more cognitive strategies and better reveals the learning process. It is stated that open-ended questions are more advantageous in measuring higher-order thinking skills than other question types (Bahar, Nartgun, Durmus, & Bıçak, 2012), and they allow individuals to present results using their reasoning power and creative thinking skills in solving problems with multiple solutions but lacking information (Akay, Soybaş, & Argün, 2006). Open-ended questions are an effective type of evaluation for measuring many higher-order cognitive processes, such as critical thinking, problem-solving, and generating new ideas (Wright, 2010). Additionally, open-ended questions reveal different aspects of the feature or achievement to be measured and allow for the development of individuals' writing, speaking, and reading comprehension skills (Hancock, 1995). There are several points to consider when writing open-ended questions (Cooney et al., 2004). These include: • Being open to interpretation. • Providing context. • Allowing for multiple solution paths, not just a single one. • Being clear and understandable. • Using a graded scoring key during the evaluation process. There are some disadvantages to using open-ended questions in examining school learning alongside their numerous advantages. One of these is that the implementation and scoring of open-ended questions take more time compared to other question types (Dogan, 2013). Additionally, if this question type is not well-prepared, it can fall short in ensuring content validity (Ozcelik, 2011; Romagnano, 2001). Furthermore, there are disadvantages in an exam composed of open-ended questions such as effective time usage, limited number of questions, and scoring objectivity (Atılgan, 2004; Kan, 2013). Given the advantages and disadvantages highlighted above, field experts stress that the open-ended questions used to measure higher order thinking skills in educational institutions should be written correctly. Therefore, this study aimed to examine the open-ended question writing skills of secondary school teachers working in Artvin province used to measure high-level mental skills. To this end, the authors selected previously defined learning outcomes in the curricula for the subjects taught by the teachers who volunteered for the study and asked them to Asian Journal of Education and Training, 2023, 9(2): 42-47 44 © 2023 by the author; licensee Asian Online Journal Publishing Group write appropriate open-ended questions. They then analyzed the questions under four main headings in line with the principles of measurement and evaluation, and sought answers to the following questions: Questions written by teachers; in the perspective of, • Language and expression. • Compliance with the content and cognitive level of the skill to be acquired. • Relevance to question type. • Originality. Are written in accordance with the principles of measurement and evaluation? 2. Method 2.1. Research Model In this study, middle school teachers were asked to write open-ended questions from a total of 4 courses, namely Sciences, Mathematics, Turkish and Social Studies in accordance with the learning outcomes and the questions were examined by field, language and measurement and evaluation experts in line with measurement and evaluation rules, grammar rules and open-ended question writing principles. This research was designed in the document analysis pattern, which is one of the qualitative research methods. Yildirim and Simsek (2011) defined document analysis as the examination of visual materials containing information about the phenomena or events to be investigated. In the study designed in qualitative research design, the questions were analyzed by preparing an evaluation form. The results were summarized with graphs. 2.2. Working Group For this purpose, 20 questions written by 18 teachers who voluntarily participated in the question writing study were analyzed. Table 1 shows the distribution of the teachers who wrote questions and the questions they wrote by courses. Table 1. Distribution of teachers and the questions they wrote. Courses Number of teachers Number of questions Turkish 6 6 Science 5 5 Mathematics 5 6 Social studies 2 3 Total 18 20 2.3. Data and Analysis The data of the study were analyzed using the review form developed by the researcher for the evaluation of teachers' questions. In this form, there are themes of language and expression, conformity to the learning outcome, conformity to the question type and originality. The items under these themes were evaluated by marking "yes" (1) and "no" (0). In the process of analyzing the data. Each teacher was given the achievements in his/her own field, which are included in the curriculum and allow the measurement of high-level cognitive processes. They were asked to write questions appropriate to these achievements. Some of the achievements given to the teachers are as follows: • Analyzes and distinguishes types of texts. • Generates different solutions to problems. • It makes inferences about the characteristics of its states. • Discusses the causes and consequences of migration in societies. • Compare the densities of liquids by experimenting. • Relates the cause of the refraction of light to the change of environment. • It poses problems that require four operations. • Recognizes and associates area measurement units. Each teacher wrote one or two questions from his/her own field (Science, Mathematics, Turkish and Social Studies) in accordance with the learning outcomes. There were 20 questions in total. The questions were analyzed by two measurement and evaluation experts. In analyzing the data, common points were identified in the evaluations made by the experts and these points were compared with graphs according to the courses. The harmony between the evaluations of the experts was determined using the Kappa statistics (α). The results are given in Table 2. Table 2. Evaluation process and kappa statistics results. Assessment themes Kappa (α) statistics Language and expression 0.93* Eligibility for benefits 0.95* Question type eligibility 0.97* Originality 0.92* Note: *p<0.05. As a result of the Kappa statistics in Table 2, all themes showed harmony ranging from 0.92-0.97 (p<0.05). 3. Results/Findings Figure 1 shows the results of the analysis of the questions written by the teachers in terms of language and comprehension (Spelling error, poor punctuation, subject-verb mismatch, unnecessary word use, and lack of clarity). Asian Journal of Education and Training, 2023, 9(2): 42-47 45 © 2023 by the author; licensee Asian Online Journal Publishing Group Figure 1. Analysis of questions in terms of language and comprehension. It can be seen that the vast majority of the questions written by the teachers contain narrative errors. In addition, the experts found almost all of the questions written for science to be unclear and incomprehensible. Most of the teachers paid attention to punctuation. However, most of the science and mathematics teachers made spelling mistakes in their questions. Figure 2 shows the results of the analysis of the teachers' written questions in terms of being appropriate for the required learning outcome. Figure 2. Analysis of questions in terms of learning outcome appropriateness. While almost all of the questions written by the teachers are appropriate for the learning outcome in terms of content, the number of questions appropriate for the required outcome at a cognitive level is low. Figure 3 shows the results of the analysis of the teachers' questions in terms of being appropriate for the type of question. Figure 3. Analysis of questions in terms of being appropriate for the type of question. The vast majority of the questions written by science and mathematics teachers were not found appropriate for the level. No branch showed any problems in terms of scientific accuracy. Half of the questions written for Turkish and Social Studies used unnecessary visuals. Figure 4 shows the results of the analysis of the teachers' written questions in terms of originality. Asian Journal of Education and Training, 2023, 9(2): 42-47 46 © 2023 by the author; licensee Asian Online Journal Publishing Group Figure 4. Analysis of the teachers' written questions in terms of originality. In all branches, it was seen that teachers wrote some of the questions inspired by another question. More teachers in the science branch than in the other branches were found to have copied the questions they wrote from other sources verbatim. Moreover, it was observed that most of the teachers tended to write the questions in the same way. 4. Conclusion and Discussion As a result of the analysis, it was seen that the teachers were only interested in the content dimension of the learning outcomes while writing questions, but they ignored the cognitive level. In the studies conducted, it is seen that teachers mostly use the first two steps of Bloom's taxonomy, recall and comprehension, when writing questions and do not prefer to write questions that examine higher level cognitive skills (Aslan, 2011; Karaman, 2005). In parallel with these results, Usta (2022) stated in his study that teachers had difficulty in understanding the cognitive levels of the questions and that they had problems in writing questions that associate higher-order cognitive skills with learning outcomes. The relationship between the questions asked and their cognitive levels is very important in the development of students' higher-order thinking skills (Savage, 1998). Therefore, eliminating these deficiencies of teachers is an important point in improving the quality of education. In the researches (Aydın, 2019; Khan & Inamullah, 2011; Maden & Durukan, 2013), it is seen that teachers generally use exams in which different question types (short answer, multiple choice, open-ended, matching, etc.) are used together in line with a recommendation made by the Ministry of National Education (MEB, 2003) without looking at the cognitive levels of the achievements while writing questions (Özen, 2020). And herein, the use of question types that measure lower-level skills while testing the acquisitions for higher level cognitive skills will cause learning level of the students to be measured with an exam that is not valid and decisions will be made accordingly. In this respect, it is important that the exam questions are examined with a valid question type in terms of content and cognitive level. In the study, it was found that instructors in all areas had very poor language and expressive skills. Particularly expression abnormalities, subject-predicate inconsistencies, and poor punctuation were seen in the written questions. Moreover, the fact that Turkish teachers made grammatical errors just like other branch teachers, is an important finding that needs to be emphasized. Although there is not much research on this finding in the literature, Arıcı (2021), in his research examining the general writing skills of pre-service teachers, found that errors related to content, main idea, coherence, expression disorder, title, information mistakes, grammar mistakes, word repetitions, spelling and punctuation were made in the texts written regardless of the branches. As a result of the findings of the research, it should be emphasized that the ability to use written language correctly should be acquired both during university education and while working as a teacher. Another conclusion of this study is that teachers prefer to duplicate questions from sources when creating open- ended questions, are not particularly creative, or are limited by the question-writing style they have chosen (similar to the one in another source). These findings lead to the recommendation that teachers get in-service training in order to understand the significance of creating accurate questions and employing accurate language when evaluating their students and to remedy their shortcomings in this area. References Akay, H., Soybaş, D., & Argün, Z. (2006). Problem-building experiences and the use of open-ended questions in mathematics teaching. Kastamonu Journal of Education, 14(1), 129–146. Anderson, L. W., & Krathwohl, D. R. (2001). A taxonomy for learning, teaching, and assessing: A revision of bloom’s taxonomy of educational objectives. New York: Addison Wesley Longman. Arıcı, B. (2021). Written expression errors of teacher candidates. Journal of Basic Education(2), 42-50. https://doi.org/10.52105/temelegitim.3.2.4 Aslan, C. (2011). The effect of teaching practices aimed at improving questioning skills on the question creation skills of teacher candidates. Education and Science, 36(160), 236-249. Atılgan, H. (2004). A study of generalizability theory and comparison of the multivariate Rasch model. Unpublished PhD Thesis, Hacettepe University, Institute of Social Sciences, Ankara. Aydın, M. (2019). Ortaokul Türkçe dersi sınav sorularının incelenmesi (Examination of Middle School Turkish course exam questions)(PhD Thesis). It was Obtained from the National Thesis Center of the Council of Higher Education Thesis No. 561793. Badger, E., & Thomas, B. (1991). Open-ended questions in reading. Practical Assessment, Research & Evaluation, 3(1), 1-4. Bahar, M., Nartgun, Z., Durmus, S., & Bıçak, B. (2012). Traditional-complementary assessment and evaluation techniques: Teacher's handbook. Marka: Pegem Academy. https://doi.org/10.52105/temelegitim.3.2.4 Asian Journal of Education and Training, 2023, 9(2): 42-47 47 © 2023 by the author; licensee Asian Online Journal Publishing Group Boaler, J. (1998). Open and closed mathematics: Student experiences and understandings. Journal for Research in Mathematics Education, 29(1), 41-62. https://doi.org/10.5951/jresematheduc.29.1.0041 Brookhart, S. M. (2015). Making the most of multiple choice. Educational Leadership, 73(1), 36-39. Cooney, T. J., Sanchez, W. B., Leatham, K., & Mewborn, D. S. (2004). Open-ended assessment in math: A searchable collection of 450+ questions. Portsmouth, NH: Heinemann Educational Books. Demirtasli, N. (2010). Everyday life element in measuring higher order thinking skills. Cito Education: Theory and Practice, 7, 9-26. Dogan, N. (2013). Written examinations. In H. Atılgan (Ed.), Measurement and evaluation in education. In (pp. 145-168). Ankara: Ani Press. Erkus, A. (2006). Measurement and evaluation concepts and practices for homeroom teachers. Ankara: Equinox. Friborg, O., & Rosenvinge, J. H. (2013). A comparison of open-ended and closed questions in the prediction of mental health. Quality & Quantity, 47(3), 1397-1411. https://doi.org/10.1007/s11135-011-9597-8 Hadenfeldt, J. C., Bernholt, S., Liu, X., Neumann, K., & Parchmann, I. (2013). Using ordered multiple-choice items to assess students’ understanding of the structure and composition of matter. Journal of Chemical Education, 90(12), 1602-1608. https://doi.org/10.1021/ed3006192 Hancock, C. L. (1995). Implementing the assessment standards for school mathematics: Enhancing mathematics learning with open-ended questions. The Mathematics Teacher, 88(6), 496-499. https://doi.org/10.5951/mt.88.6.0496 Kan, A. (2013). Mandatory qualifications measurement tools. (Ed: Hakan Atılgan), measurement and evaluation in education. In (6th ed., pp. 23-80). Ankara: Ani Press. Karaman, I. (2005). Analysis of physics exam questions in high schools in Erzurum according to the steps of bloom taxonomy. Faculty of Education Journal, 25(1), 77– 90. Khan, W. B., & Inamullah, H. M. (2011). A study of lower-order and higher-order questions at secondary level. Asian Social Science, 7(9), 149- 157. https://doi.org/10.5539/ass.v7n9p149 Kutlu, Ö., Doğan, C. D., & Karakaya, İ. (2010). Determination of student success. Ankara: Pegem Academy. Maden, A., & Durukan, A. (2013). Evaluation of Turkish lesson written questions according to learning areas, question types and taxonomies (Erzurum province example). Journal of Ataturk University Kazim Karabekir Education Faculty, 19 95-114. MEB. (2003). Ministry of national education (MEB) regulation on primary education institutions. Official Gazette dated 27 August 2003 and numbered 25212. Retrieved from https://www.resmigazete.gov.tr Miri, B., David, B., & Uri, Z. (2007). Purposely teaching for the promotion of higher-order thinking skills: A case of critical thinking. Research in Science Education, 37, 353–369. https://doi.org/10.1007/s11165-006-9029-2 O'Neil Jr, H. F., & Brown, R. S. (1998). Differential effects of question formats in math assessment on metacognition and affect. Applied Measurement in Education, 11(4), 331-351. https://doi.org/10.1207/s15324818ame1104_3 Ozcelik, D. A. (2011). Measurement and evaluation. Ankara: Pegem Academy Press. Özen, O. (2020). Examination of Turkish teachers' open-ended question preparation skills. Unpublished Master's Thesis, Ataturk University, Erzurum. Romagnano, L. (2001). Implementing the assessment standards: The myth of objectivity in mathematics assessment. The Mathematics Teacher, 94(1), 31-37. https://doi.org/10.5951/mt.94.1.0031 Savage, L. B. (1998). Eliciting critical thinking skills through questioning. The Clearing House, 71(5), 291-293. https://doi.org/10.1080/00098659809602727 Usta, H. G. (2022). Writing test items to evaluate higher order thinking of school teachers. Inonu University Journal of the Faculty of Education, 23(1), 486-514. Wright, R. J. (2010). Multifaceted assessment for early childhood education. Los Angeles: Sage Publications. Yildirim, A., & Simsek, H. (2011). Qualitative research methods in the social sciences (8th ed.). Ankara: Seçkin Yayınevi. Zoller, U., Ben-Chaim, D., Ron, S., Pentimalli, R., Scolastica, S., Chiara, M. S., & Borsese, A. (2000). The disposition toward critical thinking of high school and university science students: An interintra Israeli-Italian study. International Journal of Science Education, 22(6), 571-582. https://doi.org/10.1080/095006900289679 Asian Online Journal Publishing Group is not responsible or answerable for any loss, damage or liability, etc. caused in relation to/arising out of the use of the content. Any queries should be directed to the corresponding author of the article. https://doi.org/10.5951/jresematheduc.29.1.0041 https://doi.org/10.1007/s11135-011-9597-8 https://doi.org/10.1021/ed3006192 https://doi.org/10.5951/mt.88.6.0496 https://doi.org/10.5539/ass.v7n9p149 https://www.resmigazete.gov.tr/ https://doi.org/10.1007/s11165-006-9029-2 https://doi.org/10.1207/s15324818ame1104_3 https://doi.org/10.5951/mt.94.1.0031 https://doi.org/10.1080/00098659809602727 https://doi.org/10.1080/095006900289679