ii Annals of Applied Psychophysiology December 2024 Volume 11 ______________________________________________________________________________ Assessing the Effectiveness of Training Logic and Reasoning Skills Among Middle and High School Students Using a Retrospective Pre to Post Training Assessment Gabrielle D. Timlin1 Published online: December 2024 © The Author(s) 2024 Abstract Background: Both students studying psychophysiology and professionals performing research in the field need to be versed in the basics of logic so they can do a better job assessing the literature and designing their studies. Hence, it is imperative to assist middle and high school students with the development of this crucial skill. The purpose of this study was to assess the results obtained from instruction in logic in middle and high school students. This retrospective study ascertained if students gained improved logic and reasoning skills after participating in the specified course in formal logic. Methodology: Logic and critical thinking skills were taught using an established curricula with a custom pre and post assessment to determine baseline and learning. The participants were middle and high school students for whom one-half hour of instruction for ten days was conducted in a regular classroom setting, as part of their regular instruction. The first and last days consisted of pre and post-assessments using custom pre and post-assessments of their logical abilities. The instruction on days 2-9 was accompanied by clearly identified learning goals, follow-up group Socratic discussion, and written individual exit tickets. Data collection consisted of documenting assessment results. 31 matched pairs for Group 1 (the middle school students) and 32 matched pairs for Group 2 (the high school students) were analyzed using the Wilcoxon Signed Rank Sum test (Glanz, 2012). Results: Overall, students performed better on the post-test after instruction. The data from the assessments were gathered retrospectively for the purposes of this study. Results indicate that this curriculum is a good choice for logic instruction for middle and high school students. Conclusion: Future research should include larger sample sizes, student adherence to instruction and participation, inclusion of the psychomotor learning domain and validity. Keywords: Psychophysiology, Students, Curriculum, Logic and Critical Thinking Skills 1. Gabrielle D. Timlin1 Saybrook University, Pasadena, California, USA iii Introduction Both students studying psychophysiology and professionals performing research in the field need to be versed in the basics of logic so they can do a better job assessing the literature and designing their studies. Genesereth and Chaudhri (n.d.) assert that teaching logic is imperative for the creation of a responsible population that is equipped to detect logical fallacies, engage in good decision making, think critically and question their appointed leaders with skill. Students of mathematics benefit from learning logic when attempting to solve difficult problems like proofs (Bako, 2002). However, people who are neither mathematicians nor scientists may not continue to use these disciplines as an analytic foundation (Kobylarek, 2020). Furthermore, the exercise of analyzing one’s own thinking through the Socratic method elucidates gaps in thinking using questions to pursue logic and reasoning (Perdue, 2014). A solution is to teach formal logic as it is applicable to daily life (Genesereth & Chaudhri, n.d.). Modern researchers have provided updated guidelines and structure for teaching and assessing logic and critical thinking, including Robinson (2011) who argues that logic education should be relatable to key studies. The purpose of this retrospective project was to explore the results obtained from a short- term implementation of a course of study designed to develop skills in logic and critical thinking in middle and high school students. The research question to be answered in this study was, do students gain improved logic and reasoning skills after participating in this course in formal logic? Methodology The logic training program was performed in two different school settings in classrooms as part of the regular student curriculum. Participants Classes of middle school and high school students received pre- and post-tests with logic instruction in between. The researcher moved from one school to another and taught the same logic module in each school. The first group consisted of 31 total matched sets of 8-12 grade students and the second group consisted of 32 matched sets of 12 grade students only, where matched sets refer to pre and post-tests. The first group had 2 classes of 65 total 8th grade science students, 2 classes of 9th -12th grade science students of 27 total, and 1 class of 5 total physics senior high school students and one junior. Of these, the 8th grade classes had 25 males and 40 females, the Earth Science classes had 16 males and 11 females, and the physics class had 3 males and 2 females. The second group included 4 classes of 74 seniors total comprised of 31 males and 43 females. All groups were de-identified. Both groups were comprised of students from middle-class socioeconomic backgrounds. Out of both groups, there were students who elected not to participate. Additionally, the instructions were to choose a non-identifiable name consisting of a letter, a number and a symbol and use that identifier for both pre and post-tests. As not everyone chose to follow the instructions and tests were matched afterward, 31 matched sets from group 1 were used and 32 matched sets from group 2 were used as there was strong confidence that these were indeed matched sets. Both schools were the workplace of the researcher who moved from one place to the next. The researcher both taught and assessed classes in logical thinking using a questionnaire iv created for that purpose. The research was limited by the retrospective nature of that intervention and data. This was a retrospective study that utilized data collected from a middle school and a high school which had modules emphasizing instruction in logic in place. As part of the curriculum, the instructor developed pre and post-training questionnaires to assess the effectiveness of the instruction. Each school gave written permission to use the retrospective data gathered while teaching the logic modules. Both the instrument and the instruction were all given in English. All students participated in the units as these were required, however students were given the option to refrain from participating in completing the inventories. Only 3 out of the 100 students in Group 1 (middle school) and 7 out of the 74 students in Group 2 (high school) elected to not participate in completing the inventories. Procedure A baseline was obtained in the form of an assessment used to determine current ability in logic, reasoning, and critical thinking. The Inventory of Instruments of Critical Thinking was consulted (Follman, et al., 1996). Several tests were ruled out as they were either inappropriate for the intended age group, older, test-retest reliability was poor, such as free online tests available at the time, or student laptops were not consistently available. Additionally, while the Test of Logical Thinking (TOLT) exists for 6th grade through college, this instrument assessed information already being assessed in math and science classes at the sites (Tobin & Capie, 1981). It did not assess whether logic skills acquired in math and/or science resulted in increased student ability to reason in logical fallacies through conversation. After reviewing the literature, no short instrument was discovered that was suitable to the intended population or purpose. Hence, custom pre and post-tests based on logical fallacies were created (Sherman, 2021). The questions with logical fallacies are shown below in Figures 1-2. It is a standard procedure for inventories given close together in time to have different versions and/or test questions that test the same information; hence it was necessary for the pre and post-tests to have different questions. The questions were initially informally evaluated for validity and understanding by two teachers and ten students, five of whom were girls, and five of whom were boys, at the first school where the instruction was to be tested. No concerns were registered by any student or teacher. Prior to participating, the learning outcomes were clearly stated. After participating, students were required to produce outcomes, which were a variety of spoken group discussions or written feedback. Because the baseline assessment was likely to provide valuable information from which lessons can be best determined, because lessons were taken from an established curriculum, and because this was integrated into different science classes, structured lesson plans were not included in this protocol (Cothran, 2012; Cothran, 2017; Cothran 2018a; Cothran 2018b). After the initial assessment on Day 1, lessons were given on Days 2-9 during regular class time, with stated desired outcomes, activity, Socratic discussion with questions intended to elicit higher thought, and a written exit ticket from each student on the topic of the day’s learning objectives. Questions may be an analysis of a logical fallacy, for example. These lessons consisted of lessons from the Traditional Logic 1 curriculum that were designed to enhance logic and critical thinking skills, but that were not correlated to the test (Cothran, 2012; Cothran, 2017; Cothran 2018a; Cothran 2018b). Since the content was correlated to the test, this reduced the likelihood that the students were trained to succeed on an assessment as opposed to reaching the v desired goal: strengthening their logic and critical reasoning skills. This curriculum utilizes a student workbook and textbook. A mixture of individual tasks, small group work and whole class readings and discussions were utilized. The assessment was given again on Day 10 to ascertain the extent to which students learned logic and critical thinking skills. Because it was unlikely that a perfectly correct or a perfectly incorrect answer would be given in all cases, a Likert scale was created to grade the pre and post assessments. Results In most cases, the p value was less than the significance level (alpha) meaning that the results are statistically significant and there is a difference between the pre and posttests, for a two tailed (non-directional) interpretation, which means that students either improved or declined in skill (Glanz, 2012). The exception to this is the third question for Group 1 where there is no difference between the pre and posttests. However, a one-tailed interpretation is necessary because the point of the intervention is to determine if students are able to employ logic better post instruction than before. To do this, the significance level needs to be divided by two. This means that 0.05 which indicates a 95% level of confidence, needs to be divided by two to become 0.025. Hence, when using total averages in the Wilcoxon Signed Rank Sum, each group showed improvement post instruction (Glanz, 2012). When averaging each individual question, most questions indicated improvement post instruction except for Questions 1 and 4 in Group 2, and Question 3 in Group 1, where no improvement was indicated. In each case, the W score indicates that many participants had the same rank, except for Question 1 for Group 2 (Glanz, 2012). The second question yielded the highest W score for each group, thus skewing the total average for the W score to a higher number than it would have otherwise been. In comparison, question 3 yielded the highest P score for group 1 and was the only question for Group 1 where there was no positive difference between the intervention of logic education and no intervention at all. This was the case for questions 1 and 4 for Group 2. A Wilcoxon Signed-Rank Test indicated that scores were significantly different before and after the intervention when averaged for all questions, with W= 373, p = 0 for Group 1 and W = 310, p = 0 for Group 2 (Glanz, 2012). For each independent question, the only question Group 1 did not show improvement with the intervention was question 3. For Group 2, it was questions 1 and 4. vi Table 1 Student Breakdown. Group 1 Group 2 # of students who completed the questionnaire as a matched set 31 32 8th Grade Science total students 65 0 8th grade 65 0 Males 25 0 Females 40 0 Earth Science total students 27 0 Males 16 0 Females 11 0 9th grade 21 0 Males 12 0 Females 9 0 10th grade 3 0 Males 2 0 Females 1 0 11th grade 2 0 Males 2 0 Females 0 0 12th grade 6 74 Males 3 31 Females 3 43 Physics total students 5 74 vii Table 2 Statistics By Group (Glanz, 2012). Test Group 1 Group 2 Standard Deviation 0.82 0.49 Power Analysis 1 0.989 Wilcoxon Avg W 373 310 Wilcoxon Avg P 0 0 Wilcoxon Q1 W 120 0 Wilcoxon Q1 P 0 >0.046 Wilcoxon Q2 W 337 435 Wilcoxon Q2 P 0 0 Wilcoxon Q3 W 107 88 Wilcoxon Q3 P 0.057 0.007 Wilcoxon Q4 W 182 61 Wilcoxon Q4 P 0 0.044 Wilcoxon Q5 W 135 72 Wilcoxon Q5 P 0.003 0.019 Figure 1 Box and Whisker Plot for Group 1 W Values Distribution by Question Averages W V al ue s 0 50 100 150 200 250 300 350 Group 1 W values by question Group 1 ii Figure 2 Box and Whisker Plot for Group 2 W Values Distribution by Question Averages Figure 3 Box and Whisker Plot for Group 1 P Values Distribution by Question Averages The following are the data. W V al ue s 0 50 100 150 200 250 300 350 400 450 500 Group 2 W values by question Group 2 1 P Va lu es 0 0.01 0.02 0.03 0.04 0.05 0.06 Group 1 P Values by Question Group 1 iii Figure 4 Box and Whisker Plot for Group 2 P Values Distribution by Question Averages Discussion With the exception of one question in Group 1 and two questions in Group 2, there is a statistically significant difference between the pre and posttests, indicating that this particular curriculum is a good choice for future logic instruction in middle and high school settings. Both sites were public classical charter schools. Since the population represented those students enrolled in specified science classes at that time, this sample population is representative of middle and high school students enrolled in classical charter schools. Hence, these results are generalizable to middle and high school students enrolled in classical charter schools. Limitations include small sample sizes in both groups, lack of student adherence to instructions, lack of student participation, short study duration, limited number of science classes and schools, lack of time to devote solely to logic in a science classroom and integration of logic into the science curriculum. Further limitations include a general reluctance by students to value logic instruction in a science classroom and answer interpretation done by only one person, the researcher. Finally, the exploratory, simple nature of the study and the survey are limitations. Delimitations involve teaching this in a science classroom. The activity was integrated into the science curriculum for middle and high school students in two schools. Recommendations include repeating instruction in a dedicated logical class including at least 30 minutes of class time up to one hour and consisting of a combination of individual reading to start class, teacher instruction and a question-and-answer session afterwards followed by small group participation where the students perform exercises in the curriculum and finally ending with an exit ticket demonstrating understanding of the day’s learning. Also, logic P Va lu es 0 0.005 0.01 0.015 0.02 0.025 0.03 0.035 0.04 0.045 0.05 Group 2 P Values by Question Group 2 iv instruction should be connected with math and science classes reinforcing math and science instruction as well as be performed at sites that exemplify excellent teacher support and prioritize student responsibility for their own learning. Additionally, more research is necessary to study the inclusion of the psychomotor learning domain, as this learning domain is applicable in practical application, as opposed to simply acquiring information as in the learning domain. Furthermore, as this was exploratory research, evaluation of validity should be included in future research. Conclusion In conclusion, it is unwise to assume that students will learn logic and reasoning skills by exposure to math and science courses. Instead, logic and reasoning should be facilitated in middle and high school classes in conjunction with math and science education to facilitate grasp of basic concepts and promote student responsibility for independent thinking. 1 References Aksu, G., & Koruklu, N. (2015). Determination the effects of vocational high school students’ logical and critical thinking skills on mathematics success. Eurasian Journal of Educational Research, 15(59). https://doi.org/10.14689/ejer.2015.59.11 Bako, M. (2002). Why we need to teach logic and how can we teach it? International Journal for Mathematics Teaching and Learning, 1473. https://cimt.org.uk/journal/bakom.pdf Cothran, M. (2012). Traditional logic 1: Introduction to formal logic: Quizzes & tests (2nd ed.). Memoria Press. Cothran, M. (2017). Traditional logic 1: Introduction to formal logic: Student text (3rd ed.). Memoria Press. Cothran, M. (2018a). Traditional logic 1: Introduction to formal logic: Teacher key (3rd ed.). Memoria Press. Cothran, M. (2018b). Traditional Logic I: Introduction to formal logic: Workbook (Classical Trivium Core) (3rd ed.). Memoria Press. Crockett, L. (2023, July 26). 5 of the best practices for improving critical thinking skills: Future focused learning. Future Focused Learning. https://futurefocusedlearning.net/blog/critical-thinking/5-of-the-best-practices-for- improving-critical-thinking-skills Follman, J., Lavely, C., & Berger, N. (1996). Inventory of instruments of critical thinking. Informal Logic, 18(2). https://doi.org/10.22329/il.v18i2.2383 Genesereth, M. & Chaudhri, V. (n.d.). Logic in secondary education. Stanford University. Retrieved November 3, 2021, from http://intrologic.stanford.edu/miscellaneous/intro.html Glantz, S. A. (2012). Primer of biostatistics: Seventh edition. The McGraw-Hill Companies, Inc. History of Logic | Ancient, medieval, modern, & contemporary logic. (n.d.). Encyclopedia Britannica. Retrieved November 29, 2021, from https://www.britannica.com/topic/history-of-logic Ikuenobe, P. (2001). Teaching and assessing critical thinking abilities as outcomes in an informal logic course. Teaching in Higher Education, 6(1), 19–32. https://doi- org.tcsedsystem.idm.oclc.org/10.1080/13562510020029572 Kobylarek, A. (2020). Power as knowledge. The reverse logic of the post-scientific world. Journal of Education, Culture & Society, 11(2), 5–14. https://doi- org.tcsedsystem.idm.oclc.org/10.15503/jecs2020.2.5.14 https://doi.org/10.14689/ejer.2015.59.11 https://cimt.org.uk/journal/bakom.pdf https://futurefocusedlearning.net/blog/critical-thinking/5-of-the-best-practices-for-improving-critical-thinking-skills https://futurefocusedlearning.net/blog/critical-thinking/5-of-the-best-practices-for-improving-critical-thinking-skills https://doi.org/10.22329/il.v18i2.2383 http://intrologic.stanford.edu/miscellaneous/intro.html https://www.britannica.com/topic/history-of-logic https://doi-org.tcsedsystem.idm.oclc.org/10.1080/13562510020029572 https://doi-org.tcsedsystem.idm.oclc.org/10.1080/13562510020029572 https://doi-org.tcsedsystem.idm.oclc.org/10.15503/jecs2020.2.5.14 https://doi-org.tcsedsystem.idm.oclc.org/10.15503/jecs2020.2.5.14 2 Morozova, E. (2014). Development of the system of pedagogical monitoring of students’ logical thinking and logical reflection. Concept / Koncept, 10, 1–7. Perdue, S. M. (2014, September 19). The big three of greek philosophy: Socrates, plato, and aristotle. | English Blog. Penn State. Retrieved November 29, 2021, from https://sites.psu.edu/rclperdue/2014/09/19/the-big-three-of-greek-philosophy-socrates- plato-and-aristotle/ Piaget, J. (1999). The child’s conception of physical causality (1st ed.). Routledge. Publication manual of the American Psychological Association (7th ed.). (2020). In American Psychological Association eBooks. https://doi.org/10.1037/0000165-000 Robinson, S. R. (2011). Teaching logic and teaching critical thinking: revisiting McPeck. Higher Education Research and Development, 30(3), 275–287. https://doi.org/10.1080/07294360.2010.500656 Rochat, P. (2024). The evolution of developmental theories since piaget: A metaview. Perspectives on Psychological Science, 19(6), 921–930. https://doi.org/10.1177/17456916231186611 Sherman, R. (2021). Logical fallacies. [Unpublished manuscript]. Department of Applied Psychophysiology, Saybrook University. The sophists; Stanford encyclopedia of philosophy. (2020, August 18). Stanford Encyclopedia of Philosophy. Retrieved November 29, 2021, from https://plato.stanford.edu/entries/sophists/ Tobin, K.G., & Capie, W. (1981). The development and validation of a group test of logical thinking. Educational and psychological measurement, 41(2), 413–423. Well, T., PhD. (2023, July 5). The decline of critical thinking skills. Psychology Today. https://www.psychologytoday.com/us/blog/the-clarity/202306/the-decline-of-critical- thinking-skills UNC-Chapel Hill Writing Center. (2021, April 14). Fallacies. The Writing Center at University of North Carolina at Chapel Hill. Retrieved November 29, 2021, from https://writingcenter.unc.edu/tips-and-tools/fallacies/ https://sites.psu.edu/rclperdue/2014/09/19/the-big-three-of-greek-philosophy-socrates-plato-and-aristotle/ https://sites.psu.edu/rclperdue/2014/09/19/the-big-three-of-greek-philosophy-socrates-plato-and-aristotle/ https://doi.org/10.1037/0000165-000 https://doi.org/10.1080/07294360.2010.500656 https://doi.org/10.1177/17456916231186611 https://plato.stanford.edu/entries/sophists/ https://www.psychologytoday.com/us/blog/the-clarity/202306/the-decline-of-critical-thinking-skills https://www.psychologytoday.com/us/blog/the-clarity/202306/the-decline-of-critical-thinking-skills https://writingcenter.unc.edu/tips-and-tools/fallacies/