Student Performance and Critical Thinking Skills: Online Discussion Boards in a Computer Science Operating Systems Course Page 137 - Developments in Business Simulation and Experiential Learning, Volume 46, 2019 ABSTRACT As online courses and programs proliferate, more and more students are experiencing the benefits of online technologies within their coursework. Technology tools such as the online discussion forum (ODF) are becoming commonplace and researchers want to know the impact of these tools on student learning and achievement. In this study, 159 computer science students use the ODF in an online operating systems course. The use of ODF, based upon problem-based learning and engagement theories, are explored as to the effectiveness of higher-order skill attainment and overall student performance in the course. Performance is evaluated using the final exam and final grade. Students who scored well in the ODFs that asked higher order skill questions, such as analysis and application, attained better results on some higher order thinking questions on the final exam. Significant results are also found in the ANOVA and regression analysis between the effective use of critical thinking skills in the discussion boards and student performance measures overall. INTRODUCTION In the practical fields of applied sciences such as computer science, information systems and business, we believe that students need higher order thinking skills for success, both academically and professionally. Bloom and others (1956) led the way in classifying and distinguishing lower order from higher order thinking skills and has been used pervasively in the research dealing with learning and educational assessment. The introduction of computer mediated communications (CMC) has brought about the usage of new teaching/learning tools such as asynchronous discussion or online discussion forum (ODF), which “makes up a significant and very important component in distant interaction” (Biriyai & Thomas, 2014, 111). As professors, we are the primary designers and users of learning environments that create activities to promote improved skill levels and learning for students (Mazzolini & Maddison, 2007). So how do we know that students are learning critical thinking skills and what impact they are having on their performance? In addition, which online tools have been effective in learning higher order skills? In the online environment, we often design our courses to assess various levels of skills. This research is interested in the intersection of assessment of student’s level of higher order skills using a popular online tool, the asynchronous discussion board or ODF, and the impact on students’ performance in an online computer science course. LITERATURE REVIEW In the literature review, we started with a review of studies investigating and summarizing different levels of higher order skill attainment. Next, we summarized the theoretical bases of engagement theory and problem-based learning with respect to online tools used in the applied sciences, particularly computer science. Lastly, we reviewed the research on the effectiveness of the online discussion forum (ODF) tool to enhance higher order thinking skills and strengthen student performance. Higher Order Skills Research Higher-order thinking is an instructional strategy supported by research (Ennis, 1985). Often referred to as critical thinking, it is more than simple recall of facts or information. It is a function of the interaction between cognitive strategies, meta-cognition, and nonstrategic knowledge when solving problems (O’Neill & Murphy, 2010). Higher-order thinking is based on the concepts in the cognitive domain of Bloom et al.’s Taxonomy (1965) where some learning requires more cognitive processing than others. Higher order skills including analysis, evaluation and application, often need different instructional practices (Bloom et al., 1965). They also suggest that higher-order thinking involves “the learning of complex judgmental skills such as critical thinking and problem solving.” Higher-order thinking is thought to be more useful and necessary for survival in the 21st century because such skills (analysis, application and synthesis) are considered more likely to be transferable to situations outside those in which the skill Student Performance and Critical Thinking Skills: Online Discussion Boards in a Computer Science Operating Systems Course Richard Fulton Troy University Rfulton33574@troy.edu Diane Fulton Clayton State University DianeFulton@clayton.edu mailto:Rfulton33574@troy.edu mailto:DianeFulton@clayton.edu Page 138 - Developments in Business Simulation and Experiential Learning, Volume 46, 2019 was initially learned (Lim & Tay, 2003; Paul, 1990). Similarly, Krathwohl (2002) explains there are two dimensions in learning: 1) the knowledge dimension which ranges from factual and conceptual to meta-cognitive; and 2) the conceptual process dimension, where he delineates 6 critical thinking skills. The conceptual skills range from lower level skills such as “remembering’ (recognizing, recalling) and ‘understanding’ (classifying, summarizing) to higher order skills such as ‘applying’ (executing, implementing, using the knowledge in another situation), ‘analyzing’ (differentiating, comparing), and ‘evaluating’ (critiquing) (Huitt, 2011). Questioning is one of the “essential nine” instructional practices identified by Marzano, Pickering and Pollock, (2007). It is closely linked to higher-level thinking and Bloom et al.’s Taxonomy (1965). In a marketing education course, Hernandez (2002) effectively used questions to access student learning along the lines of Bloom et al.’s (1956) Taxonomy comparing lower and higher- level thinking skills. While teachers’ use of questions is predominantly low-level, many researchers recommend more professional development to help teachers develop the skill to design and use questions that engage students in higher-level instructional processes (Baeten, Dochy & Struyven, 2008; Marzano et al., 2007; Newmann & Wehlege, 1993). As computer science faculty at a military institution, Fagin, Harper, Baird, Hadfield, and Sward (2006) found in their research that critical thinking skills are developed both implicitly, through the practice of questioning assumptions and explicitly, through online discussion assignments designed to test such thinking skills. In the computer science major, Fagin et al. (2006) outline 3 examples of enhancing analytical, application and evaluation thinking skills. For example, in the networking course, students outline similarities and differences between Java and .NET frameworks; in their first programming course, they use a simulation trying different scenarios, analyze the results and draw conclusions applying what they have found; in the software engineering course, students give a critical review of open source versus strictly proprietary capabilities of software while recommending their preferences thus enhancing their evaluation skills. Next, we looked at research that tells how we encourage higher level learning in students and found some studies that gave us some pedagogical strategies to achieve this goal. Over three-quarters of math teachers indicated they often use an online system to track student progress and grades and the use of online resources was found to be higher in the highest performing schools in the State of Washington (Elfers, Plecki, Knapp, Yeo & McGowan (2007). The majority of these teachers agreed on the efficacy of using “multiple instructional strategies and emphasized problem-solving and critical thinking” in their curriculum as well (p. 42). The Engagement Theory, PROBLEM-BASED LEARNING and Technology TOOLS The underlying basis of “The Engagement Theory” is that students should be “meaningfully involved in their learning through interactive and worthwhile tasks” (Kearsley & Shneiderman, 1998, 20). Technology often can promote engagement that is difficult to achieve otherwise (Fink, 2013). One technology that has been used to engage students in a meaningful way in distance learning is the online discussion forum (ODF). The social interaction and the collaborative active learning nature of the ODF environment have been found to increase achievement when students utilize ODFs to learn (Alzahrani, 2017; Wu & Hiltz, 2004). The problem of communication between students and teachers in higher educational institutions is a major concern. This is because some students find it difficult to participate in traditional classroom lectures because of their inability to socialize or lack of confidence; sometimes faculty do not have enough time to engage each student through questions or elaborate on the topics they want to cover in the traditional classroom, which can both decrease students’ understanding of a given topic. Cormer and Lenaghan (2013, 261) give evidence that ODFs are more “inclusive” and “facilitate student learning” more effectively than “face to face” discussions for some students. In addition, Cox and Cox (2008) found that collaboration and reflective learning are effectively taking place in online discussions. Students rate ODFs highly since they are engaging, vibrant and active (Revill and Terrell, 2005). As Koole (2009) suggests, any tool that has the integration of device, learner, and social aspects, has a higher chance of success for mobile learning. In illustration of how the engagement theory works, Miliszewska & Horwood (2006) assigned a real-life software development project to their students online who resided in Australia and Hong Kong. Three learning objectives were emphasized in the project: 1) “to appreciate the needs of the business client for whom they are expected to build the software system; 2) to apply software engineering and database design methodologies to the design and implementation of a complete system; and 3) to confront issues developers face on a daily basis, such as liaison with clients, working in a team, documenting the project” (p. 1). Through projects, students gain hands-on experience and problem-solving skills that enhance their learning in a “Problem-based Learning (PBL)” environment (Campidoglio, Frattolillo & Landolfi, 2009; Fogarty, 1997; Kingsland, 1996). The “role of the teacher is different too: it is one of consultant rather than instructor” and students often work in groups (Miliszewska & Horwood, 2006, 2). The learning that occurs in working with a problem enables students to acquire new knowledge, and deepen their existing skills and knowledge (Schiller, Ostwald & Chen, 1994). Students have a greater responsibility for their own learning and receive less guidance from the teacher in the online environment (Dolmans, 1992). In fact, faculty at Victoria University, feel that the project- based activity is “essential to transforming computing students into competent graduates” (Miliszewska et al., 2006, 3). Lomo-David & Shannon (2009) also support the idea that projects, increasing student’s practice and problem-solving skills in informational systems, increase their employability. Cockburn (2004) advocates using gaming to simulate software development projects in the online classroom as well since they can be controlled and evaluated more effectively. Kazemi and Ghoraishi (2012) conducted a comparison of Problem-based Learning (PBL) approaches (Duch, Groh & Allen, 2001) and traditional teaching on attitude, misconceptions and mathematics performance of university students in an applied math class; Hillman (2003) also emphasized problem-based learning approaches to teacher education as a strategu to engage the students more effectively in their own learning. The results showed that the “PBL teaching approach is more effective than traditional methods in teaching mathematics” since it enhanced “students’ performance and attitude toward mathematics” (Kazemi & Ghoraishi, 2012, 3852). Struyven, Dochy, Janssens and Gielen (2006) focused on the relationships between experiences with portfolio assessment, students’ approaches to learning and their assessment preferences in an office management course. The surface approach (as opposed to deeper learning such as higher level critical thinking skills) “proved https://www.researchgate.net/profile/Iwona_Miliszewska?_sg=m_o-J8v9f5yZj8GFN0rGWPcYziScnvAMfevrHQUZa0cDhtPySOyYhclbztlSGzcLUnE14O8.6sd59D2aUfQGVHWL6N_ubjQ3zrZP1AFjxIBI1kxgQHrII48d0xzfm4R5k5acB5RjRNlOHXydmlz8C7PzukkUbg https://www.researchgate.net/scientific-contributions/10653473_John_Horwood?_sg=m_o-J8v9f5yZj8GFN0rGWPcYziScnvAMfevrHQUZa0cDhtPySOyYhclbztlSGzcLUnE14O8.6sd59D2aUfQGVHWL6N_ubjQ3zrZP1AFjxIBI1kxgQHrII48d0xzfm4R5k5acB5RjRNlOHXydmlz8C7PzukkUbg https://www.researchgate.net/profile/Iwona_Miliszewska?_sg=m_o-J8v9f5yZj8GFN0rGWPcYziScnvAMfevrHQUZa0cDhtPySOyYhclbztlSGzcLUnE14O8.6sd59D2aUfQGVHWL6N_ubjQ3zrZP1AFjxIBI1kxgQHrII48d0xzfm4R5k5acB5RjRNlOHXydmlz8C7PzukkUbg https://www.researchgate.net/scientific-contributions/10653473_John_Horwood?_sg=m_o-J8v9f5yZj8GFN0rGWPcYziScnvAMfevrHQUZa0cDhtPySOyYhclbztlSGzcLUnE14O8.6sd59D2aUfQGVHWL6N_ubjQ3zrZP1AFjxIBI1kxgQHrII48d0xzfm4R5k5acB5RjRNlOHXydmlz8C7PzukkUbg https://www.researchgate.net/profile/Iwona_Miliszewska?_sg=m_o-J8v9f5yZj8GFN0rGWPcYziScnvAMfevrHQUZa0cDhtPySOyYhclbztlSGzcLUnE14O8.6sd59D2aUfQGVHWL6N_ubjQ3zrZP1AFjxIBI1kxgQHrII48d0xzfm4R5k5acB5RjRNlOHXydmlz8C7PzukkUbg https://www.researchgate.net/scientific-contributions/10653473_John_Horwood?_sg=m_o-J8v9f5yZj8GFN0rGWPcYziScnvAMfevrHQUZa0cDhtPySOyYhclbztlSGzcLUnE14O8.6sd59D2aUfQGVHWL6N_ubjQ3zrZP1AFjxIBI1kxgQHrII48d0xzfm4R5k5acB5RjRNlOHXydmlz8C7PzukkUbg Page 139 - Developments in Business Simulation and Experiential Learning, Volume 46, 2019 to be a significant negative predictor of the portfolio assessment score” (Struyven et al., 2006, 263). Effectiveness of THE ONLINE DISCUSSION FORUM Computer mediated communication (CMC) has created the space for the effective use of the online forum for communication (Biriyai & Thomas, 2014). The online discussion forum (ODF) is a web-based application that brings people together with shared interests and mindsets (Edwards, 2005). It is an e-learning platform where students can post messages to the discussion threads, interact and receive feedback from other students and the instructor, and hence creates a deeper understanding of the subject matter being discussed. In computer education and business, ODFs have been deployed to complement traditional learning techniques such as lectures and tutorials (Dube, Bourhis & Jacob, 2006; McDonnell, 2000). ODFs also “harmonize with the educational philosophy that makes communication a necessary tool and fundamental mechanism for effective learning” (Harman & Koohang, 2005, 69). The ODF has emerged as a common tool (Tolmie & Boyle, 2000; Balaji & Charkrabarti, 2010) and an “effective way of engaging students outside the classroom” (Balaji & Charkrabarti, 2010, 1). “A study on the usage of online forum in a graduate science education course showed that students’ contributions to a subject under discussion were more detailed and deliberate when interacting online compared to face-to-face interactions” (Rodrigues, 1999, 263). Thomas (2002) explains that online discussions “will enhance learning, including increased motivation, engagement, and deeper levels of understanding.” Anderson (2004) believes that assessment determines if the learning objectives of using online tools has been accomplished. Woo and Wang (2009) suggested that the meaning of critical thinking often depends on values and culture which may be interpreted as “argumentative” or “being critical of others” thus care in wording and clarity about what the professor is looking for, is required. Thus, from a pedagogic perspective, critical thinking skills can be learned in a given situation by the level of questions asked (King, 1990). Jalongo, Twiest, and Gerlach (1999) observed that critical thinking increases through the following higher order skills: 1) Application, where students use knowledge and understanding to complete a practical task; 2) Analysis, where students break a task down into its component parts; 3) Synthesis, where students integrate various sources of information; and 4) Evaluation, where students assess the value, or worth of something when completing a task. In an ODF, the researchers used questions on a case in an ODF to successfully increase students’ critical thinking skills (Li-Jen & Bennett, 2012). Pena-Shaff, Altman and Stephenson (2005) found several significant correlations between pre-course attitudes, expectations about online class discussion boards, and expected learning. They also discovered greater levels of participation were tied to greater satisfaction. In their research, Zhang, Zhou, Briggs, and Nunamaker (2006, 27) revealed that the “interaction of the learners with both human and inanimate objects, and their participation in technology mediated education, were essential for the quality of their learning experience”, by enriching the knowledge exchange processes among students/faculty and positively effecting students’ performance. Other researchers note that online discussion forums (ODFs) can be successful in enhancing collaborative learning by attracting students to participate and interact (Dube et al., 2006; Swan, Fredericksen, Pickett, Pelz & Maher, 2000). Three key factors of “consistency in course design, contact with course instructors, and active discussion” were crucial to the success of online courses and the creation of an online learning community (Swan et al., 2000, 359). ODFs have been described as “facilitating discourse, reflective thinking, assessment and connectedness” by Balagi et al. (2010, 17). Using a variety of instructional media in the ODF leads to enhanced learning as well as shapes students’ attitudes, expectations and perceptions of the online learning experience (Balagi et al., 2010; Pena-Shaff et al., 2005). Past research delineates the following benefits of ODF: 1) higher participation by students; 2) more in-depth discussion of topics; 3) ability to easily form small group discussions; and 4) feedback for students and faculty about what students understand and where they need clarification (Balaji et al. 2010; Brodie, Karat & Feng, 2005; Fagin et al., 2006; Mazzolini et al. 2007; Menchaca & Bekele, 2008). McDuffie and Slavit (2003) go on to suggest that the dialogical nature of an online discussion provides a forum for students to share their reflections with other participants in their introductory math classes. Critical reflection fosters deeper learning, according to Moon (1999). Comparison of students in a tourism and leisure management class, that were taught with either a traditional or electronic tutorial, was made by McDonnel (2000). A tutorial is a period of class outside the ‘regular lecture classes’ where lecture material is discussed between students and a teacher assistant or a professor. Communication in the electronic tutorial was in an asynchronous manner and occurred in a discussion board. From his assessment, students liked the e-tutorials more since they were available for future reference and there were no differences in the quality of the tutorials (McDonnel, 2000). Online forums provide an avenue where class material can be reviewed, discussed and reinforced. The “underlying presumption is that knowledge can be constructed by an online asynchronous dialogue of class material” (McDonnel, 2000, 374). The conversational model of learning by Thomas (2000) stipulates that it will enhance learning, including increased motivation, engagement, and deeper levels of understanding. Communications technologies – such as online forums – enable students to discuss class material in an asynchronous manner. This communication channel supports discussion outside the classroom and is flexible for students who work full-time (Thomas, 2000). Kanuka and Anderson (1998) used a constructivist interaction analysis model to evaluate learning in an online community. Alzahrani (2017) investigated the effect of using online discussion forums (ODFs) on students’ learning, particularly on their achievement, including a higher grade. Statistical analyses revealed significant and positive relationships between student participation in ODFs and their final course mark, but no significant relationships between their participation in ODFs and grade point average. In a similar context, AlJeraisy et al. (2015) found significant relationships between the number of times students participated in ODFs and their achievement. Collaborative findings in other contexts were also reported (Alghamdi, 2013; Carceller et al., 2013, 2015; Hartnett, 2012; Koole, 2009; Palmer & Holt, 2010; Palmer et al., 2008). These findings consistently confirmed that student participation in ODFs did contribute significantly to their achievement. In contrast, no significant relationships were found in Song and McNary’s (2011) study. It should be noted that, while the majority of previous studies with significant findings were at the undergraduate level, Song and McNary’s (2011) study was with graduate students, and little variation in students’ grades Page 140 - Developments in Business Simulation and Experiential Learning, Volume 46, 2019 was reported. Research shows that ODFs often provide expanded opportunities for students to share their thinking with each other and the instructor. Consistent with existing research evidence, these results confirm that reflective thinking cultivates meaningful interactions in ODFs. These results are important as Ellis et al. (2007) in his study reported an absence of reflection in face-to-face classrooms compared to online discussions. Ellis and others (2007) also indicated that sense of connectedness or community among the students increased the interactions in online environment. The multiple levels of discourse in ODF create ties among the students and promote academic participation. ODFs can fulfill an important social network function and build an effective learning community for students. Previous studies report reciprocity and mutuality as prerequisites to formation of online learning communities (LaPointe & Reisetter, 2008; Rovai, 2002). O’Reilly et al. (2007) and Dawson (2006) found that a sense of community or connectedness among the students, influences their forum contributions. RESEARCH QUESTIONS In this exploratory research, we are interested in how online discussion boards, assessing higher-order skills of application and analysis, impacted student achievement. Performance was measured by final exam and final grade. The research questions posed were: 1. Can online discussion forums be an effective way to assess the higher-order critical thinking skills, including analysis and application skills? 2. Can higher level skill attainment in online discussion forums impact student achievement on the final exam? 3. Will performance in online discussion forums using critical thinking skills be related to performance in the course? 4. What combination of factors impacts student performance? METHODOLOGY Data from 159 students in the fully online operating systems class for computer science majors at Troy University’s global campus from 2016 to 2018 were analyzed in this study. This is the time period covering 10 distinct course sections. The confidential data was aggregated from the learning management system (LMS) called CANVAS. Assessment data on three weekly discussion boards were compared with performance scores on the final exam and final grade scores. Three weeks of discussion boards posing questions that used higher-order thinking skills were the focus of this study. In Week 4’s discussion, students were asked to “apply their experiences to the security intrusion that has happened to one or more of the following firms: Yahoo, Verizon, Linked In, MySpace or Target”; in Week 6, the skills of analysis were emphasized by asking questions such as: “Discuss the pros and cons of installing different operating systems like Linux, VMWare Player, Fedora, Ubuntu or Raspberry Pi 3”; in Week 7, the faculty member posed an analytical question: “Discuss the Android File Management Systems and explain if and why you would recommend using them”. For approximately 40% of the students (N=63) and courses (N=4), final exam item analysis was available to calculate a score on just the application and analysis type of questions on the final exam, given as the percentage of analysis questions answered correctly (AN%) and percentage of application questions correctly answered (AP%). Item analysis data on the final exam questions for the rest of the sample (60%, N= 96, 6 courses) was not available, however. T tests, ANOVAs and Regression Analysis were performed using SPSS to answer the various research questions. This is an exploratory research study and all student data was aggregated from the LMS statistics and online gradebook information and kept anonymous. RESULTS To answer the first research question, can online discussion forums (ODFs) be an effective way to measure student achievement of higher order thinking skills, we looked at the limited LMS data (N= 63) we had with the item analysis of the final exam questions and the scores on the weekly discussion boards for the application and analysis skills. We found that none of the Table 1 T Test Results of Higher Order Skills Comparing Online Discussion Forums with AP%, AN% and Significant Individual Question Analysis Scores. Higher Skill Attainment Measures Week 4 ODF T value (sig) Week 6 ODF T value (sig) Week 7 ODF T value (sig) AP% .985 (.345) -.17 (.866) 1.07 (.300) AN% .987 (.343) -.15 (884) .15 (.883) Q 16 (AN) 3.51 (.002) ** 3.56 (.002) ** -.89 (.399) Q 25 (AP) 5.29 (.000) *** -.06 (.955) 1.56 (.181) Q 160 (AP) 4.38 (.001) ** -.34 (.745) .14 (.896) Note: ***=p< .001; **=p< .01; *=p< .05 Page 141 - Developments in Business Simulation and Experiential Learning, Volume 46, 2019 discussion board scores correlated with the AP% or AN% scores on the final exam but there were a few significant correlations with some specific final exam questions. For example, the Week 4 ODF t test showed significant relationships with two application questions from the final exam, namely, Q 25 (question about concept used in LINUX) (t (14) = 5.29, p=.000 and Q 160 (question asking for an example of an I/O processor) (t (12) =4.38, p=.001). In addition, WK4 ODF and WK6 ODF were significantly tied to an analysis question Q16 (question asking at what layer are operations and data best converted into an appropriate sequence of I/O instructions and controller orders), WK4 (t (22) = 3.51, p=.002) and WK6 (t (19) =3.56, p=.002). Therefore, research question #1 was partially supported. Please refer to Table 1 for t test results. For the second research question, will scores on the higher order skills for each discussion board impact the final exam grade, we used ANOVA tests to determine if the ODFs affected the final exam performance. ODF performance cutoff was 17 and above (85% and better) for good skill attainment and 16 and below (80% and below) for poor skill attainment. Unfortunately, none of the ODFs’ skill attainment level showed a significant link to the final exam score. However, the AP% score (percentage correct of all application questions on the exam) was significantly tied to the final exam grade (F (2, 51) = 5.509, p=.007). Again, research question #2 was partially supported. Table 2 presents the ANOVA results for the second research question. To answer research question #3 about the relationship between the skill attainment on the ODFs and the performance in the course, (measured by the final grade), the results were highly significant. All three ODFs showed strong significance to the final grade. Week 4 (WK4) ODF ANOVA results were positive (F (2, 158) =6.11, p=.003); WK 6 ODF (F (2, 158) =17.34, p =.000) and WK7 OCT (F (2, 158) = 18.82, p= .000) ANOVA statistics were even stronger. In post hoc analysis using Scheffe’s test, WK 4 showed significant differences particularly between final course grade of A and C, p= .006); WK 6 had high differences between A and C grades (p=.000) and B and C (p=.000) marks; WK 7 again strongly differentiated between A and C grades (p=.000) and B and C final course grades (p=.000). Please see Table 3 for complete summary. In the fourth research question, what combinations of factors impacted performance, we used stepwise regression analysis to examine the various factors. The overall regression model was significant at the .000 level and showed four factors that contributed 76% of the final grade. The strongest factor, final exam, explained 62% of the variance in the final grade performance at the .000 level. Wk7 discussion was the next important factor, accounting for an additional 10% of the final grade performance with Wk6 discussion 4% (p=.000) and WK4 discussion at 1% (p=.005). Please see Table 4 for regression analysis results. Table 2 e ANOVA Results of Comparison of Higher Level Skills (AP%, AN%, Online Discussion Forums scores) and Final Exam Grades Higher Skill Attainment Measures Final Exam Grade F value (sig) AP% 5.509 (.007)** AN% 2.401 (.100) WK 4 ODF (AP) 1.630 (.199) WK 6 ODF (AN) .375 (.688) WK 7 ODF(AN) .586 (.558) : Note: ***=p< .001; **=p< .01; *=p< .05 TABLE 3 ANOVA RESULTS OF COMPARISON OF HIGHER LEVEL SKILLS (AP%, AN%, ONLINE DISCUSSION FORUMS SCORES) AND FINAL COURSE GRADES Higher Skill Attainment Measures Final Exam Grade F value (sig) AP% 3.090 (.054) AN% 0.288 (.751) WK 4 ODF (AP) 6.108 (.003) ** WK 6 ODF (AN) 17.340 (.000) *** WK 7 ODF (AN) 18.820 (.000) *** Note: ***=p< .001; **=p< .01; *=p< .05 Page 142 - Developments in Business Simulation and Experiential Learning, Volume 46, 2019 DISCUSSION, LIMITATIONS AND CONCLUSIONS The current research study sought to determine the interrelationships between the higher-order thinking skills in the ODF and learning outcomes. The research was based on the frameworks of Taxonomy of Learning by Bloom and others (1956), Engagement Theory (Kearnsley et al., 1998) as well as Problem-based Learning Theory (PBL) adapted to college teaching (Duch et al., 2001; Hillman, 2003). Four research questions were proposed in this initial exploratory approach with several important findings. First, we researched the interrelationship between the types of questions asked in the ODF and the performance on the final exam questions evaluating those higher thinking skills. Although this research question was not found in the direct relationship between the score on the higher-order discussion forums and the final exam analysis (AP%) or application question (AP%) scores as expected, there was a link found between the weekly discussion board performance and three of the critical thinking questions on the final exam. This finding supports the importance of the role of instructor in facilitating the discussion in the ODF. It provides evidence that facilitating discourse using questions that promote higher-order critical thinking skills can have a positive effect on student achievement of enhanced critical thinking skills such as application and analysis, corroborating findings by Dennen, Darabi and Smith (2007), Fagin et al. (2006), Li-Jen and Bennett (2012) and Menchaca and Bekele (2008). We did not have significant results on the interrelationship between the ODF scores and the final exam in research question #2, however, the percentage of analysis (AN%) and percentage of application (AP%) questions correctly answered on the exam were highly related to higher final exam grades. We know that higher-order thinking questions on the final exam are some of the most difficult questions on the exam, based on the LMS item analysis giving difficulty percentage (% of students who answered the question incorrectly) for the final exam questions. We compared 40% of the sample on these final exam questions, so it is not surprising that they were more predictive of overall performance on the final exam and discriminated on final exam scores. This emphasizes the need for faculty to carefully design consistent and challenging ODF discussion questions and their evaluation measures, that align well with summary assessments, such as the final exam. Another area we studied was the impact of the weekly ODFs on students’ performance in the course, namely the course grade (Research Question #3). Findings were positive for all three ODFs having an impact on course performance. One of the limitations of this study was that individualized analysis of higher-order questions on the final exam were only available for a portion (40% or 63) of the students since courses are wiped clean from the learning management system after a year. In addition, the final exam questions (50) were randomly generated from a larger question bank (approximately 400 questions) making it difficult to have similar questions that students answered, to compare across semesters and years, in this current study. We also wanted to know all the factors that contributed to the final course grade (Research Question #4) and found the final exam, followed by WK7, WK6 and WK4 scores to be the highest predictors of a higher course grade, nearly 76% of the variance being explained. A limitation to the research is the level of multicollinearity that exists between the factors leading to a final course grade. There are correlations between factors as they are all part of the final course grade, but the statistics shows the relative importance of the factors very well in the Regression Analysis. The research indicates how the professor can shape the ODFs for higher learning outcomes and incorporate critical thinking questions into the final exam, thus impacting the overall course performance of the students. This study hints at the important role of the faculty member and brings about a deeper understanding of the course content as found by past researchers (Alzahrani, 2017; Carceller et al., 2013, 2015; Garrison and Arbaugh, 2007; Wu and Hiltz, 2004). Future research would do well to measure the impact of the faculty member’s feedback specifically, perhaps through TABLE 4 REGRESSION MODEL SUMMARY OF SIGNIFICANT FACTORS CONTRIBUTING TO FINAL COURSE GRADES Model R R Square Adjusted R Square Change Statistics R Square Change F Change df1 df2 Sig. F Change 1 .785a .617 .614 .617 252.517 1 157 .000 2 .846b .716 .712 .099 54.283 1 156 .000 3 .866c .750 .746 .035 21.600 1 155 .000 4 .873d .763 .756 .012 7.940 1 154 .005 Predictors: (Constant), Final Exama Predictors: (Constant), Final Exam, Wk7Discb Predictors: (Constant), Final Exam, Wk7Disc , Wk6Discc Predictors: (Constant), Final Exam, Wk7Disc , Wk6Disc , WK4Discd Dependent Variable: Final Gradee Page 143 - Developments in Business Simulation and Experiential Learning, Volume 46, 2019 student end of class surveys. Did the score students receive in their earlier weekly ODF assessments significantly influence the student’s interactions in subsequent weekly discussion? Again, this was not the purpose of this study, but we did see a stronger effect of WK 7 and WK 6 over WK 4 scores on the final course grade for Research Question #4, which would indicate iterative learning on the higher-level skills. Gilbert and Dabbagh (2005) suggested that evaluation rubrics positively influenced meaningful discourse and enabled course improvement in their research. Perhaps the use of a formal rubric could enhance subsequent performance in weekly ODFs and could be studied in future research on the topic. Another limitation of the research is the use of students as a sample due to internal and external validity issues. The students were not selected randomly but rather course sections were convenient samples, which threatens the internal validity or rigor of the sampling process. However, the longitudinal nature of the data (over 3 years of data) helps to ensure higher internal validity. The data is inclusive of all the operating courses (10 sections) taught by the same professor over the 3-year time period, which increases its reliability (consistency) and generalizability to other universities and student populations in similar courses. The students in these courses do not reside in one location but are located around the world, so cross-cultural communication issues and norms may affect the results as well. This is another future research stream to investigate. In ODFs, the assessment opportunities can be more frequent than in face-to-face classrooms as students can simultaneously communicate with each other and with the instructor. In this study, in all 10 courses there are seven ODF grades (i.e. number grade out of 20 possible points and comments), nearly every week (only 2 weeks of the 9-week term are used for the midterm and final exams so there are no ODFs those two weeks). The ODFs provide a greater opportunity for the instructor to provide faster feedback and individualized comments regarding the student’s work. The feedback and comments are usually captured in written or audio form and can be kept for future review by students. In addition, students have access to quick feedback from peers. The immediacy, permanency and richness of feedback in the online environment thus gives a greater opportunity to communicate with instructor and other students leading to enhanced student interactions. Future research could be conducted to tease out the type and timing of feedback loops in deeper learning. Also, the quantity and quality of ODF participation could be examined further for the impact on student’s in-depth learning. Students are not told that the ODF is the only way they can communicate with the professor or the other students in the class, so there may be additional opportunities to assess the faculty-student and student-student communications throughout the course in future research studies, such as via e-mail, recording teleconferences or Skype sessions or collection of data through other group collaboration tools. Learning critical thinking and problem-solving skills are often espoused as vital to student’s future success in computer science and applied sciences. This research gives evidence that higher-order skills can successfully be taught using an online discussion forum in a fully online operating systems course. Students who scored well in the weekly forums, that focused on higher thinking skills, were better able to answer questions on the final exam that measured these skills, performed better on the final exam and earned a higher course grade. Alghamdi, A. (2013). Pedagogical implications of using discussion board to improve student learning in higher education. Higher Education Studies, 3(5), 68-80. AlJeraisy, M. N., Mohammad, H., Fayyoumi, A., & Alrashideh, W. (2015). Web 2.0 in education: The impact of discussion board on student performance and satisfaction. The Turkish Online Journal of Educational Technology, 14(2), 247-259. Alzahrani, Majed Gharmallah (2017). Effect of using online discussion forums on students’ learning. The Turkish Online Journal of Educational Technology, 16(1), online. Anderson, T. (2004). Towards a theory of online learning. In T. Anderson, & F. Elloumi (Eds.), Theory and Practice of Online Learning (pp. 33-60), Edmonton, Canada: Athabasca University Press. Anderson, L.W. (Ed.) & Krathwohl, D.R. (Ed.) (2001). A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom’s Taxonomy of Educational Objectives (Complete edition). New York: Longman. Balaji, M. S., & Charkrabarti D, (2010). Student interactions in online discussion forum: Empirical research from ‘Media Richness Theory’ perspective”, IBS Hyderbad, Journal of Interactive Online Learning, 9(1), 1-22. Baeten, M., Dochy, F. & Struyven, Katrien (2008). Students’ approaches to learning and assessment preferences in a portfolio-based learning environment. Instructional Science, 36, 359-374. Biriyai, A. H. & Thomas, E.V. (2014). Online discussion forum: A tool for effective student teacher interaction. International Journal of Applied Science Research and Review, 1(3), 111-116. Bloom, B., Englehart, M. Furst, E., Hill, W., & Krathwohl, D. (1956). Taxonomy of educational objectives: The classification of educational goals. Handbook I: Cognitive domain. New York, Toronto: Longmans, Green. Brodie, C., Karat, C., & Feng, J. (2005). Usable security and privacy: A case study of developing privacy management tools. Pittsburgh, PA: Symposium on Usable Privacy and Security (SOUPS), July, 6-8. Carceller, C., Dawson, S., & Lockyer, L. (2013). Improving academic outcomes: Does participating in online discussion forums pay off. International Journal of Technology Enhanced Learning, 5(2), 117-132. Carceller, C., Dawson, S., & Lockyer, L. (2015). Social capital from online discussion forums: Differences between online and blended modes of delivery. Australasian Journal of Educational Technology, 31(2), 150-163. Campidoglio, M., Frattolillo, F., & Landolfi, F. (2009). The copyright protection problems: Challenges and suggestions. 2009 Fourth International conference on Internet and Web Applications and Services, 522-526. Cockburn, Alistair (2004). The end of software engineering and the start of economic-cooperative gaming, ComSIS, 1(1), 1-32. REFERENCES Page 144 - Developments in Business Simulation and Experiential Learning, Volume 46, 2019 Cormer, D. R. & Lenaghan, J.A. (2013). Enhancing discussion in the asynchronous online classroom. Journal of Management Education, 37(2), 261-294. Cox, B. & Cox, B. (2008), Developing interpersonal and group dynamics through asynchronous threaded discussion: The use of discussion board in collaborative learning. Education, 128(4), 553-565. Dennen, V. P., Darabi, A. A., & Smith, L. J. (2007). Instructor– learner interaction in online courses: The relative perceived importance of particular instructor actions on performance and satisfaction. Distance Education, 28(1), 65-79. Dolmans, D. (1992). The relationship between student- generated learning issues and self-study in problem- based learning. Instructional Science, 22(4), 251-267. Dube, L., Bourhis L, & Jacob R. (2006). Towards a typology of virtual communities of practice, Interdisciplinary Journal of Information, Knowledge and Management, 1, 69-93. Duch, Barbara J., Groh, Susan; & Allen, Deborah E. (2001). The Power of Problem-Based Learning: A Practical "How To" for Teaching Undergraduate Courses in Any Discipline (1st ed.). Sterling, VA: Stylus Publications. Edwards, J. (2005). A Short History of Online Discussion Groups. Harrisonburg, VA: James Madison University Libraries. Elfers, A. M., Plecki, M. L., Knapp, M. S., Yeo, G. J. & McGowan, M. L. (2007). Teaching Math in Washington’s High Schools: Insights from a Survey of Teachers in High Performing or Improving Schools. Seattle, WA: University of Washington. Ennis, R. H. (1985). A logical basis for measuring critical thinking skills. Educational Leadership, 43(2), 44–48. Fagin, B, Harper, J., Baird, L., Hadfield, S., & Sward, R. (2006). Critical thinking and computer science: implicit and explicit connections. Journal of Computing Sciences in Colleges, 21, 171-177. Fink, L. D. (2013). Creating Significant Learning Experiences: An Integrated Approach to Designing College Courses. eBook: Jossey-Bass Higher and Adult Education Series. Fogarty, R. (1997). Problem-Based Learning and Other Curriculum Models for the Multiple Intelligences Classroom, Arlington Heights, IL: IRI/SkyLight Training and Publishing. Harman, K., & Koohang, A. (2005). Discussion board: A learning object. Interdisciplinary Journal of Knowledge and Learning Objects, 1, 67-77. Hartnett, M. (2012). Relationships between online motivation, participation, and achievement: More complex than you might think. Journal of Open, Flexible, and Distance Learning, 16(1), 28-41. Hernandez, S. (2002). Team learning in a marketing principles course: Cooperative structures that facilitate active learning and higher-level thinking. Journal of Marketing Education, 24(1), 73-85. Hillman, W. (2003). Learning how to learn: Problem Based Learning. Australian Journal of Teacher Education, 28 (2), 1-10. Huitt, W. (2011). Bloom et al.'s taxonomy of the cognitive domain. Educational Psychology Interactive. Valdosta, GA: Valdosta State University. Retrieved October 1, 2018, from http://www.edpsycinteractive.org/topics/ cognition/bloom.html Jalongo, M.R., Twiest, M.M., & Gerlach, G.J. (1999). The college learner: Reading, studying, and attaining academic success. (2nd Ed.) New Jersey: Prentice- Hall. Kanuka, H., & Anderson, T. (1998). Online social interchange, discord, and knowledge construction. The Journal of Distance Learning, 13(1): 1-19. Kazemi, F., & Ghoraishi, M. (2012). A comparison of the Problem-based Learning Approach and traditional teaching on attitude, misconceptions and mathematics performance of university students. Procedia - Social and Behavioral Sciences, 46, 3852 – 3856. Kearsley, G., & Shneiderman, B. (1998). Engagement Theory: A framework for technology-based teaching and learning. Educational Technology, 38(5), 20-23. King, A. (1990). Enhancing peer interaction and learning in classroom through reciprocal questioning. American Educational Research Journal. 17(1). 25-43. Kingsland, A. J. (1996). Time expenditure, workload, and student satisfaction in problem‐based learning, New Directions for Teaching and Learning, August (68), 73 -81. Koole, M. L. (2009). A model for framing mobile learning. In M. Ally (Ed.), Mobile Learning: Transforming the Delivery of Education and Training (pp. 25-47). Edmonton, Canada: AU Press. Krathwohl, D. R. (2002). A revision of Bloom’s Taxonomy: An overview. Theory into Practice, 41(4), 212-218. LaPointe, L., & Reisetter, M. (2008), Belonging online: Students' perceptions of the value and efficacy of an online learning community. International Journal on E -Learning, 7(4), 641-665. Li-Jen, S., & Bennett, Judith (2012). A case study: Applying critical thinking skills to computer science and technology. Information Systems Education Journal (ISEDJ), 10(4), 41-48. Lim, C. P., & Tay, L. Y. (2003). Information and Communication Technologies (ICT): Students' engagement in higher order thinking. Journal of Educational Multimedia and Hypermedia, 12(4), 425- 451. Lomo-David, E., & Shannon, L.J. (2009). Information systems security and safety measures: The dichotomy between students’ familiarity and practice. Academy of Information and Management Sciences Journal, 12(1), 29-48. Marzano, R.J., & Kendall, J.S. (2007). The New Taxonomy of Educational Objectives. Thousand Oaks, CA: Corwin Press. Mazzolini, M., & Maddison, S. (2007). When to jump in: The role of the instructor in online discussion forums. Computers & Education, 49(2), 193-213. McDonnell, I. (2000). An electronic tutorial: A teaching innovation for tourism management, International Journal of Tourism Research, 2, 367-374. McDuffie, A. R., & Slavit, D. (2003). Utilizing online discussion to support reflection and challenge beliefs in elementary mathematics methods classrooms. Contemporary Issues in Technology and Teacher Education, 2(4), 446-466. Miliszewska, I. & Horwood, J. (2006). Engagement theory: A framework for supporting cultural differences in transnational education. ACM SIGCSE Bulletin, 38(1), 1-9. http://www.edpsycinteractive.org/topics/cognition/bloom.html http://www.edpsycinteractive.org/topics/cognition/bloom.html https://www.researchgate.net/profile/Iwona_Miliszewska?_sg=EO1Ly5bh9V99TniNxO-A90PtQKc2uZQ59JbR_ljRGb_aKJVQ3ZN3hSCVbRLabCTXCB-AjEQ.GANtZWpJVvNrveyVfrSwVI0iOdEo4FBWrKVqBihG0sLDTxu1pguV8hrJadQwg6gmFUHawJ4tRUSMCbPBE1jIqw Page 145 - Developments in Business Simulation and Experiential Learning, Volume 46, 2019 Menchaca, M. P., & Bekele, T. A. (2008). Learner and instructor identified success factors in distance education. Distance Education, 29(3), 231-252. Moon, J. A. (1999). Reflection in learning & professional development, Theory & Practice. London: Kogan Page Ltd. Newmann, F.M., & Wehlege, G. G. (1993). Five standards of authentic instruction. Association for Supervision and Curriculum Development, 50(7), 8-12. O’Neill, G. & Murphy, F. (2010). Guide to Taxonomies of Learning, Dublin. IL: University College Teaching and Learning/Resources. Palmer, S., & Holt, D. (2010). Online discussion in engineering education: Student responses and learning outcomes. In L. Shedletsky & J. E. Aitken (Eds.), Cases on Online Discussion and Interaction: Experiences and Outcomes (pp. 105-122). Hershey, PA: IGI Global. Palmer, S., Holt, D., & Bray, S. (2008). Does the discussion help? The impact of a formally assessed online discussion on final student results. British Journal of Educational Technology, 39(5), 847-858. Paul, R. (1990). Critical Thinking: What Every Person Needs to Survive in a Rapidly Changing World. Sonoma, CA: Center for Critical Thinking and Moral Critique. Pena-Shaff, J., Altman, W., & Stephenson, H. (2005). Asynchronous online discussions as a tool for learning: Students' attitudes, expectations, and perceptions. Journal of Interactive Learning Research, 16(4), 409- 430. Revill, G., and Terrell, I. (2005). Learning in the workplace: A new degree online. Innovations in Education & Teaching International, 42, 231-245. Rodrigues, S., (1999). Evaluation of an online masters course in science teacher education, Journal of Education for Teaching, 25(3), 263-270. Schiller, J., Ostwald, M., & Chen, S. (1994). Implementing a problem-based, distance education undergraduate course in construction management. Distance Education, 15(2), 300-317. Song, L., & McNary, S. W. (2011). Understanding students’ online interaction: Analysis of discussion board postings. Journal of Interactive Online Learning, 10 (1), 1-14. Struyven, K., Dochy, F., Janssens, S., & Gielen, S. (2006). On the dynamics of students' approaches to learning: The effects of the teaching/learning environment. Learning and Instruction, 16(4), 279-294. Swan, K., Fredericksen P., Pickett A. E., Pelz W., & Maher, G., (2000). Building knowledge building communities: Consistency, content and communication in the virtual classroom, Journal of Educational Computing Research, 23(4), 359-383. Thomas, M. J. W., (2002). Learning within incoherent structures: The space of online discussion forums. Journal of Computer Assisted Learning, 18(3): 351– 366. Tolmie, A., & Boyle, J. (2000). Factors influencing the success of Computer Mediated Communication (CMC) environments in university teaching: A review and case study. Computers & Education, 34(2), 119-140. Woo, H. L. & Wang, Q. (2009). Using weblog to promote critical thinking – An exploratory study. Proceedings of World Academy of Science, Engineering and Technology, January(37), 431-439. Wu, D., & Hiltz, S. R. (2004). Predicting learning from asynchronous online discussions. Journal of Asynchronous Learning Networks, 8(2), 139-152. Zhang, D., Zhou, L., Briggs, R. O., & Nunamaker, J. F. (2006). Instructional video in e-learning: Assessing the impact of interactive video on learning effectiveness. Information and Management, 43, 15-27. http://www.ucd.ie/t4cms/ucdtla0034.pdf http://www.ucd.ie/t4cms/ucdtla0034.pdf