Microsoft Word - 1. Stobaugh-Dec 2015.docx Vol. 4, No. 1, Dec 2015, pp. 3-18 Technology-Infused Instructional Framework: Improving Pre-service Teacher Lesson Planning Rebecca Stobaugh, Margaret Maxwell, Janet Tassell Western Kentucky University Abstract The focus of this research is to examine the impact of an instructional instrument to improve the quality of pre-service teachers’ lesson plans. The HEAT instrument focuses on four components essential to high-quality lesson plans: Higher-Order Thinking, Engaged Learning, Authentic Learning, and Technology. The research study examined a) data from elementary education classes for two semesters to measure the impact of the HEAT instrument on instructional planning during the semester and b) these pre-service teachers’ subsequent performance on the Teacher Work Sample compared to a control group of student teachers to measure the impact of the instrument on pre-service teacher performance. In the treatment group, pre-service teachers’ scores on the HEAT instrument were lower each successive semester of the study; however, during the student teaching semester the teacher candidates had higher scores on the Teacher Work Sample which measured the four components embedded in the HEAT instrument. Keywords: lesson plans; Bloom’s Taxonomy; teacher education; cognitive complexity; higher-order thinking; technology integration; authentic learning; engaged learning 4   R. Stobaugh, M. Maxwell & J. Tassell   With state teaching standards requiring technology integration and schools spending thousands of dollars on technologies, there are increasing pressures on teacher candidates to become more proficient in technology integration. However, oftentimes technologies are simply used in basic ways to present information or to capture students’ attention. Educators are looking for meaningful ways to integrate educational technology to advance student learning. Districts are concerned that technology is too expensive to not make an impact on learning, and teachers feel the pressure to use the technologies in meaningful and impactful ways. The International Society for Technology Education established standards for teachers (ISTE, 2008) and for students (ISTE, 2007). These standards promote students using technology to be creative, communicate, collaborate, and think critically. Similarly, the Partnership for 21st Century Skills promotes students working collaboratively to create media products while engaging in critical thinking. In the framework for 21st century teaching, the Partnership for 21st Century Skills (2009) stated that a learning experience should be one that “Enables innovative learning methods that integrate the use of supportive technologies, inquiry- and problem-based approaches and higher order thinking skills” (p. 8). These new expectations indicate higher student competence when using technology to collaborate with students on higher-cognitive learning tasks about real-world topics. This inclusive view of technology—one where technology integration is linked to higher-level thinking, authenticity, and engagement—is the basis of the HEAT instrument. The HEAT Instrument Moersch (2002) originally developed the HEAT instrument and the researchers have made revisions to clarify wording to teach lesson plan design and to assess the quality of lesson plans. The HEAT instrument includes components: Higher-Order Thinking, Engaged Learning, Authentic Learning, and Technology. Higher-Order Thinking The 21st-century learner is expected to be a problem solver and critical thinker. Anderson and Krathwohl (2001) define higher-order thinking as “the mental processes that allow students to develop factual, conceptual, and metacognitive knowledge within the creative and critical domains.” Bloom (1956) defined and quantified six levels of student thinking: Knowledge, Comprehension, Application, Analysis, Synthesis, and Evaluation levels. The model designates foundational knowledge (Knowledge and Comprehension) as important to developing higher levels of thinking (Application, Analysis, Synthesis, and Evaluation). Krathwohl (2002) better defined the cognitive processes within each level, expanding upon Bloom’s work. The updated levels include: Remembering, Understanding, Applying, Analyzing, Evaluating, and Creating (Krathwohl 2002). Liu Ru-De (2010) investigated the importance of leveraging higher-order thinking. Her study found students who were able to understand at a deep level were able to construct their own rationalizations, explanations, and extrapolations (Ru-De, 2010). In Marzano’s research about delivering high-quality teaching and learning in the 21st-century classroom, cognitive thinking skills were identified as a critical element (Marzano, 2010). The learning target or objective of a lesson can be raised to higher levels of cognitive thinking. As teachers raise the learning target of a particular lesson, it can be argued that instruction has improved. When objectives, activities, and assessments are properly aligned at higher levels of cognitive thinking, not only has instruction improved but also student learning improves (Raths, 2002). Technology-Infused Instructional Framework 5 Engaged Learning Today’s 21st century learners need to move to the center of learning process as they utilize new technological tools, search for credible information, make meaning, and create new products. This leads to a changing role of teachers as they coach and guide student learning. Prensky (2010) calls this partnering—a 21st century method of students and teachers working and collaborating together to produce and ensure student learning while preparing them for a new and different future. Instead of telling information, the teacher designs a variety of interesting, open-ended, thought-provoking questions or problems to be answered. Partnering teachers find that the process involving students more in the learning leads to higher engagement. Utilizing students’ passions and interests are the perfect routes and filters through which partnering teachers create individualized learning—learning that will stick in their minds, be valuable in their lives, and make them want more (Prensky, 2010). Bogaert, Pressley, and Hawkins (2006) collected artifacts from ten sixth grade classrooms and categorized teachers into either highly- engaging, moderately-engaging, or low-engaging. Results indicated the teachers with the most variety of instructional strategies coupled with providing support for student independence and choice were the most engaging. This higher level of engagement with choice and more student- directed tasks is expressed in the Level 3 and higher on the HEAT framework. Authentic Learning Splitter (2008) makes a case for educational authenticity being referenced from the earliest philosophical writings of Plato and Rousseau. Certo, Conley, Moxley, and Chafin (2008) reported students stated “un-authentic” assignments include completing worksheets or taking notes. Authentic learning has recently been clearly defined by Prensky (2010). Marc Prensky (2010) makes a sharp demarcation between relevance and real. Relevance means that students can relate to the learning to something they know. In other words, the context is familiar to the students. Prensky articulates that relevance is that it does not go far enough. Real means that there is a clear connection by the students between what they are learning and their ability to use that learning to do something useful in the world. Real learning connects content to current or future issues and also involves making an impact on those current issues or events. Making education real goes beyond teaching content just because it is required in the curriculum. Instruction and content should relate to the students’ world in a real (not just a theoretical) way. Further, learning should also involve changing and improving their world. Students need to relate math not to real-world or relevant math word problems but to “real” experiences actually taking place such as a bridge collapsing or being built (computing forces or stress), an election that’s taking place (probability, percentages, statistics), a space launch (trajectories, fuel consumption, rates of speed and acceleration), golf tournament (parabolas), baseball or football (statistics), or a song being recorded (timing, notes, compression, sampling rates (Prensky, 2010). Teachers can help students make real-world connections with workers, practitioners, and outside “experts” as possible. These community or field experts can serve as models, guide in research, and assist with problem solving (Prensky, 2010). Technology Little research exists on the combined impact of technology integration with higher-level thinking. While researchers continue to make the case for the positive correlation between higher-order thinking skills and integration of technology (Agnew 2002; Lee 2002; Thomas 2002), making the exclusive connection remains difficult (Sherry & Jesse 2000; Trucano 2005). 6   R. Stobaugh, M. Maxwell & J. Tassell   The International Society for Technology in Education (ISTE) published the National Education Technology Standards (NETS) for Students in 2007. These standards support the holistic view of Technology found for the HEAT instrument by calling attention to: creativity and innovation; communication and collaboration; research and information fluency; critical thinking, problem-solving, and decision-making; digital citizenship; and technology operation and concepts. These standards have pieces that are directly related to the HEAT instrument document. Related to Higher-Order Thinking, a connection to Standard 1, Creativity and Innovation, is that students “created original works as a means of personal or group expression.” Another is “apply existing knowledge to generate new ideas, products, or processes. Also related to Higher-Order Thinking, in Standard 3, Research and Information Fluency, a connection to the HEAT framework T column for Technology integration is “process data and report results.” Related to Engaged Learning, in Standard 2, Communication and Collaboration, a connection to the Technology integration on the instrument is made with “contribute to project teams to produce original works or solve problems.” Also, related to Engaged Learning, from Standard 5, Digital Citizenship, the HEAT document connects with “exhibit a positive attitude toward using technology that supports collaboration, learning and productivity.” Related to Authentic Learning, from Standard 4, Critical Thinking, Problem-Solving, and Decision-Making, three components relate to Technology component on the HEAT framework: “identify and define authentic problems and significant questions for investigation,” “plan and manage activities to develop a solution or complete a project,” and “collect and analyze data to identify solutions, and/or make informed decisions.” As a pure Technology integration connection, from Standard 6, Technology Operations and Concepts, the HEAT framework includes the essence of “select and use applications effectively and productively.” It appears that Standard 6 is what technology in the past would have encompassed. An interesting observation one can make from the consideration of the NETS for Students (2007) is that Standard 6 has been and is still readily the only standard being utilized and addressed in the classroom by many teachers at all levels. The hope is that the HEAT instrument can push the thinking of how Technology can bolster student learning through all areas of Higher-Order Thinking, Engaged Learning, and Authentic Learning (International Society for Technology in Education, 2007). HEAT Framework Levels The HEAT instrument incorporates six levels of performance for each component: Higher-Order thinking, Engaged Learning, Authentic Learning, and Technology (see Table 1 for HEAT Framework). The levels are as follows: 0 = Non-Use, 1 = Awareness, 2 = Application, 3 = Exploration, 4 = Integration, 5 = Expansion, and 6 = Refinement (see Table 1 for HEAT instrument). A target level of 3 or higher on the HEAT instrument was established because students are using higher-level thinking (Analyzing or higher), engaging in project-based learning with more choice in their projects, simulating the real world, and creating technology products even if they are an add-on to the lesson (see Table 2 for HEAT example). At higher levels of HEAT, students are more responsible for their own learning, beginning to think like experts, planning their own learning experiences while learning is embedded in the real world, and technology is seamlessly integrated and a necessary part of the learning experience. In the Technology component student use of technology is emphasized, not teacher use of technology. Several overarching themes began to influence the development of this HEAT Framework. One Technology-Infused Instructional Framework 7 significant theme is whether the instruction is teacher-directed or student-directed. This is the separating line between levels 3 and 4 through all components. As pre-service teachers are exposed to HEAT instrument and design lessons around these components, the hope is that teacher candidates should possess greater abilities to integrate technology in a research-based framework that will positively impact future lesson planning as they develop their Teacher Work Sample. Teacher Work Sample Through the Renaissance Partnership, a consortium of 11 universities designed the Teacher Work Sample (TWS). The TWS purpose was to measure the abilities of pre-service teachers along with assessing their ability to impact student learning (The Renaissance Partnership for Improving Teacher Quality, 2010). Through this performance-based assessment tool, pre-service teachers can demonstrate the ability to plan, deliver, and assess a standards- based instructional sequence, analyze student assessment, and reflect on their own instruction and student learning to improve teaching practice. The goal was that the TWS would promote higher expectations of teacher candidate performance during student teaching semester. The TWS has been adapted from the Western Oregon University Teacher Work Sample Methodology (Schalock, Schalock, & Girod, 1997). The original TWS assessed pre-service teacher preparation on seven critical teaching processes to improve P-12 instruction and student learning: Contextual Factors, Learning Goals, Assessment Plan, Design for Instruction, Instructional Decision-Making, Analysis of Student Learning, and Reflection and Self- Evaluation (Denner, Norman, Salzman, Pankratz, & Evans, 2004). After years of using the TWS as a culminating or capstone performance project, one university developed a task force to revise the TWS to better align with state teacher standards, improve the reliability of the assessment, and refine TWS sections. The revised TWS included five sections: • Contextual Factors: Determining the student and classroom context to design instruction • Learning Goals and Assessment Plan: Establishing learning goals and developing pre-, post-assessments to measure and learning results • Design for Instruction: Designing a plan for instruction for all students that addressed unit learning goals and were aligned with concepts and processes assessed • Analysis of Student Learning: Reporting and analyzing learning results for all students • Reflection and Self-Evaluation: Reflecting and evaluating on the teaching and learning within in the unit One noticeable refinement was in the Design for Instruction section. In the original TWS, for teacher candidates to score be proficient on the technology indicator within the Design for Instruction section, a pre-service teacher had to explain how they incorporated technology in their unit or offer a strong rationale for not using technology. With the increasing expectations that teacher candidates integrate technology, the TWS Task Force felt there needed to be higher expectations of technology use. In the revised TWS, for a Proficient rating in technology, the pre-service teacher must, “Demonstrate technology integration in planning and instruction and how P-12 student use of technology will be integrated in unit for higher level thinking activities and in a real-world context” (TWS, 2011, p. 16). The revised TWS requires several new elements for technology incorporation including: (a) student use of technology, (b) technology used for higher-level thinking tasks, and (c) technology embedded in real-world contexts. First, technology integration centers on students in the P-12 classroom using technology. In the 8   R. Stobaugh, M. Maxwell & J. Tassell   previous studies at the university, primarily pre-service teachers were using the technology to plan lessons and present information (Stobaugh, McDonald, & Tassell, 2010). For teacher candidates to be proficient, the P-12 students in the classrooms must be using the technology for learning. Another requirement in the revised TWS is that technology must be utilized to engage students in critical thinking. Technology should not simply be used as an attention-getting device, but rather as a way to enhance student learning. Finally, the indicator requires students to utilize the technology in a real-world context. The revised TWS aligns closely with the HEAT components of higher-level thinking, engagement, authentic connections, and technology integration. Purpose Through teaching pre-service teachers to utilize the HEAT instrument to design higher quality lessons, the authors believe they can positively impact pre-service teacher performance on the Teacher Work Sample where these pre-service teachers implement the lesson plans in teaching and assessing students. This research will provide data on whether this instrument can help better train teacher candidates to develop high-quality lessons. Research Questions This study examined the HEAT of lesson plans and the TWS created by pre-service teachers from two semesters in teacher education courses at a southeastern university. The research questions are as follows: 1. What percentage of lesson plans prepared by pre-service teachers score at a level 3 or higher in higher-order thinking activities? 2. What percentage of lesson plans prepared by pre-service teachers score at a level 3 or higher in engaged learning activities? 3. What percentage of lesson plans prepared by pre-service teachers score at a level 3 or higher in authentic learning? 4. What percentage of lesson plans prepared by pre-service teachers score at a level 3 or higher in technology activities? 5. Is there a significant difference in the TWS scores of the pre-service teachers who learned to plan lessons with the HEAT instrument and pre-service teachers who did not learn to plan lessons with the HEAT instrument? Methodology The authors believe that pre-service teachers can design lessons at a HEAT level of 3 or higher. This section will discuss the participants, instruments, procedures, and data analysis. Participants The participants for the HEAT group included pre-service teacher education students in two undergraduate elementary education methods courses in the Spring 2010 and Fall 2010 semesters. The students randomly selected themselves when they registered for sections of the methods courses with the researchers or sections with other instructors. The students had no prior knowledge of the HEAT Framework or that it would be used in these particular methods course sections. Technology-Infused Instructional Framework 9 The participants for the non-HEAT group included pre-service teacher education students in undergraduate elementary education methods courses in the Spring 2010 and Fall 2010 semesters that were taught by instructors other than the researchers. One course was ELED 465 Elementary Education Senior Project and the second course was ELED 405 Teaching Mathematics in the Elementary School. The participants in the ELED 405 and ELED 405 course were the same students during the respective semester and both courses were taught face-to-face. The participants in the non-HEAT group were located at different locations and taught via Instructional Television equipment. The numbers of students can be found in Table 1. Table 1 Participants in HEAT Study from Spring 2010 and Fall 2010 Group Course # Sections Spring 2010 # Sections Fall 2010 HEAT ELED 405 1 22 1 24 ELED 465 1 22 1 24 nonHEAT ELED 405 4 58 5 82 ELED 465 4 58 5 82 There were 144 students in ELED 490 Student Teaching in the Fall 2010 who completed the course and submitted a completed TWS. This included 21 students from the Spring 2010 courses that used the HEAT instrument in lesson planning and 123 students from other sections of ELED 405 and ELED 465 from previous semesters. There were 95 students in ELED 490 Student Teaching in the Spring 2011 who completed the course and submitted a completed TWS. This included 21 students from the Fall 2010 courses that used the HEAT instrument in lesson planning and 74 students from other sections of ELED 405 and ELED 465 from previous semesters. The number of students in the ELED 405 and ELED 465 is different from the number of students reported in the ELED 490 Student Teaching because students sometimes skip a semester between their prior coursework and student teaching for various reasons. The reasons may be placement issues, submission of paperwork by the due dates, or personal reasons. Some students received a grade of Incomplete and were not counted in the study. Only students who completed Student Teaching within the semester and submitted a final TWS were counted in the study. Instruments The HEAT Framework and the Teacher Word Sample instrument that are described above were used in this study. Procedures Teaching Procedures The participants in the courses that were introduced to the HEAT instrument were required to develop lesson plans as part of the typical course requirements. The pre-service teachers in ELED 465 area were required to design one lesson plan that embedded the HEAT components. In the ELED 405 course, the pre-service teachers were required to design a problem-solving lesson. In the classes, model examples of HEAT lessons at a level 3 were showcased, professors and peers provided formative feedback to initial HEAT lesson plan ideas, 10   R. Stobaugh, M. Maxwell & J. Tassell   and finally peers critiqued lessons prior to submission for additional feedback. A portion of the grade for the assignment was based on the students meeting all the 3 level HEAT indicators at a level 3. Teacher candidates were given a score and written comments. After the first year of introducing the HEAT instruction, the professor and research team met and discussed their successes and improvements needed, their understanding of the HEAT Framework, and ways to improve both the HEAT Framework and their instruction in the second year. The team did revise some wording within the HEAT Framework and instruction for the elementary methods courses were discussed. The same professors taught the courses the next year; therefore, there was consistency in the professors and course content. The participants in the other sections of ELED 405 and ELED 465 were taught the same content but were not introduced to the HEAT Framework for lesson planning. Scoring Procedures Forty-two pre-service teachers’ lesson plans from the Spring 2010 and 46 plans from the Fall 2010 semester from the two courses were assessed with the HEAT instrument. Student names were removed from the lesson plans, numbered, and randomly divided. Next, blind scoring was conducted by the researchers and trained scorers using the HEAT instrument. The scores were recorded in Excel spreadsheet files. In total, six evaluators rated the lessons – three were the researchers, two were Assistant Superintendents from local school districts (both were familiar with the HEAT instrument and used the instrument in their school districts), and one was an Instructional Designer (who had taught the HEAT instrument in graduate courses) at the same institution. The researchers trained the three other scorers on the use of the HEAT framework for the scoring of the lesson plans. A main focus of the training was on calibration of the scoring of the evaluators. To establish the calibration, the researchers chose four anchor lessons with agreed upon ratings, and trained and discussed these in detail for scoring calibration of the application of the instrument. The new members of the scoring team each scored the “training” lesson plans, shared and discussed their ratings for each of the four HEAT components. At this point in the study, the calibration goal was to score two consecutive lessons with HEAT component ratings no more than one level apart on each component from the score set by the researchers. After three training lessons, the calibration goal was met. After the calibration was established, three teams of scorers were paired together with one researcher in each of the pairs. The lesson plans were randomly distributed among the three scoring teams. A scoring team evaluated the same set of lessons – giving every lesson in the study two sets of scores. The ratings were recorded on spreadsheets. The scores were averaged when the scorers did not agree upon a score, which will explain why the ratings chart includes some scores containing “.5”. For example, scores of 2 and 3 would average to give a score of 2.5 on the Tables 2-6 as seen in the Results section. The TWS data were collected in Fall 2010 and Spring 2011, the subsequent semesters after the pre-service teachers completed the methods classes and were student teaching. All student teachers in the elementary program submitted a TWS at the end of the semester as their capstone project. Using analytic scoring rubrics developed by the Renaissance Partnership institutions, faculty score each component and associated indicators of the TWS based on a scale of 1 = Beginning, 2 = Developing, and 3 = Proficient, and 4 = Exemplary. The education program faculty deems a score of two on the four-point scale to be considered mastering the component or associated state teacher standard. For a Beginning rating, pre-service teachers would use minimal technology in their planning and instruction. To be classified as Developing, Technology-Infused Instructional Framework 11 pre-service teachers would utilize some technology in planning and instruction. A Proficient rating could be achieved if students demonstrate technology integration in planning and instruction with students utilizing technology for higher-level thinking tasks and in real world contexts. An Exemplary rating would indicate students demonstrated skills beyond Proficient with minimal assistance and on the first attempt. Professors of the student teacher class assess pre-service teachers’ abilities to meet the technology requirements as well as the other TWS indicators. Data Analysis Pre-service teachers that were taught to use the HEAT instrument in lesson planning were compared to other elementary education pre-service teachers in student teaching who took the same methods classes with other instructors, but were not trained to use the HEAT instrument in lesson planning. For research questions 1-4, frequencies and percentages were calculated for each component of HEAT. A T-test was used to determine if there was a significant difference in the TWS mean scores of the HEAT group and the non-HEAT group for research question 5. Results and Discussion Analysis of HEAT Levels of Pre-service Teacher Lesson Plans In this section, the results are provided where the lesson plans were double scored by the scoring team. When the double scores were not in agreement, the two scores were averaged together. The range of scores could span 0-6. With the possibility of the average between the two scorers, the increments increase by 0.5. Tables 1-4 depict the results of the data analysis for each of the HEAT components to be discussed in detail below, while Table 5 depicts the scores on the HEAT framework as a holistic score. Research Question 1. What percentage of lesson plans prepared by pre-service teachers score at a level 3 or higher in Higher-Order Thinking activities? Table 1: Frequencies and Percentages for Higher-Order Thinking Using the HEAT Instrument Average Rating Spring 2010 Fall 2010 N % N % 0 .5 1.0 2 5 5 10.0 1.5 1 2.1 2.0 13 31 16 34.7 2.5 20 43.4 3.0 27 64 4 8.7 3.5 4.0 Mean 2.6 2.1 N-Size 42 46 Ratings >=3 27 64 4 8.7 12   R. Stobaugh, M. Maxwell & J. Tassell   For the Higher-Order Thinking component, the pre-service teachers’ assignment was to plan instructional tasks with “students learning at an Analyzing, Evaluating, or Creating levels of Bloom’s Taxonomy” (Bloom, 1956) with “Teacher-directed questioning and instruction” at higher level thinking. The undergraduate frequency scores for Spring 2010 and Fall 2010 ranged from 1.0 and 3.0 with a target of 3.0. In the Spring 2010 semester mean scores were 2.6 with a decline to 2.1 in the Fall of 2010. Sixty-four percent of the students achieved a score of 3 in Spring 2010 compared to 8.7% in the Fall 2010 with a decrease of 55.3%. Research Question 2: What percentage of lesson plans prepared by pre-service teachers score at a level 3 or higher in Engaged Learning activities? Table 2: Frequencies and Percentages for Engaged Learning Using the HEAT Instrument Average Rating Spring 2010 Fall 2010 N % N % 0 .5 1.0 3 7 7 15.2 1.5 5 10.8 2.0 33 79 25 54.3 2.5 7 15.2 3.0 6 14 2 4.3 3.5 4.0 Mean 2.1 1.9 N-Size 42 46 Ratings >=3 6 14 2 4.3 For the Engaged Learning component, the pre-service teacher frequency scores for Spring 2010 and Fall 2010 ranged from 1.0 and 3.0 with a target of 3.0 for the Engaged Learning component of HEAT. To achieve a 3, the lesson plan had to include: “Student choice for projects or to solve a problem posed by teacher”; “Students are engaged in projects based on preferred learning styles, interests or passions”; “Used multiple instructional strategies.” In Spring 2010 the mean was 2.1 compared to the mean score of 1.9 for Fall 2010 semester. Only 14% of the pre-service teachers (N=6) achieved a score of 3 in Spring 2010 compared to 4.3% (N=2) in the Fall 2010 showing a decrease of 9.7%. Research Question 3: What percentage of lesson plans prepared by pre-service teachers score at a level 3 or higher in Authentic Learning? Technology-Infused Instructional Framework 13 Table 3: Frequencies and Percentages for Authentic Learning Using the HEAT Instrument Average Rating Spring 2010 Fall 2010 N % N % 0 1 2 .5 1.0 12 29 4 8.7 1.5 7 15.2 2.0 24 57 29 63.0 2.5 5 10.8 3.0 5 12 1 2.1 3.5 4.0 Mean 1.79 1.9 N-Size 42 46 Ratings >=3 5 12 1 2.1 The Authentic Learning component of HEAT measures if “Learning may be relevant to the real world or the past”; “Learning occurs in a simulated real-world situation such as a class store.” The pre-service frequency scores for Spring 2010 and Fall 2010 ranged from 1.0 and 3.0 with a target of 3.0. In the Spring 2010 semester the 2.0 frequency was the highest rating with 57.0% of the scores and 73.8% of the scores with a 2.0 or 2.5 rating. There was a noticeable decrease in the number of 1 scores with 29% (N= 12) in Spring 2010 to 8.7% (N=4) in Fall 2010. There were 12% (N=5) of the pre-service students achieving a 3 rating in the Spring 2010 and 2.1% (N=1) achieving a 3 rating in Fall 2010 showing a decrease 9.9%. Research Question 4: What percentage of lesson plans prepared by pre-service teachers score at a level 3 or higher in Technology activities? Table 4: Frequencies and Percentages for Technology Using the HEAT Instrument Average Rating Spring 2010 Fall 2010 N % N % 0 6 14 2 4.3 .5 1 2.1 1.0 20 48 11 23.9 1.5 2 4.3 2.0 7 17 14 30.4 2.5 9 19.5 3.0 9 21 7 15.2 3.5 4.0 Mean 1.45 1.8 N-Size 42 46 Ratings >=3 9 21 7 15.2 14   R. Stobaugh, M. Maxwell & J. Tassell   For the Technology component, to obtain a level 3 score, the lesson plan needed to include: "Technology use appears to be an add-on or alternative—not essential for task completion”; “Technology is used for higher-order thinking tasks such as analysis and decision- making." In Fall 2010 and Spring 2011, there was a large percentage of students that were below the target score of three. However, the level 3 scores were higher than any other component (except the Spring 2010 Higher level thinking) with 21% (N=9) and 15.2% (N=7) in the fall 2012. Only 15.2% achieved a 3 in Fall 2010 and 21.6% achieved a 3 or above in the Spring 2010 semester with a 5.8% decrease. Table 5. Percentage Scoring a Three or Above Average Rating Undergraduate Spring 2010 Fall 2010 % % H 64 8.7 E 14 4.3 A 12 2.1 T 21 15.2 For each of the HEAT components there was a decrease in HEAT scores from Spring to Fall 2010. There was a 55.3% decrease in the higher-level thinking component. The lowest decrease was for technology component (5.8%). Analysis of Teacher Work Sample Mean Scores of HEAT and non-HEAT Groups Research Question 5: Is there a significant difference in the TWS scores of the pre-service teachers who learned to plan lessons with the HEAT instrument and pre-service teachers who did not learn to plan lessons with the HEAT instrument? Table 6. Ratings on the Teacher Work Sample by Pre-Service Teachers from the HEAT Group and the non-HEAT Group Average Rating HEAT Non-HEAT HEAT Non-HEAT Fall 2010 Spring 2011 N % N % N % N % Beginning 0 0 0 0 0 0 0 0 Developing 7 33.3 32 26.0 2 9.5 9 12.2 Proficient 9 42.9 67 54.5 7 33.3 54 72.9 Exemplary 5 23.8 24 19.5 12 57.1 11 14.9 TOTAL 21 100 123 100 21 99.9 74 100 Twenty-one elementary education students who were used the HEAT instrument in the Spring 2010 semester completed the TWS in the Fall 2010 semester. Another 21 students who used the HEAT framework in the Fall 2012 semester completed the TWS in the Spring 2011 semester. For the control group of elementary education pre-service teachers who were not exposed to the HEAT instrument, 123 completed the TWS in Fall 2010 and 74 in Spring 2011. Technology-Infused Instructional Framework 15 In both semesters, all elementary education pre-service scores ranged between the Developing, Proficient, and Exemplary categories. In Fall 2010, the number of pre-service teachers instructed with the HEAT Framework had 23.8% (N=5) score at an Exemplary rating compared to the 19.5% (N=24) in the control group who had not been instructed with the HEAT Framework. However, in the Spring 2011, the HEAT group had 57.1% (N=12) compared to 14.9% (N=11) of the non-HEAT group scoring Exemplary. That is a 42.2% difference between the HEAT and the control group’s percentage scoring Exemplary. This group was the second group exposed to the HEAT Framework. Table 7. Mean TWS Data from Fall 2010 and Spring 2011 Group Fall 2010 Spring 2011 HEAT 2.9048 3.4762 Non-HEAT 2.9350 3.0270 Table 8. T-test Using Combined Data from Fall 2010 and Spring 2011 TWS Group N Mean SD Non-HEAT 197 2.9695 0.6220 HEAT 42 3.1905 0.7726 T-value -2.00; Probability 0.0468 The T-test was used to determine significant differences between the HEAT and non- HEAT groups because this statistical test can account for different sample sizes. The data from the two semesters were combined for this calculation for two reasons: the different sample sizes in the groups and to combine the effects of the HEAT instruction over two semesters. The purpose was not to test the differences between the two semesters but to test the differences between the two groups over the 2010-2011 academic year. There was a significantly higher score for the HEAT group, t(239) = -2.00, p <0.05, on the Teacher Work Sample than the non- HEAT group (see Table 8). This indicates that pre-service teachers who used the HEAT instrument for lesson planning performed better on the Teacher Work Sample than pre-service teachers who were not exposed to the HEAT instrument for lesson planning. Conclusions The four components of the HEAT instrument provide a means for teachers to conceptualize how to improve the instructional quality of their lessons. When teachers design lessons that “raise the HEAT,” instruction is engaging, cognitively complex, authentic, and embedded with technology. The combination of the HEAT elements ensure that technology is integrated in a research-based method to increase the quality of lessons that meet 21st century expectations. The Higher-Order Thinking component was the highest scoring component (64%) in the Spring 2010 semester with the Technology component as the second highest (21%). The payoff is reflected in the increased percentages of pre-service teachers who scored Exemplary (57.1%) on the TWS the next semester compared to 14.9% of the other pre-service teachers who were not exposed to the HEAT instrument. The HEAT implementation in methods courses significantly improved the pre-service teachers’ performance on the TWS. Therefore, we can conclude that 16   R. Stobaugh, M. Maxwell & J. Tassell   when pre-service teachers use the HEAT instrument to design higher quality lessons, it does impact their lesson design and performance; i.e., implementation or teaching the lesson. This increase in quality lessons and instruction for students can positively impact K-12 student learning. Implications for Future Research A version of the HEAT instrument will continue to be used to score pre-service and advanced teachers’ lesson plans in our coursework. The authors will continue to refine the instrument and collaborate on how to use the HEAT framework in pre-service and advanced teacher courses. If pre-service and advanced teachers are consistently designing instruction at a level 3 or higher on a framework such as this, the goal is to prove that these high quality lessons can positively impact K-12 student learning. The Teacher Work Sample capstone project has provided this research study with an end product study opportunity that is quite opportune to capture and consider. The HEAT instruction for the lesson plan application for the pre-service teachers takes place typically in the semester immediately before the TWS semester. What would then be interesting for future study is to follow these same participants as they progress into the field as teachers and/or graduate students. As teachers, it would be of utmost interest to continue the study as to whether or not HEAT framework lesson plan design is continues to impact their ability to design high-quality lessons. However, a larger concern is whether or not the HEAT lesson is actually being implemented or taught at the level that the lesson written or designed. In other words, oftentimes what is seen in the research during the lesson plan instruction semester, pre-service teachers have the opportunity to deliver or teach the lesson. What is observed happens to be at a lower level of cognitive complexity in many cases than the lesson was designed. When the lesson is not taught with appropriate rigor, students are not learning to their potential. Another research concern is to examine how pre-service and practicing teachers are designing the assessments for their students to exhibit their learning in regard to their HEAT lesson. What is initially found in the lesson plans tends to be of good quality. However, when the lesson plan designer is asked to create a model student product and score it with their rubric, many issues surface regarding their depth of understanding of assessment. A misalignment of objectives, instruction, and assessment is observed when the pre-service teachers design this model student product and attempt to score it with their rubric for the lesson. There appears to be a disconnect between the quality and expectations in the lesson plan in comparison to the product that the students are to complete to show their learning from the lesson. Another angle to achieve this research is to require the pre-service teachers to submit a student assessment product that is directly from their HEAT Lesson Plan and develop a scoring plan. The hope in the entire conversation regarding assessment is that pre-service teachers will better see the lesson as a complete cycle with the assessment included. Technology-Infused Instructional Framework 17 References Anderson, L.W., & Krathwohl, D.R. (2001). A taxonomy for teaching, learning and assessing: A revision of Bloom’s taxonomy of educational objectives. New York: Addison Wesley. Bloom, B. S. (1956). Taxonomy of educational objectives. Philadelphia: David McKay Publishing Company. Bloom, B.S., Englehart, M.D., Furst, E.J., Hill, W.H., & Krathwohl, D.R. (1956). Taxonomy of educational objectives: The classification of educational goals. Handbook 1: Cognitive domain. New York: David McKay. Dener, P. R., Norman, A. D., Salzman, S. A., Pankratz, R., & Evans, S.E. (2004). The Renaissance Partnership Teacher Work Sample: Evidence supporting score generalizability, validity, and quality of student learning assessment. In E.M. Guyton, & J. R. Dangel (Eds.), Research linking teacher preparation and student performance, (pp. 23-55). Dubuque, IA: Kendall/Hunt Publishing Company. Driscoll, M. (2000). Psychology of learning for instruction. Needham Heights, MA: Allyn & Bacon. Herrington, J., & Oliver, R. (2000). An instructional design framework for authentic learning environments. Educational Technology Research & Development, 48(3), 23-48. International Society for Technology in Education (2007). The national educational technology standards for students. Eugene, OR: International Society for Technology in Education. Retrieved from http://www.iste.org/standards/nets-for-students.aspx International Society for Technology in Education (2008). The national educational technology standards for teachers. Eugene, OR: International Society for Technology in Education. Retrieved from http://www.iste.org/standards/nets-for-teachers.aspx 18   R. Stobaugh, M. Maxwell & J. Tassell   Jones, B., Valdez, G., Nowakowski, J., & Rasmussen, C. (1994). Designing learning and technology for educational reform. Oak Brook, IL: North Central Regional Educational Laboratory. Marzano, R. J. (2010). Teaching inference. Educational Leadership, 67(7), 80-81. Moersch, C. (2002). Measures of success: Six instruments to assess teachers’ use of technology. Learning & Leading with Technology, 30(3), 10-18. Partnership for 21st Century Skills (2009). P-12 Framework Definitions. Retrieved from http://www.p21.org/storage/documents/P21_Framework_Definitions.pdf The Renaissance Partnership for Improving Teacher Quality. (2010). Teacher Work Sample: performance prompt, teaching process standards, scoring rubrics. Retrieved from http://www.uni.edu/itq/RTWS/. Schalock, H. D., Schalock, M., & Girod, G. (1997). Teacher Work Sample methodology as used at Western Oregon State College. In J. McMillan (Ed.), Grading teachers, grading schools: Is student achievement a valid evaluation measure? (pp.15-45). Thousand Oaks, CA: Corwin Press. School of Teacher Education. (2011). Teacher Work Sample. Bowling Green, KY: Western Kentucky University. Siemens, G. (2004). Connectivism: A learning theory for the digital age. Retrieved from http://www.elearnspace.org/Articles/connectivism.htm Stobaugh, R.S., McDonald, M. L., & Tassell, J.L. (2010). Student teacher use of technology to facilitate teaching and learning. In C. Maddux (Ed.), Research highlights in technology and teacher education 2010, (pp. 43-52). Chesapeake, VA: Society for Information Technology & Teacher Education (SITE).