
































Microsoft Word - 1. Stobaugh-Dec 2015.docx


Vol. 4, No. 1, Dec 2015, pp. 3-18   
 
 
 
 
Technology-Infused Instructional Framework:  Improving Pre-service Teacher Lesson 
Planning 
Rebecca Stobaugh, Margaret Maxwell, Janet Tassell 
Western Kentucky University 
 
 
 
 
 

Abstract 
 

The focus of this research is to examine the impact of an instructional instrument to 

improve the quality of pre-service teachers’ lesson plans.  The HEAT instrument focuses on four 

components essential to high-quality lesson plans:  Higher-Order Thinking, Engaged Learning, 

Authentic Learning, and Technology.  The research study examined a) data from elementary 

education classes for two semesters to measure the impact of the HEAT instrument on 

instructional planning during the semester and b) these pre-service teachers’ subsequent 

performance on the Teacher Work Sample compared to a control group of student teachers to 

measure the impact of the instrument on pre-service teacher performance.  In the treatment 

group, pre-service teachers’ scores on the HEAT instrument were lower each successive 

semester of the study; however, during the student teaching semester the teacher candidates had 

higher scores on the Teacher Work Sample which measured the four components embedded in 

the HEAT instrument.   

Keywords:  lesson plans; Bloom’s Taxonomy; teacher education; cognitive complexity; 

higher-order thinking; technology integration; authentic learning; engaged learning 

  



4	
    R. Stobaugh, M. Maxwell & J. Tassell	
  
 

 
With state teaching standards requiring technology integration and schools spending 

thousands of dollars on technologies, there are increasing pressures on teacher candidates to 
become more proficient in technology integration.  However, oftentimes technologies are simply 
used in basic ways to present information or to capture students’ attention.  Educators are 
looking for meaningful ways to integrate educational technology to advance student learning.  
Districts are concerned that technology is too expensive to not make an impact on learning, and 
teachers feel the pressure to use the technologies in meaningful and impactful ways.    

The International Society for Technology Education established standards for teachers 
(ISTE, 2008) and for students (ISTE, 2007).  These standards promote students using technology 
to be creative, communicate, collaborate, and think critically. Similarly, the Partnership for 21st 
Century Skills promotes students working collaboratively to create media products while 
engaging in critical thinking.  In the framework for 21st century teaching, the Partnership for 21st  

Century Skills (2009) stated that a learning experience should be one that “Enables innovative 
learning methods that integrate the use of supportive technologies, inquiry- and problem-based 
approaches and higher order thinking skills” (p. 8).  These new expectations indicate higher 
student competence when using technology to collaborate with students on higher-cognitive 
learning tasks about real-world topics.  This inclusive view of technology—one where 
technology integration is linked to higher-level thinking, authenticity, and engagement—is the 
basis of the HEAT instrument.  

The HEAT Instrument 
Moersch (2002) originally developed the HEAT instrument and the researchers have 

made revisions to clarify wording to teach lesson plan design and to assess the quality of lesson 
plans.  The HEAT instrument includes components:  Higher-Order Thinking, Engaged Learning, 
Authentic Learning, and Technology.  
 
Higher-Order Thinking 

The 21st-century learner is expected to be a problem solver and critical thinker.  
Anderson and Krathwohl (2001) define higher-order thinking as “the mental processes that allow 
students to develop factual, conceptual, and metacognitive knowledge within the creative and 
critical domains.”  Bloom (1956) defined and quantified six levels of student thinking:   
Knowledge, Comprehension, Application, Analysis, Synthesis, and Evaluation levels.  The 
model designates foundational knowledge (Knowledge and Comprehension) as important to 
developing higher levels of thinking (Application, Analysis, Synthesis, and Evaluation).  
Krathwohl (2002) better defined the cognitive processes within each level, expanding upon 
Bloom’s work.  The updated levels include: Remembering, Understanding, Applying, 
Analyzing, Evaluating, and Creating (Krathwohl 2002).  

Liu Ru-De (2010) investigated the importance of leveraging higher-order thinking.  Her 
study found students who were able to understand at a deep level were able to construct their 
own rationalizations, explanations, and extrapolations (Ru-De, 2010).  In Marzano’s research 
about delivering high-quality teaching and learning in the 21st-century classroom, cognitive 
thinking skills were identified as a critical element (Marzano, 2010).  The learning target or 
objective of a lesson can be raised to higher levels of cognitive thinking.  As teachers raise the 
learning target of a particular lesson, it can be argued that instruction has improved.  When 
objectives, activities, and assessments are properly aligned at higher levels of cognitive thinking, 
not only has instruction improved but also student learning improves (Raths, 2002). 



 Technology-Infused Instructional Framework    5 
 

Engaged Learning 
Today’s 21st century learners need to move to the center of learning process as they 

utilize new technological tools, search for credible information, make meaning, and create new 
products.  This leads to a changing role of teachers as they coach and guide student learning. 
Prensky (2010) calls this partnering—a 21st century method of students and teachers working and 
collaborating together to produce and ensure student learning while preparing them for a new 
and different future.  Instead of telling information, the teacher designs a variety of interesting, 
open-ended, thought-provoking questions or problems to be answered.  Partnering teachers find 
that the process involving students more in the learning leads to higher engagement.  Utilizing 
students’ passions and interests are the perfect routes and filters through which partnering 
teachers create individualized learning—learning that will stick in their minds, be valuable in 
their lives, and make them want more (Prensky, 2010).  Bogaert, Pressley, and Hawkins (2006) 
collected artifacts from ten sixth grade classrooms and categorized teachers into either highly-
engaging, moderately-engaging, or low-engaging.  Results indicated the teachers with the most 
variety of instructional strategies coupled with providing support for student independence and 
choice were the most engaging.  This higher level of engagement with choice and more student-
directed tasks is expressed in the Level 3 and higher on the HEAT framework. 
 
Authentic Learning 

Splitter (2008) makes a case for educational authenticity being referenced from the 
earliest philosophical writings of Plato and Rousseau.  Certo, Conley, Moxley, and Chafin 
(2008) reported students stated “un-authentic” assignments include completing worksheets or 
taking notes.  Authentic learning has recently been clearly defined by Prensky (2010).  Marc 
Prensky (2010) makes a sharp demarcation between relevance and real.  Relevance means that 
students can relate to the learning to something they know.  In other words, the context is 
familiar to the students.  Prensky articulates that relevance is that it does not go far enough.  Real 
means that there is a clear connection by the students between what they are learning and their 
ability to use that learning to do something useful in the world.  Real learning connects content to 
current or future issues and also involves making an impact on those current issues or events. 
 Making education real goes beyond teaching content just because it is required in the 
curriculum.  Instruction and content should relate to the students’ world in a real (not just a 
theoretical) way.  Further, learning should also involve changing and improving their world. 
Students need to relate math not to real-world or relevant math word problems but to “real” 
experiences actually taking place such as a bridge collapsing or being built (computing forces or 
stress), an election that’s taking place (probability, percentages, statistics), a space launch 
(trajectories, fuel consumption, rates of speed and acceleration), golf tournament (parabolas), 
baseball or football (statistics), or a song being recorded (timing, notes, compression, sampling 
rates (Prensky, 2010).  Teachers can help students make real-world connections with workers, 
practitioners, and outside “experts” as possible.  These community or field experts can serve as 
models, guide in research, and assist with problem solving (Prensky, 2010).  
 
Technology  

Little research exists on the combined impact of technology integration with higher-level 
thinking.  While researchers continue to make the case for the positive correlation between 
higher-order thinking skills and integration of technology (Agnew 2002; Lee 2002; Thomas 
2002), making the exclusive connection remains difficult (Sherry & Jesse 2000; Trucano 2005).  



6	
    R. Stobaugh, M. Maxwell & J. Tassell	
  
 

The International Society for Technology in Education (ISTE) published the National 
Education Technology Standards (NETS) for Students in 2007. These standards support the 
holistic view of Technology found for the HEAT instrument by calling attention to: creativity 
and innovation; communication and collaboration; research and information fluency; critical 
thinking, problem-solving, and decision-making; digital citizenship; and technology operation 
and concepts.  These standards have pieces that are directly related to the HEAT instrument 
document.  Related to Higher-Order Thinking, a connection to Standard 1, Creativity and 
Innovation, is that students “created original works as a means of personal or group expression.”  
Another is “apply existing knowledge to generate new ideas, products, or processes.  Also 
related to Higher-Order Thinking, in Standard 3, Research and Information Fluency, a 
connection to the HEAT framework T column for Technology integration is “process data and 
report results.”  Related to Engaged Learning, in Standard 2, Communication and Collaboration, 
a connection to the Technology integration on the instrument is made with “contribute to project 
teams to produce original works or solve problems.”  Also, related to Engaged Learning, from 
Standard 5, Digital Citizenship, the HEAT document connects with “exhibit a positive attitude 
toward using technology that supports collaboration, learning and productivity.”  Related to 
Authentic Learning, from Standard 4, Critical Thinking, Problem-Solving, and Decision-Making, 
three components relate to Technology component on the HEAT framework: “identify and 
define authentic problems and significant questions for investigation,” “plan and manage 
activities to develop a solution or complete a project,” and “collect and analyze data to identify 
solutions, and/or make informed decisions.” As a pure Technology integration connection, from 
Standard 6, Technology Operations and Concepts, the HEAT framework includes the essence of 
“select and use applications effectively and productively.” It appears that Standard 6 is what 
technology in the past would have encompassed. 

An interesting observation one can make from the consideration of the NETS for 
Students (2007) is that Standard 6 has been and is still readily the only standard being utilized 
and addressed in the classroom by many teachers at all levels. The hope is that the HEAT 
instrument can push the thinking of how Technology can bolster student learning through all 
areas of Higher-Order Thinking, Engaged Learning, and Authentic Learning (International 
Society for Technology in Education, 2007). 
 
HEAT Framework Levels 

The HEAT instrument incorporates six levels of performance for each component: 
Higher-Order thinking, Engaged Learning, Authentic Learning, and Technology (see Table 1 for 
HEAT Framework). The levels are as follows: 0 = Non-Use, 1 = Awareness, 2 = Application, 3 
= Exploration, 4 = Integration, 5 = Expansion, and 6 = Refinement (see Table 1 for HEAT 
instrument).  A target level of 3 or higher on the HEAT instrument was established because 
students are using higher-level thinking (Analyzing or higher), engaging in project-based 
learning with more choice in their projects, simulating the real world, and creating technology 
products even if they are an add-on to the lesson (see Table 2 for HEAT example).  At higher 
levels of HEAT, students are more responsible for their own learning, beginning to think like 
experts, planning their own learning experiences while learning is embedded in the real world, 
and technology is seamlessly integrated and a necessary part of the learning experience.  In the 
Technology component student use of technology is emphasized, not teacher use of technology.  
Several overarching themes began to influence the development of this HEAT Framework.  One 



 Technology-Infused Instructional Framework    7 
 

significant theme is whether the instruction is teacher-directed or student-directed. This is the 
separating line between levels 3 and 4 through all components.  
 As pre-service teachers are exposed to HEAT instrument and design lessons around these 
components, the hope is that teacher candidates should possess greater abilities to integrate 
technology in a research-based framework that will positively impact future lesson planning as 
they develop their Teacher Work Sample. 
 

Teacher Work Sample 
 Through the Renaissance Partnership, a consortium of 11 universities designed the 
Teacher Work Sample (TWS).  The TWS purpose was to measure the abilities of pre-service 
teachers along with assessing their ability to impact student learning (The Renaissance 
Partnership for Improving Teacher Quality, 2010).  Through this performance-based assessment 
tool, pre-service teachers can demonstrate the ability to plan, deliver, and assess a standards-
based instructional sequence, analyze student assessment, and reflect on their own instruction 
and student learning to improve teaching practice.  The goal was that the TWS would promote 
higher expectations of teacher candidate performance during student teaching semester.  The 
TWS has been adapted from the Western Oregon University Teacher Work Sample 
Methodology (Schalock, Schalock, & Girod, 1997).  The original TWS assessed pre-service 
teacher preparation on seven critical teaching processes to improve P-12 instruction and student 
learning: Contextual Factors, Learning Goals, Assessment Plan, Design for Instruction, 
Instructional Decision-Making, Analysis of Student Learning, and Reflection and Self-
Evaluation (Denner, Norman, Salzman, Pankratz, & Evans, 2004). 

After years of using the TWS as a culminating or capstone performance project, one 
university developed a task force to revise the TWS to better align with state teacher standards, 
improve the reliability of the assessment, and refine TWS sections.  The revised TWS included 
five sections: 

• Contextual Factors:  Determining the student and classroom context to design instruction  
• Learning Goals and Assessment Plan:  Establishing learning goals and developing pre-, 

post-assessments to measure and learning results  
• Design for Instruction:  Designing a plan for instruction for all students that addressed 

unit learning goals and were aligned with concepts and processes assessed  
• Analysis of Student Learning:  Reporting and analyzing learning results for all students 
• Reflection and Self-Evaluation:  Reflecting and evaluating on the teaching and learning 

within in the unit 
 One noticeable refinement was in the Design for Instruction section.  In the original 
TWS, for teacher candidates to score be proficient on the technology indicator within the Design 
for Instruction section, a pre-service teacher had to explain how they incorporated technology in 
their unit or offer a strong rationale for not using technology.  With the increasing expectations 
that teacher candidates integrate technology, the TWS Task Force felt there needed to be higher 
expectations of technology use.  In the revised TWS, for a Proficient rating in technology, the 
pre-service teacher must, “Demonstrate technology integration in planning and instruction and 
how P-12 student use of technology will be integrated in unit for higher level thinking activities 
and in a real-world context” (TWS, 2011, p. 16).  The revised TWS requires several new 
elements for technology incorporation including:  (a) student use of technology, (b) technology 
used for higher-level thinking tasks, and (c) technology embedded in real-world contexts.  First, 
technology integration centers on students in the P-12 classroom using technology.  In the 



8	
    R. Stobaugh, M. Maxwell & J. Tassell	
  
 

previous studies at the university, primarily pre-service teachers were using the technology to 
plan lessons and present information (Stobaugh, McDonald, & Tassell, 2010).  For teacher 
candidates to be proficient, the P-12 students in the classrooms must be using the technology for 
learning.  Another requirement in the revised TWS is that technology must be utilized to engage 
students in critical thinking.  Technology should not simply be used as an attention-getting 
device, but rather as a way to enhance student learning.  Finally, the indicator requires students 
to utilize the technology in a real-world context.  The revised TWS aligns closely with the HEAT 
components of higher-level thinking, engagement, authentic connections, and technology 
integration. 

 
Purpose 

 Through teaching pre-service teachers to utilize the HEAT instrument to design higher 
quality lessons, the authors believe they can positively impact pre-service teacher performance 
on the Teacher Work Sample where these pre-service teachers implement the lesson plans in 
teaching and assessing students.  This research will provide data on whether this instrument can 
help better train teacher candidates to develop high-quality lessons. 
 
Research Questions 
  This study examined the HEAT of lesson plans and the TWS created by pre-service 
teachers from two semesters in teacher education courses at a southeastern university. The 
research questions are as follows:  

1. What percentage of lesson plans prepared by pre-service teachers score at a level 3 or 
higher in higher-order thinking activities?  

2. What percentage of lesson plans prepared by pre-service teachers score at a level 3 or 
higher in engaged learning activities? 

3. What percentage of lesson plans prepared by pre-service teachers score at a level 3 or 
higher in authentic learning? 

4. What percentage of lesson plans prepared by pre-service teachers score at a level 3 or 
higher in technology activities?  

5. Is there a significant difference in the TWS scores of the pre-service teachers who learned 
to plan lessons with the HEAT instrument and pre-service teachers who did not learn to 
plan lessons with the HEAT instrument? 

 
Methodology 

The authors believe that pre-service teachers can design lessons at a HEAT level of 3 or 
higher.  This section will discuss the participants, instruments, procedures, and data analysis. 

Participants 

The participants for the HEAT group included pre-service teacher education students in 
two undergraduate elementary education methods courses in the Spring 2010 and Fall 2010 
semesters. The students randomly selected themselves when they registered for sections of the 
methods courses with the researchers or sections with other instructors. The students had no prior 
knowledge of the HEAT Framework or that it would be used in these particular methods course 
sections.   



 Technology-Infused Instructional Framework    9 
 

The participants for the non-HEAT group included pre-service teacher education students 
in undergraduate elementary education methods courses in the Spring 2010 and Fall 2010 
semesters that were taught by instructors other than the researchers. 

One course was ELED 465 Elementary Education Senior Project and the second course 
was ELED 405 Teaching Mathematics in the Elementary School. The participants in the ELED 
405 and ELED 405 course were the same students during the respective semester and both 
courses were taught face-to-face. The participants in the non-HEAT group were located at 
different locations and taught via Instructional Television equipment. The numbers of students 
can be found in Table 1.  

Table 1 Participants in HEAT Study from Spring 2010 and Fall 2010 

Group Course # Sections Spring 2010 # Sections Fall 2010 
HEAT ELED 405 1 22 1 24 

 
ELED 465 1 22 1 24 

nonHEAT ELED 405 4 58 5 82 

 
ELED 465 4 58 5 82 

 

There were 144 students in ELED 490 Student Teaching in the Fall 2010 who completed 
the course and submitted a completed TWS.  This included 21 students from the Spring 2010 
courses that used the HEAT instrument in lesson planning and 123 students from other sections 
of ELED 405 and ELED 465 from previous semesters.  There were 95 students in ELED 490 
Student Teaching in the Spring 2011 who completed the course and submitted a completed 
TWS.  This included 21 students from the Fall 2010 courses that used the HEAT instrument in 
lesson planning and 74 students from other sections of ELED 405 and ELED 465 from previous 
semesters.  The number of students in the ELED 405 and ELED 465 is different from the number 
of students reported in the ELED 490 Student Teaching because students sometimes skip a 
semester between their prior coursework and student teaching for various reasons.  The reasons 
may be placement issues, submission of paperwork by the due dates, or personal reasons.  Some 
students received a grade of Incomplete and were not counted in the study.  Only students who 
completed Student Teaching within the semester and submitted a final TWS were counted in the 
study. 

Instruments 

The HEAT Framework and the Teacher Word Sample instrument that are described 
above were used in this study.  

Procedures 

Teaching Procedures 
The participants in the courses that were introduced to the HEAT instrument were 

required to develop lesson plans as part of the typical course requirements.  The pre-service 
teachers in ELED 465 area were required to design one lesson plan that embedded the HEAT 
components.  In the ELED 405 course, the pre-service teachers were required to design a 
problem-solving lesson.  In the classes, model examples of HEAT lessons at a level 3 were 
showcased, professors and peers provided formative feedback to initial HEAT lesson plan ideas, 



10	
    R. Stobaugh, M. Maxwell & J. Tassell	
  
 

and finally peers critiqued lessons prior to submission for additional feedback.  A portion of the 
grade for the assignment was based on the students meeting all the 3 level HEAT indicators at a 
level 3.  Teacher candidates were given a score and written comments.  After the first year of 
introducing the HEAT instruction, the professor and research team met and discussed their 
successes and improvements needed, their understanding of the HEAT Framework, and ways to 
improve both the HEAT Framework and their instruction in the second year.  The team did 
revise some wording within the HEAT Framework and instruction for the elementary methods 
courses were discussed.  The same professors taught the courses the next year; therefore, there 
was consistency in the professors and course content.  The participants in the other sections of 
ELED 405 and ELED 465 were taught the same content but were not introduced to the HEAT 
Framework for lesson planning.  

Scoring Procedures 
Forty-two pre-service teachers’ lesson plans from the Spring 2010 and 46 plans from the 

Fall 2010 semester from the two courses were assessed with the HEAT instrument.  Student 
names were removed from the lesson plans, numbered, and randomly divided.  Next, blind 
scoring was conducted by the researchers and trained scorers using the HEAT instrument.  The 
scores were recorded in Excel spreadsheet files.  In total, six evaluators rated the lessons – three 
were the researchers, two were Assistant Superintendents from local school districts (both were 
familiar with the HEAT instrument and used the instrument in their school districts), and one 
was an Instructional Designer (who had taught the HEAT instrument in graduate courses) at the 
same institution.  The researchers trained the three other scorers on the use of the HEAT 
framework for the scoring of the lesson plans.  A main focus of the training was on calibration of 
the scoring of the evaluators.  To establish the calibration, the researchers chose four anchor 
lessons with agreed upon ratings, and trained and discussed these in detail for scoring calibration 
of the application of the instrument.  The new members of the scoring team each scored the 
“training” lesson plans, shared and discussed their ratings for each of the four HEAT 
components.  At this point in the study, the calibration goal was to score two consecutive lessons 
with HEAT component ratings no more than one level apart on each component from the score 
set by the researchers. After three training lessons, the calibration goal was met. 

 After the calibration was established, three teams of scorers were paired together with 
one researcher in each of the pairs.  The lesson plans were randomly distributed among the three 
scoring teams.  A scoring team evaluated the same set of lessons – giving every lesson in the 
study two sets of scores.  The ratings were recorded on spreadsheets.  The scores were averaged 
when the scorers did not agree upon a score, which will explain why the ratings chart includes 
some scores containing “.5”.  For example, scores of 2 and 3 would average to give a score of 
2.5 on the Tables 2-6 as seen in the Results section.   

The TWS data were collected in Fall 2010 and Spring 2011, the subsequent semesters 
after the pre-service teachers completed the methods classes and were student teaching. All 
student teachers in the elementary program submitted a TWS at the end of the semester as their 
capstone project. Using analytic scoring rubrics developed by the Renaissance Partnership 
institutions, faculty score each component and associated indicators of the TWS based on a scale 
of 1 = Beginning, 2 = Developing, and 3 = Proficient, and 4 = Exemplary. The education 
program faculty deems a score of two on the four-point scale to be considered mastering the 
component or associated state teacher standard.  For a Beginning rating, pre-service teachers 
would use minimal technology in their planning and instruction.  To be classified as Developing, 



 Technology-Infused Instructional Framework    11 
 

pre-service teachers would utilize some technology in planning and instruction.  A Proficient 
rating could be achieved if students demonstrate technology integration in planning and 
instruction with students utilizing technology for higher-level thinking tasks and in real world 
contexts.  An Exemplary rating would indicate students demonstrated skills beyond Proficient 
with minimal assistance and on the first attempt. Professors of the student teacher class assess 
pre-service teachers’ abilities to meet the technology requirements as well as the other TWS 
indicators. 
 
Data Analysis 
  Pre-service teachers that were taught to use the HEAT instrument in lesson planning were 
compared to other elementary education pre-service teachers in student teaching who took the 
same methods classes with other instructors, but were not trained to use the HEAT instrument in 
lesson planning. For research questions 1-4, frequencies and percentages were calculated for 
each component of HEAT.  
 A T-test was used to determine if there was a significant difference in the TWS mean 
scores of the HEAT group and the non-HEAT group for research question 5. 
 
Results and Discussion 
 Analysis of HEAT Levels of Pre-service Teacher Lesson Plans  
 In this section, the results are provided where the lesson plans were double scored by the 
scoring team.  When the double scores were not in agreement, the two scores were averaged 
together.  The range of scores could span 0-6.  With the possibility of the average between the 
two scorers, the increments increase by 0.5.  Tables 1-4 depict the results of the data analysis for 
each of the HEAT components to be discussed in detail below, while Table 5 depicts the scores 
on the HEAT framework as a holistic score. 
 
Research Question 1. What percentage of lesson plans prepared by pre-service teachers score at 
a level 3 or higher in Higher-Order Thinking activities?  
 
Table 1: Frequencies and Percentages for Higher-Order Thinking Using the HEAT Instrument  

Average 
Rating 

Spring 2010 Fall 2010 
N % N % 

0     
.5     
1.0 2 5 5 10.0 
1.5   1 2.1 
2.0 13 31 16 34.7 
2.5   20 43.4 
3.0 27 64 4 8.7 
3.5     
4.0     

 
Mean 2.6 2.1 
N-Size 42 46 
Ratings 
>=3 

27 64 4 8.7 



12	
    R. Stobaugh, M. Maxwell & J. Tassell	
  
 

 
For the Higher-Order Thinking component, the pre-service teachers’ assignment was to 

plan instructional tasks with “students learning at an Analyzing, Evaluating, or Creating levels of 
Bloom’s Taxonomy” (Bloom, 1956) with “Teacher-directed questioning and instruction” at 
higher level thinking. The undergraduate frequency scores for Spring 2010 and Fall 2010 ranged 
from 1.0 and 3.0 with a target of 3.0. In the Spring 2010 semester mean scores were 2.6 with a 
decline to 2.1 in the Fall of 2010.   Sixty-four percent of the students achieved a score of 3 in 
Spring 2010 compared to 8.7% in the Fall 2010 with a decrease of 55.3%.  
 
Research Question 2: What percentage of lesson plans prepared by pre-service teachers score at 
a level 3 or higher in Engaged Learning activities? 
 
Table 2: Frequencies and Percentages for Engaged Learning Using the HEAT Instrument  

Average 
Rating 

Spring 2010 Fall 2010 
N % N % 

0     
.5     
1.0 3 7 7 15.2 
1.5   5 10.8 
2.0 33 79 25 54.3 
2.5   7 15.2 
3.0 6 14 2 4.3 
3.5     
4.0     

 
Mean  2.1 1.9 
N-Size 42 46 
Ratings 
>=3 

6 14 2 4.3 

 
For the Engaged Learning component, the pre-service teacher frequency scores for 

Spring 2010 and Fall 2010 ranged from 1.0 and 3.0 with a target of 3.0 for the Engaged Learning 
component of HEAT. To achieve a 3, the lesson plan had to include: “Student choice for projects 
or to solve a problem posed by teacher”; “Students are engaged in projects based on preferred 
learning styles, interests or passions”; “Used multiple instructional strategies.” In Spring 2010 
the mean was 2.1 compared to the mean score of 1.9 for Fall 2010 semester.   Only 14% of the 
pre-service teachers (N=6) achieved a score of 3 in Spring 2010 compared to 4.3% (N=2) in the 
Fall 2010 showing a decrease of 9.7%.    
 
Research Question 3: What percentage of lesson plans prepared by pre-service teachers score at 
a level 3 or higher in Authentic Learning? 
 
 
 
 
 



 Technology-Infused Instructional Framework    13 
 

Table 3: Frequencies and Percentages for Authentic Learning Using the HEAT Instrument  
Average 
Rating 

Spring 2010 Fall 2010 
N % N % 

0 1 2   
.5     
1.0 12 29 4 8.7 
1.5   7 15.2 
2.0 24 57 29 63.0 
2.5   5 10.8 
3.0 5 12 1 2.1 
3.5     
4.0     

 
Mean  1.79 1.9 
N-Size 42 46 
Ratings 
>=3 

5 12 1 2.1 

 
The Authentic Learning component of HEAT measures if “Learning may be relevant to 

the real world or the past”; “Learning occurs in a simulated real-world situation such as a class 
store.” The pre-service frequency scores for Spring 2010 and Fall 2010 ranged from 1.0 and 3.0 
with a target of 3.0. In the Spring 2010 semester the 2.0 frequency was the highest rating with 
57.0% of the scores and 73.8% of the scores with a 2.0 or 2.5 rating.  There was a noticeable 
decrease in the number of 1 scores with 29% (N= 12) in Spring 2010 to 8.7% (N=4) in Fall 2010.   
There were 12% (N=5) of the pre-service students achieving a 3 rating in the Spring 2010 and 
2.1% (N=1) achieving a 3 rating in Fall 2010 showing a decrease 9.9%. 

 
Research Question 4: What percentage of lesson plans prepared by pre-service teachers score at 
a level 3 or higher in Technology activities?  
 
Table 4: Frequencies and Percentages for Technology Using the HEAT Instrument  

Average 
Rating 

Spring 2010 Fall 2010 
N % N % 

0 6 14 2 4.3 
.5   1 2.1 
1.0 20 48 11 23.9 
1.5   2 4.3 
2.0 7 17 14 30.4 
2.5   9 19.5 
3.0 9 21 7 15.2 
3.5     
4.0     
Mean  1.45 1.8 
N-Size 42 46 
Ratings 
>=3 

9 21 7 15.2 



14	
    R. Stobaugh, M. Maxwell & J. Tassell	
  
 

 For the Technology component, to obtain a level 3 score, the lesson plan needed to 
include: "Technology use appears to be an add-on or alternative—not essential for task 
completion”; “Technology is used for higher-order thinking tasks such as analysis and decision-
making." In Fall 2010 and Spring 2011, there was a large percentage of students that were below 
the target score of three.  However, the level 3 scores were higher than any other component 
(except the Spring 2010 Higher level thinking) with 21% (N=9) and 15.2% (N=7) in the fall 
2012. Only 15.2% achieved a 3 in Fall 2010 and 21.6% achieved a 3 or above in the Spring 2010 
semester with a 5.8% decrease.  
  
Table 5. Percentage Scoring a Three or Above  

Average  
Rating 

Undergraduate 
Spring 2010 Fall 2010 
% % 

H 64 8.7 
E 14 4.3 
A 12 2.1 
T 21 15.2 

 
 For each of the HEAT components there was a decrease in HEAT scores from Spring to 
Fall 2010.  There was a 55.3% decrease in the higher-level thinking component.  The lowest 
decrease was for technology component (5.8%). 
 
 Analysis of Teacher Work Sample Mean Scores of HEAT and non-HEAT Groups 
Research Question 5:  Is there a significant difference in the TWS scores of the pre-service 
teachers who learned to plan lessons with the HEAT instrument and pre-service teachers who did 
not learn to plan lessons with the HEAT instrument? 
 
Table 6.  Ratings on the Teacher Work Sample by Pre-Service Teachers from the HEAT Group 
and the non-HEAT Group 
 
Average 
Rating 

HEAT Non-HEAT HEAT Non-HEAT 

 Fall 2010 Spring 2011 
 N % N % N % N % 
Beginning 0 0 0 0 0 0 0 0 
Developing 7 33.3 32 26.0 2 9.5 9 12.2 
Proficient 9 42.9 67 54.5 7 33.3 54 72.9 
Exemplary 5 23.8 24 19.5 12 57.1 11 14.9 
TOTAL 21 100 123 100 21 99.9 74 100 
 
 
 Twenty-one elementary education students who were used the HEAT instrument in the 
Spring 2010 semester completed the TWS in the Fall 2010 semester.  Another 21 students who 
used the HEAT framework in the Fall 2012 semester completed the TWS in the Spring 2011 
semester.  For the control group of elementary education pre-service teachers who were not 
exposed to the HEAT instrument, 123 completed the TWS in Fall 2010 and 74 in Spring 2011.  



 Technology-Infused Instructional Framework    15 
 

In both semesters, all elementary education pre-service scores ranged between the Developing, 
Proficient, and Exemplary categories.  In Fall 2010, the number of pre-service teachers instructed 
with the HEAT Framework had 23.8% (N=5) score at an Exemplary rating compared to the 
19.5% (N=24) in the control group who had not been instructed with the HEAT Framework.  
However, in the Spring 2011, the HEAT group had 57.1% (N=12) compared to 14.9% (N=11) of 
the non-HEAT group scoring Exemplary.  That is a 42.2% difference between the HEAT and the 
control group’s percentage scoring Exemplary.  This group was the second group exposed to the 
HEAT Framework.   
Table 7.  Mean TWS Data from Fall 2010 and Spring 2011 
 

Group Fall 2010 Spring 2011 
HEAT 2.9048 3.4762 
Non-HEAT 2.9350 3.0270 

 
Table 8.  T-test Using Combined Data from Fall 2010 and Spring 2011 TWS 
 

Group N Mean SD 
Non-HEAT 197 2.9695 0.6220 
HEAT 42 3.1905 0.7726 

     T-value -2.00;  Probability 0.0468 
 
 
 The T-test was used to determine significant differences between the HEAT and non-
HEAT groups because this statistical test can account for different sample sizes.  The data from 
the two semesters were combined for this calculation for two reasons: the different sample sizes 
in the groups and to combine the effects of the HEAT instruction over two semesters.  The 
purpose was not to test the differences between the two semesters but to test the differences 
between the two groups over the 2010-2011 academic year. There was a significantly higher 
score for the HEAT group, t(239) = -2.00, p <0.05, on the Teacher Work Sample than the non-
HEAT group (see Table 8). This indicates that pre-service teachers who used the HEAT 
instrument for lesson planning performed better on the Teacher Work Sample than pre-service 
teachers who were not exposed to the HEAT instrument for lesson planning. 
 

Conclusions 
 The four components of the HEAT instrument provide a means for teachers to 
conceptualize how to improve the instructional quality of their lessons.  When teachers design 
lessons that “raise the HEAT,” instruction is engaging, cognitively complex, authentic, and 
embedded with technology.  The combination of the HEAT elements ensure that technology is 
integrated in a research-based method to increase the quality of lessons that meet 21st century 
expectations.  
 The Higher-Order Thinking component was the highest scoring component (64%) in the 
Spring 2010 semester with the Technology component as the second highest (21%). The payoff 
is reflected in the increased percentages of pre-service teachers who scored Exemplary (57.1%) 
on the TWS the next semester compared to 14.9% of the other pre-service teachers who were not 
exposed to the HEAT instrument. The HEAT implementation in methods courses significantly 
improved the pre-service teachers’ performance on the TWS. Therefore, we can conclude that 



16	
    R. Stobaugh, M. Maxwell & J. Tassell	
  
 

when pre-service teachers use the HEAT instrument to design higher quality lessons, it does 
impact their lesson design and performance; i.e., implementation or teaching the lesson. This 
increase in quality lessons and instruction for students can positively impact K-12 student 
learning. 
 

Implications for Future Research 
 A version of the HEAT instrument will continue to be used to score pre-service and 
advanced teachers’ lesson plans in our coursework. The authors will continue to refine the 
instrument and collaborate on how to use the HEAT framework in pre-service and advanced 
teacher courses.  If pre-service and advanced teachers are consistently designing instruction at a 
level 3 or higher on a framework such as this, the goal is to prove that these high quality lessons 
can positively impact K-12 student learning.  
 The Teacher Work Sample capstone project has provided this research study with an end 
product study opportunity that is quite opportune to capture and consider.  The HEAT instruction 
for the lesson plan application for the pre-service teachers takes place typically in the semester 
immediately before the TWS semester.  What would then be interesting for future study is to 
follow these same participants as they progress into the field as teachers and/or graduate 
students.  As teachers, it would be of utmost interest to continue the study as to whether or not 
HEAT framework lesson plan design is continues to impact their ability to design high-quality 
lessons.   
 However, a larger concern is whether or not the HEAT lesson is actually being 
implemented or taught at the level that the lesson written or designed.  In other words, oftentimes 
what is seen in the research during the lesson plan instruction semester, pre-service teachers have 
the opportunity to deliver or teach the lesson.  What is observed happens to be at a lower level of 
cognitive complexity in many cases than the lesson was designed.  When the lesson is not taught 
with appropriate rigor, students are not learning to their potential. 
 Another research concern is to examine how pre-service and practicing teachers are 
designing the assessments for their students to exhibit their learning in regard to their HEAT 
lesson.  What is initially found in the lesson plans tends to be of good quality.  However, when 
the lesson plan designer is asked to create a model student product and score it with their rubric, 
many issues surface regarding their depth of understanding of assessment. A misalignment of 
objectives, instruction, and assessment is observed when the pre-service teachers design this 
model student product and attempt to score it with their rubric for the lesson.  There appears to 
be a disconnect between the quality and expectations in the lesson plan in comparison to the 
product that the students are to complete to show their learning from the lesson.  Another angle 
to achieve this research is to require the pre-service teachers to submit a student assessment 
product that is directly from their HEAT Lesson Plan and develop a scoring plan.  The hope in 
the entire conversation regarding assessment is that pre-service teachers will better see the lesson 
as a complete cycle with the assessment included.   
 
  



 Technology-Infused Instructional Framework    17 
 

References 
 

Anderson, L.W., & Krathwohl, D.R. (2001). A taxonomy for teaching, learning and assessing:  

A revision of Bloom’s taxonomy of educational objectives. New York: Addison Wesley.  

Bloom, B. S. (1956). Taxonomy of educational objectives. Philadelphia: David McKay 

Publishing Company. 

Bloom, B.S., Englehart, M.D., Furst, E.J., Hill, W.H., & Krathwohl, D.R. (1956). Taxonomy of 

educational objectives:  The classification of educational goals. Handbook 1: Cognitive 

domain. New York: David McKay.  

Dener, P. R., Norman, A. D., Salzman, S. A., Pankratz, R., & Evans, S.E. (2004). The     

Renaissance Partnership Teacher Work Sample: Evidence supporting score 

generalizability, validity, and quality of student learning assessment. In E.M. Guyton, & 

J. R. Dangel (Eds.), Research linking teacher preparation and student performance, (pp. 

23-55). Dubuque, IA:  Kendall/Hunt Publishing Company. 

Driscoll, M. (2000). Psychology of learning for instruction. Needham Heights, MA: Allyn & 

Bacon. 

Herrington, J., & Oliver, R. (2000). An instructional design framework for authentic learning 

environments. Educational Technology Research & Development, 48(3), 23-48. 

International Society for Technology in Education (2007). The national educational technology 

standards for students. Eugene, OR: International Society for Technology in Education. 

Retrieved from http://www.iste.org/standards/nets-for-students.aspx  

International Society for Technology in Education (2008). The national educational technology 

standards for teachers. Eugene, OR: International Society for Technology in Education. 

Retrieved from http://www.iste.org/standards/nets-for-teachers.aspx  



18	
    R. Stobaugh, M. Maxwell & J. Tassell	
  
 

Jones, B., Valdez, G., Nowakowski, J., & Rasmussen, C. (1994). Designing learning and 

technology for educational reform. Oak Brook, IL: North Central Regional Educational 

Laboratory.  

Marzano, R. J. (2010). Teaching inference. Educational Leadership, 67(7), 80-81. 

Moersch, C. (2002). Measures of success: Six instruments to assess teachers’ use of technology. 

Learning & Leading with Technology, 30(3), 10-18. 

Partnership for 21st Century Skills (2009). P-12 Framework Definitions. Retrieved from 

http://www.p21.org/storage/documents/P21_Framework_Definitions.pdf 

The Renaissance Partnership for Improving Teacher Quality. (2010). Teacher Work Sample:    

performance prompt, teaching process standards, scoring rubrics. Retrieved from     

http://www.uni.edu/itq/RTWS/. 

Schalock, H. D., Schalock, M., & Girod, G. (1997). Teacher Work Sample methodology as used    

at Western Oregon State College. In J. McMillan (Ed.), Grading teachers, grading 

schools: Is student achievement a valid evaluation measure? (pp.15-45). Thousand Oaks, 

CA: Corwin Press. 

School of Teacher Education. (2011). Teacher Work Sample. Bowling Green, KY: Western     

Kentucky University. 

Siemens, G. (2004). Connectivism: A learning theory for the digital age. Retrieved from 

http://www.elearnspace.org/Articles/connectivism.htm 

Stobaugh, R.S., McDonald, M. L., & Tassell, J.L. (2010). Student teacher use of technology to 

     facilitate teaching and learning. In C. Maddux (Ed.), Research highlights in technology 

and teacher education 2010, (pp. 43-52). Chesapeake, VA: Society for Information 

Technology & Teacher Education (SITE). 


