Back to the basics: Developing a student engagement survey to evaluate the role of experiential learning on student engagement Page 204 - Developments in Business Simulation and Experiential Learning, volume 41, 2014 BACK TO THE BASICS: DEVELOPING A STUDENT ENGAGEMENT SURVEY TO EVALUATE THE ROLE OF EXPERIENTIAL LEARNING ON STUDENT ENGAGEMENT Gerald (Jerry) F. Burch Tarleton State University gburch@tarleton.edu Nathan A. Heller Tarleton State University heller@tarleton.edu Rusty Freed Tarleton State University freed@tarleton.edu ABSTRACT Experiential learning is built around the idea that experience leads to knowledge. After 40 years of research there are still many facets of experiential learning that are not completely understood. This research examines one overlooked area in this research by developing a student engagement survey that can be used to examine the effects of experiential learning on student engagement and the subsequent learning outcomes. Discussions of how to employ this survey in future research is discussed. INTRODUCTION The new AACSB guidelines (AACSB, 2013) clearly advocate the role of using experiential learning in business school curriculum to meet the learning needs of business students. In particular, AACSB Standard 13 (AACSB, 2013) says that “student academic and professional engagement occurs when students are actively involved in their educational experiences, in both academic and professional settings, and when they are able to connect these experiences in meaningful ways.” Embedded in this requirement are two very important goals. The first is a desire to actively involve the student in their educational experiences and the second is the movement from involvement to meaningful outcomes. One means of accomplishing this task would be for business schools to provide more experiential learning experiences/opportunities. Hoover and Whitehead (1975, p. 25) stated almost 40 years ago that “experiential learning exists when a personally responsible participant cognitively, affectively, and behaviorally processes knowledge, skills, and/or attitudes in a learning situation characterized by a high level of active involvement.” These comments still apply today for colleges and universities that are trying to engage their students in “meaningful ways” (AACSB, 2013). Our initial goal was to return to the basic principles of student learning and to investigate when experiential learning exists, including an understanding of how experiential learning may help students have meaningful outcomes. Based on other research, we hypothesized that experiential learning leads to student engagement and student engagement leads to student learning outcomes. A road block that we encountered on our journey was that there is no psychometrically proven student engagement survey to provide the data needed to test our hypotheses. Therefore, the purpose of this study is to develop a student engagement survey that will allow us to evaluate the relationship between experiential learning and meaningful student outcomes. In the next section we discuss the theoretical background for our hypothesis. In the following sections we review the research that has been conducted on engagement and review the currently available surveys and design a new student engagement survey grounded in engagement and learning theory. The final section is this paper is used to offer recommendations for other scholars on how this survey could be used to further research experiential learning, its antecedents, and outcomes. BACKGROUND Much research has been conducted on experiential learning and Experiential Learning Theory (ELT) (Kolb, mailto:gburch@tarleton.edu mailto:heller@tarleton.edu mailto:freed@tarleton.edu Page 205 - Developments in Business Simulation and Experiential Learning, volume 41, 2014 1984) in the lapse of time that has occurred between Hoover and Whitehead’s (1975) comments and the publication of the new AACSB standards (AACSB, 2013). However, a literature review showed that few articles address the underlying antecedents and associated outcomes of experiential learning. This fact is magnified by the analysis of the new AACSB recommendation which implies a causal order: active involvement precedes student engagement and leads to meaningful outcomes. We will take a deeper look at this comment and the implications. The original definition of experiential learning (Hoover, 1974) stated that experiential learning was learning that occurred through personal involvement where the whole person, in both his feeling and cognitive aspects, were included in the learning event. Experiential Learning Theory defines experiential learning as “the process whereby knowledge is created through the transformation of experience. Knowledge results from the combination of grasping and transforming experience” (Kolb, 1984, p. 41). Although these definitions vary slightly, we propose that scholars are in agreement that experiential learning is an active form of learning that requires student engagement. To investigate the role of engagement in learning we examined Standard 13 (AACSB, 2013) which states that “students give the appropriate attention and dedication to the learning materials and maintain their engagement with these materials even when challenged by difficult learning activities.” In the engagement step the student is willing to invest personal, internal energies regardless of task difficulty. Kahn (1990) claimed that this investment of resources results in physical, cognitive, and emotional dimensions of engagement that produce active, full performance as demonstrated by student attendance, student performance, and student products. These words mirror those of Hoover and Whitehead (1975, p. 25) where experiential learning is the result of cognitive, affective, and behavioral engagement. Although, Hoover and Whitehead (1975) did not use the word engagement, they did evaluate forms of engagement in their measure of student satisfaction. Some of the survey questions from their 18-item scale are: I felt active and “involved,” I felt the courses required me to exercise independent judgment in evaluating textbook theories, and I felt the course challenged me. Using this article as a launching point we began a literature review for student engagement surveys that could be potentially used for experiential learning research. We found three surveys that have been recently used to measure student engagement. The surveys and their questions are listed below. The most commonly used survey for student engagement is the National Survey of Student Engagement (NSSE). This survey was funded by the Pew Charitable Trusts and was developed as a new approach to gathering information about collegiate quality. The National Center for Higher Education Management Systems (NCHEMS) coordinated the development of the NSSE which consists primarily of items that are known to be related to important college outcomes. The purpose of this survey is reported to be two-fold (Gonyea & Kuh, 2009). The first is to determine the amount of time and energy that students put into their education and related activities. The second is to determine how institutions use their resources to encourage students to engage in activities that have shown to increase the student’s learning experience. In addressing the first purpose, the NSSE evaluates the student’s activities and performance. In particular, there are some NSSE questions that have been linked to student deep learning which is comprised of three subscales: higher order learning, reflective learning, and integrative learning (Nelson Laird, Shoup, & Kuh, 2005). The higher order learning subscale is comprised of four items that measure the student’s participation in university activities that required them to evaluate, synthesize, analyze, or apply information. These higher order learning ideas are based on Bloom’s (1954) taxonomy of educational objectives and represent the highest levels of learning. An obvious omission from Bloom’s (taxonomy) is the absence of the highest level of learning which is to create. No items on the scale address whether the student has created something. This omission is significant since one of the three major AACSB requirements is innovation (AACSB, 2013), which is the highest level of thinking (Bloom, 1954). It is also important to note that the items on the NSSE were not created as a student engagement survey based on engagement theory. As such, it is necessary to extend the work of Nelson Laird et al. (2005) to determine which of the NSSE items are supported by engagement theory. Another difficulty in using the NSSE to evaluate student engagement at the class or course level is that the NSSE was developed to compare universities to one another and therefore measures the overall level of student engagement at the college or university level. The NSSE questions would have to be modified to allow for evaluation across classes or across disciplines. Sample NSSE survey questions are: During the current school year, about how often have you done the following? 1. Asked questions or contributed to course discussions in other ways. 2. Prepared two or more drafts of a paper or assignment before turning it in. 3. Come to class without completing readings or assignments. 4. Attended an art exhibit, play or other arts performance (dance, music, etc.). Page 206 - Developments in Business Simulation and Experiential Learning, volume 41, 2014 During the current school year, how much has your coursework emphasized the following? 1. Memorizing course material. 2. Applying facts, theories, or methods to practical problems or new situations. 3. Analyzing an idea, experience, or line of reasoning in depth by examining its parts. 4. Evaluating a point of view, decision, or information sources. The second student engagement survey is from Hu & Wolniak (2013) who developed an Academic Engagement survey (Likert scale “strongly agree” to “strongly disagree”) with only four statements. This survey could be used to determine the level of performance at the class level, but again does not address the dimensions of student engagement that lead to the performance. The statements are: 1. Work with other students on school work outside of class. 2. Discuss ideas from readings and class with students outside of class. 3. Discuss ideas from readings and class with faculty outside of class. 4. Work harder than thought to meet an instructor’s expectation. The third survey is the Ultrecht Work Engagement Survey for Students (Schaefeli & Bakker, 2004). This survey is an adaptation from the Ultrecht employee engagement survey that has been used in many organizational behavior studies. It is separated into three dimensions, with nine total statements that focus on the dimensions of student engagement. Vigor 1. I feel strong and vigorous when I’m studying or going to class. 2. I feel fit and vigorous when I’m studying or I’m in class. 3. When I get up in the morning I feel like going to class. Dedication 1. I find my studies full of meaning and purpose. 2. My study inspires me. 3. I am proud of my studies. Absorption: 1. Time flies when I am studying. 2. When I am studying I forget everything else around me. 3. I get carried away when I am studying. 4. Rich, LePine, and Crawford (2010) modified the Ultrecht survey at the employee level to better align with the definition of engagement by Kahn (1990). This definition included the concepts of physical, cognitive, and emotional engagement that produce active, full performance. The 18 statements associated with this new scale are: Physical Engagement 1. I work with intensity on my job 2. I exert my full effort to my job 3. I devote a lot of energy to my job 4. I try my hardest to perform well on my job 5. I strive as hard as I can to complete my job 6. I exert a lot of energy on my job Emotional Engagement 1. I am enthusiastic in my job 2. I feel energetic at my job 3. I am interested in my job 4. I am proud of my job 5. I feel positive about my job 6. I am excited about my job Cognitive engagement 1. At work, my mind is focused on my job 2. At work, I pay a lot of attention to my job 3. At work, I focus, a great deal of attention on my job 4. At work, I am absorbed by my job 5. At work, I concentrate on my job 6. At work, I devote a lot of attention to my job The Rich et al. (2010) survey questions address the student engagement dimensions that lead to increased performance. However, there are five concerns for using the Rich et al. (2010) survey. (1) It needs to be modified to evaluate student engagement. (2) It does not address the student’s engagement in class versus out of class which could be important if students have a much better chance of being engaged in class based on the “entertainment factor.” One alternative is to alter the six cognitive questions from Rich et al. (2010) to address either in class or out of class (e.g. When I am in this class, my mind is focused on the subject. When I am studying for this class, my mind is focused on this subject.). (3) It does not address engagement differences between classes/courses. (4) It does not address the AACSB Standard 13 requirement to evaluate if the student is “able to connect these (academic and professional) experiences in meaningful ways.” (5) It does not address the AACSB Standard 13 requirement to determine if “students give the appropriate attention and dedication to the learning materials and maintain their engagement with these materials even when challenged by difficult learning activities”. Page 207 - Developments in Business Simulation and Experiential Learning, volume 41, 2014 Based on our findings, we propose that there are currently no student engagement surveys that can properly address the dimensions of student engagement that are necessary for experiential learning. In the next section we will discuss the development of a new student engagement survey that contains content validity, criterion-related validity, convergent and discriminant validity. SCALE DEVELOPMENT The goal of this research is to develop a scale that can measure the dimensions of student engagement and also the learning outcome measures associated with that engagement. The review of current studies show that the NSSE has many questions that address the outcome measures, but not many that address the student’s engagement. Conversely, the more frequently used organizational behavior measures of employee engagement (Rich et al., 2010) focus on the engagement dimensions and do not measure the engagement outcomes. Combining the questions from these scales would address both concerns, but may cause some overlap. This discussion shows the need for the development of a new student engagement scale. We started the construction of our scale by first looking at the content validity and dimensions of student engagement. CONTENT VALIDITY AND DIMENSIONS OF STUDENT ENGAGEMENT Careful examination of the student engagement literature showed that there are two major aspects that should be included in any student engagement measure: (1) the relationship that learning activities, whether active or passive, play on student engagement and (2) the subsequent learning outcomes associated with those activities. We will look at each of these dimensions to identify content that needs to be included in our survey. Learning activities – these events should lead to student engagement. However, it is beyond the scope of this engagement survey to include questions that directly assess the quantity, quality, or type of activities that are included in the class. The learning activity will therefore act as a distinct construct. However, our survey must provide a level of question clarity to ensure that the student understands that it is the activity that is occurring in the class/course that is being evaluated. Learning outcomes – range from memorizing facts to creative performance. The learning outcomes that we are most concerned with are those deep learning outcomes which are separated into higher order learning, reflective learning, and integrative learning. Higher order learning is seen as that learning that causes students to move up the Bloom’s (1954) taxonomy of learning to levels of analysis, evaluation, or creativity. Reflective learning is seen where students must take new knowledge and compare it to what they already know. The final general higher order learning is integrative learning where the student combines ideas across classes or domains. Student engagement – there are three sub-dimensions to this construct: physical, emotional, and cognitive (Kahn, 1990). Physical engagement can be seen through the physical effort that is exerted on the task. Examples of this type of engagement include questions from Rich et al. (2010) “I work with intensity” or “I exert my full effort.” Emotional engagement is the affective part of engagement. Examples that have been used in other engagement surveys are “I am enthusiastic about…” or “I feel energetic…” The final dimension is that of cognitive engagement. This is probably the most understood dimension of engagement. Cognitive engagement includes questions like “my mind is focused” or “I pay a lot of attention.” However, one problem with addressing cognitive engagement is that it has two sub-dimensions. These are the cognitive engagement that occurs in class and subsequently out of class. It is quite possible that student cognitive engagement in class is driven by the entertainment level provided by the professor. However, we believe that the cognitive engagement that occurs outside of class may be just as, or more, important than the in-class cognitive engagement. CONVERGENT AND DISCRIMINANT VALIDITY Convergent and discriminant validity are measures that relate to the extent that similar constructs are related (convergent) and dissimilar constructs are not related (discriminant). In this section we discuss each of the constructs previously listed and what should and should not be related. Learning activities – classroom activities can range from passive to highly active. We adopt the ideas from Gentry (1990) where basic lecture, seminar discussion, and library research papers are at the low end of the experiential learning spectrum. Those learning activities with a moderate level of experiential learning would be problem solving, laboratory and experiential exercises, case discussions, study group discussions, and individual case reports. Activities with the highest level of experiential learning would be group case assignments, simulation games, descriptive field activities/projects, and consultative field projects. Learning outcomes – we conceptualize learning outcomes as either concrete or general. For our survey we will focus on the general learning outcomes and not include concrete learning outcomes. General learning outcomes are actions that are triggered by student engagement. Examples are trying harder in class, coming to class on time, discussing ideas outside of class, etc. General learning can be further separated into as deep learning outcomes which are separated into higher order learning, reflective learning, and integrative learning. Concrete learning outcomes, on the other hand, are those learning skills or objectives that are acquired during the learning process. An example is the improvement of Page 208 - Developments in Business Simulation and Experiential Learning, volume 41, 2014 TABLE 1 FACTOR ANALYSIS AND ITEM ANALYSIS RESULTS (STUDY 1) Item Factor 1 Cognitive in class Factor 2 Physical Factor 3 Emotional Factor 4 Deep Learning IL/RL Factor 5 Deep Learning HO Factor 6 Difficult Tasks Factor 7 Cognitive out of class Factor 8 Focus I work with intensity on assignments for this class/course .332 .587 .163 .040 .170 .159 .047 -.056 I exert my full effort towards this class/course .343 .799 .231 .027 .030 .037 .147 .026 I devote a lot of energy towards this class/course .312 .796 .050 .064 .093 .056 .136 .145 I try my hardest to perform well for this class/course .215 .817 .013 .127 .054 .095 .110 .043 I strive as hard as I can to complete assignments for this class/course. .287 .726 -.025 .105 .113 .224 .192 -.001 I exert a lot of energy for this class/ course. .350 .661 .104 .160 .075 .156 .092 .192 I am enthusiastic about this class/ course. .086 .051 .055 .139 .031 -.055 .385 .014 I feel energetic when I am in this class/course. .553 .350 .446 .037 -.129 .065 .243 -.114 I am interested in the material I learn in this class/course. .417 .231 .189 .051 .227 -.004 .268 .101 I am proud of assignments I com- plete in this class/course. .366 .575 .269 -.017 .121 .070 .031 .034 I feel positive about the assignments I complete in this class/course. .363 .555 .307 .003 .196 -.011 .031 -.061 I am excited about coming to this class/course. .670 .229 .401 .158 -.053 .115 .176 -.136 When I am in the classroom for this class/course, my mind is focused on class discussion and activities .603 .300 .209 -.006 .098 .076 .097 .087 When I am in the classroom for this class/course, I pay a lot of attention to class discussion and activities .821 .263 .163 .079 .166 .076 -.019 .054 When I am in the classroom for this class/course, I focus a great deal of attention on class discussion and activities .807 .302 .183 .096 .108 .045 .018 .226 When I am in the classroom for this class/course, I am absorbed by class discussion and activities .762 .289 .316 .141 .049 .024 .108 .061 When I am in the classroom for this class/course, I concentrate on class discussion and activities .809 .294 .167 .076 .139 .088 .123 .096 When I am in the classroom for this class/course, I devote a lot of atten- tion to class discussion and activities .741 .300 .242 .086 .079 -.022 .152 .165 When I am reading or studying material related to this class/course, my mind is focused on class discus- sion and activities .550 .347 .183 .237 .055 -.076 .466 .146 When I am reading or studying material related to this class/course, I pay a lot of attention to class discus- sion and activities .527 .462 .157 .252 .057 -.031 .407 .169 When I am reading or studying material related to for this class/ course, I focus, a great deal of atten- tion on class discussion and activi- ties .562 .385 .204 .195 .064 .030 .403 .196 When I am reading or studying material related to this class/course, I am absorbed by class discussion and activities .539 .330 .250 .228 .140 -.019 .491 .123 Page 209 - Developments in Business Simulation and Experiential Learning, volume 41, 2014 When I am reading or studying material related to this class/course, I concentrate on class discussion and activities .538 .401 .144 .223 .151 .022 .429 .227 When I am reading or studying material related to this class/course, I devote a lot of attention to class discussion and activities .571 .358 .174 .189 .035 .053 .420 .221 I use information and skills that I have learned from this class/course when I work with other students on school work outside of class. .478 .053 .172 .417 .235 .168 .266 .080 I discuss ideas from the readings and class/course discussion with friends and family outside of class. .332 -.094 .033 .133 .215 .346 .474 .080 I discuss ideas from the readings and course discussion with faculty and other professionals outside of class. .265 .102 .477 .058 .116 .339 .312 -.119 I work harder than I thought to meet the instructor’s expectation in this class/course. -.048 .478 -.099 .097 .198 .468 -.092 .231 During difficult assignments I work harder to produce good products in this class/course. .370 .431 -.043 .361 .231 .386 -.136 .086 During difficult assignment I seek outside help to complete the assign- ments in this class/course. .162 .291 .036 .147 .009 .653 -.091 .247 I often ask questions or contribute to class/course discussions. .313 .281 .381 .057 -.105 .339 .049 .049 I have prepared two or more drafts of a paper or assignment before turning it in for this class/course. .070 .165 .382 .093 -.167 .360 .293 .181 I often come to class without com- pleting readings or assignments for this class/course. -.072 -.082 .114 .513 -.266 .181 .097 -.298 I often ask other students to help me understand course material for this class/course. -.274 .068 .154 .297 -.157 .612 .239 .020 I often explain course material to one or more students for this class/ course. .180 .113 .184 .206 .197 .676 -.084 .152 I often prepare for exams/ assignments for this class by discuss- ing or working through course mate- rial with other students. -.020 .065 .363 .069 .155 .689 .140 .137 I combine ideas from different courses when completing assign- ments for this class/course. .190 .068 .275 .504 .259 .301 -.055 -.010 I connect learning in this class/ course to societal problems or issues. .164 .146 -.001 .564 .249 .114 .145 .168 I include diverse perspectives (political, religious, racial/ethnic, gender, etc.) in course discussion or assignments for this class/course. -.031 .098 .324 .544 .089 .213 .284 .143 I examine the strengths and weak- nesses of my own views on a topic or issue in this class/course. .287 .152 .224 .608 .135 .326 .015 .120 I try to better understand someone else’s views by imagining how an issue looks from his or her perspec- tive in this class/course. .251 .157 .221 .620 -.010 .043 .157 .244 I discussed class/course topics, ideas, or concepts with faculty member outside of class. .016 .069 .537 .072 .033 .427 .261 -.062 I memorized course material for assignments and tests in this class/ course. .097 .004 -.013 .160 .713 .293 .151 .069 Page 210 - Developments in Business Simulation and Experiential Learning, volume 41, 2014 I applied facts, theories, or methods to solve practical problems or new situations in this class/course. .112 .134 .233 .192 .774 .072 .123 .041 I analyzed an idea, experience, or line of reasoning in depth by exam- ining its parts in this class/course. .209 .261 .306 .136 .711 -.010 .061 .026 I evaluated a point of view, decision, or information sources in this class/ course. .025 .267 .391 .304 .574 .000 .056 .122 I formed a new idea or understand- ing from various pieces of infor- mation in this class/course. .178 .152 .238 .423 .535 -.164 .175 .083 I reached conclusions based on my own analysis of numerical infor- mation (numbers, graphs, statistics, etc.) in this class/course. .236 -.017 .727 .149 .316 .061 -.138 .097 I used numerical information to examine a real-world problem or issue (unemployment, climate change, public health, etc.) in this class/course. .270 .002 .583 .305 .128 .181 .011 .028 I evaluated what others have con- cluded from numerical information in this course/class. .287 .091 .622 .159 .192 .145 .075 .030 I identified key information from reading assignments in this class/ course. .418 .277 -.081 .094 .400 .125 .347 .169 I reviewed my notes after class in this class/course. .478 .214 .035 .029 .304 .167 .489 .078 I summarized what I learned in class or from course materials in this class/course. .212 .200 .168 -.064 .278 .217 .546 .178 This class/course helped me write clearly and effectively. .031 .157 .700 .197 .110 .055 .294 .336 This class/course helped me speak clearly and effectively. .130 .138 .730 .221 .109 -.043 .194 .339 This class/course helped me think critically and analytically. .559 .183 .201 .300 .216 .071 .160 .270 This class/course helped me analyze numerical and statistical information. .439 .209 .630 .036 .098 .074 -.044 .095 This class/course helped me acquire job or work-related knowledge and skills. .347 .099 .663 .026 .071 .045 .069 .308 This class/course helped me work effectively with others. .314 .080 .680 .100 .102 .174 .051 .241 This class/course helped me develop or clarify a personal code of values and ethics. .164 -.005 .196 -.035 .118 .266 .099 .641 This class/course helped me under- stand people of other backgrounds (economic, racial/ethnic, political, religious, nationality, etc.) .087 .114 .167 .134 -.029 .209 .166 .793 This class/course helped me study complex real-world problems. .394 .125 .262 .394 .119 .025 .037 .505 This course helped me be an in- formed and active citizen. .220 .095 .369 .195 .145 .117 .069 .700 Eigenvalue 23.738 4.606 3.084 2.553 2.156 1.944 1.682 1.640 Percentage of variance explained 37.680 7.311 4.895 4.053 3.423 3.085 2.670 2.602 Cumulative percentage of variance explained 37.680 44.991 49.886 53.939 57.362 60.447 63.117 65.719 Page 211 - Developments in Business Simulation and Experiential Learning, volume 41, 2014 communication, leadership, teamwork, decision making, and planning skills that were demonstrated by using experiential learning techniques with MBA students (Hoover, Giambatista, Sorenson, & Bommer, 2010). These concrete learning outcomes are instrumental in evaluating the effects of student engagement, but should be seen as a separate construct and therefore not included in this student engagement survey. One contributor to higher order learning that is not included in this survey is general mental ability. It is predicted that general mental ability will be significantly related to learning outcomes but will not be related to student engagement. CRITERION-RELATED VALIDITY There has been no previous work containing the criterion-related validity of student engagement surveys. This makes it impossible to compare our new scale to other scales to evaluate the concurrent validity portion of criterion-related validity. We propose that the development of our survey will be able to address the predictive validity part of criterion-related validity based on the theoretical link between active learning and student engagement and student engagement and learning outcomes. STUDY 1: STUDENT ENGAGEMENT SURVEY DEVELOPMENT AND INITIAL VALIDATION Survey development We modified the questions from Rich et al. (2010) to address the physical, emotional, and cognitive (in class and not in class). This resulted in 24 questions (6 questions for each area). To address the outcome measures associated with student engagement we modified statements from the NSSE and from Hu & Wolniak (2013), and included two new statements that specifically addressed the AACSB requirement (AACSB, 2013) for difficult assignments. There are 42 questions associated with outcome measures. The result is 66 total survey items for our initial survey. Sample Our first sample was of 214 undergraduate students at a southern United States medium sized university. The survey was administered during class and no incentives were provided. The average age of the respondents was 21.7 (σ = 3.2) and 53% of the sample were female. Results We perform exploratory factor analysis to determine the factor structure of the 63 item scale to help develop subscales of each of the engagement dimensions. Using the results from the Scree Plot we determined that an eight factor structure appeared to be sufficient. Table 1 shows that the eigenvalues ranged from 1.64 to 23.74 with 65.7 percent of the variance explained by the eight factors. The factor explaining the most variance was the cognitive (in class) items that were derived from the Rich et al. (2010) work engagement scale. The second most influential factor was the physical in class engagement also derived from Rich et al. (2010). The emotional engagement items from Rich et al. (2010) were identified as the third most important factor and the cognitive engagement (out of class) represented the seventh factor. The four remaining factors were comprised of the NSSE deep learning integrative learning and reflective learning, the NSSE deep learning higher order learning items, the items directly linked to the AACSB difficult task items, and the NSSE outcome measure items that identified the NSSE items that addressed student outcomes associated with being a better developed global citizen. We used the results from the exploratory factor analysis to reduce the engagement scale items. The three highest loaded items for each factor were chosen to represent that factor. The only exception was the deep learning integrative learning and reflective learning where two items that represented integrative learning and two items representing reflective learning were selected. The result was a 25 item scale designed to measure the four components of student engagement (student-in class, TABLE 2 DESCRIPTIVE STATISTICS: CORRELATIONS OF ENGAGEMENT SCALE FACTORS (STUDY 2) Factor M SD α Physical Emotional Cognitive In class Cognitive Out of class Persistence Deep Learning IL/ RL Deep Learning Higher Physical 3.98 .91 .89 - Emotional 3.73 1.12 .88 .71** - Cognitive – In class 3.82 1.12 .95 .63** .74** - Cognitive –Out of class 3.70 1.06 .93 .57** .60** .71** - Persistence 3.19 1.02 .75 .42** .38** .27** .28** - Deep Learning – IL/RL 3.49 .89 .66 .50** .51** .43** .48** .45** - Deep Learning – Higher Order 3.71 .90 .80 .62** .58** .59** .57** .32** .63** - Global Perspective 3.12 1.04 .69 .39** .41** .42** .32** .41** .59** .52** N = 179 *p .05 **p ..01 Page 212 - Developments in Business Simulation and Experiential Learning, volume 41, 2014 student-out of class, physical, and emotional) and four outcome components (deep learning integrative/reflective learning, deep learning higher order learning, persistence with difficult tasks, and development as global citizens). STUDY 2: FACTOR STRUCTURE CONFIRMATION AND CONSTRUCT VALIDITY Sample We administered the revised survey to 169 undergraduate students at a southern United States medium sized university. The average age of the respondents was 21.3 (σ = 3.1) and 56% of the sample were female. Results Table 2 shows the Cronbach Alpha reliabilities, descriptive statistics, and the correlations between factors. The Cronbach Alpha reliabilities range from .66 for deep learning integrative and reflective learning to .95 for cognitive in class engagement. The highest reliabilities were associated with the four engagement factors. There are moderate to strong correlations between factors, but confirmatory factor analysis showed that an eight factor model was preferred to a one factor model and a 2 factor model (engagement and outcomes) (Anderson & Gerbing, 1988). The resulting scale showed sufficient reliability across the eight factors and demonstrated the validity of an eight factor scale. STUDY 3: CRITERION-RELATED VALIDITY EVIDENCE Sample We administered the revised survey to 196 undergraduate students at a southern United States medium sized university. The students were separated into two groups. The first group was composed of students in four sections of an undergraduate business statistics course. The statistics course is built around in class exercises, group case studies, and two group presentations based on case studies. The total number of students in group 1 was 85. The second group was one large section of undergraduate psychology class where the primary means of delivery is lecture. The second group had 111 students. The average age of the respondents was 21.0 (σ = 3.6) for group 1 and 20.2 (σ = 1.7) for group two. Results Due to the lack of previous student engagement surveys it is impossible to demonstrate criterion-related validity of this new scale by comparing to a previous “gold standard” (Devillis, 2003). Table 3 shows that there is a significant difference between the engagement across groups one and two and also the outcome measures. This difference was predicted due to the number of exercises that allow for increased engagement in group one versus the passive approach used for group 2. DISCUSSION The process used in developing this 25-item student engagement survey ensures the adequate psychometric properties of the dimensions of student engagement and also the outcomes can be appropriated measured. We performed three separate studies to take advantage of the TABLE 3 DESCRIPTIVE STATISTICS: ANOVA OF ENGAGEMENT SCALE FACTORS (STUDY 3) Group 1 Group 2 Factor M SD M SD F Sig. Physical 4.34 .74 3.64 .90 57.58 .00** Emotional 4.46 .69 3.01 .99 231.30 .00** Cognitive – In class 4.48 .74 3.20 1.03 160.84 .00** Cognitive – Out of class 4.10 .92 3.31 1.03 51.51 .00** Persistence 3.49 .98 2.91 .97 28.57 .00** Deep Learning – IL/RL 3.76 .78 3.24 .91 31.21 .00** Deep Learning – Higher Order 3.95 .75 3.50 .96 21.58 .00** Global Perspective 3.32 1.02 2.96 .98 10.06 .00** N = 196, Group 1 N = 85, Group 2 N = 111 *p .05 **p ..01 Page 213 - Developments in Business Simulation and Experiential Learning, volume 41, 2014 previous research that had been conducted in the area of work engagement, student engagement, and tied them to the desired outcomes listed by the AACSB Business Standards (AACSB, 2013) and NSSE measures. The exploratory factor analysis showed that the strongest research items for the four engagement factors were closely tied to those used in work engagement (Rich et al., 2010). No engagement questions from the NSSE, or other scales loaded as strongly and were removed from the final scale. Three of the four outcome dimensions came directly from items currently being used by the NSSE. Previous research (Nelson Laird et al., 2005) showed that a subset of the NSSE items loaded onto 3 factors of deep learning. This study confirmed that one component (higher order learning) loaded as a single factor but that integrative learning and reflective learning loaded together as a single component. The third component borrowed from the NSSE was the global citizenship factor which is composed of questions that determine if the class led the student to consider their place in a global environment. The final factor identified was directly connected to the AACSB (2013) requirement to determine if the level of persistence that the student was willing to perform. CONCLUSION The growing need to measure student engagement has made it necessary to develop a student engagement survey that can be used across all disciplines. The 25-item scale developed in this study was developed for the sole purpose of measuring the multiple dimensions of student engagement and also the outcomes desired by AACSB and other stakeholders. Universities are under considerable pressure to increase the readiness of today’s graduates. As the cost of education continues to rise these expectations will certainly continue to grow. One means of accomplishing this task is to develop, and employ, instruction methods that increase student engagement. This research fills the current gap and provides a means for instructors to measure the level of student engagement at the class/course level. This information will allow individual professors to implement methods and evaluate results as well as providing departments, colleges, and universities to measure the overall engagement of all of their students to evaluate how changes in curricula are affecting engagement and hopefully, learning. REFERENCES AACSB (2013). Eligibility procedures and accreditation standards for business accreditation. Apr 8, 2013. Tampa, FL. Anderson, J.C., & Gerbing, D.W. (1988). Sructural equation modeling in practice: A review and recommended two-step approach. Psychological Bulletin, 103, 411-423. Bloom, B.S. (1954). Taxonomy of educational objectives. Handbook 1: Cognitive domain. Longman: New York. Devillis, R.F. (2003). Scale Development: Theory and applications. Sage Publications: Thousand Oaks. Gentry, J.W. (1990). What is experiential learning? Guide to Business Gaming and Experiential Learning. London: Nichols/GP Publishing Hoover, J.D. (1974). Experiential learning: Conceptualization and definition. 1974 ABSEL Proceedings, 31-35. Hoover, J.D., Giambatista, R.C., Sorenson, R.L., & Bommer, W.H. (2010). Assessing the effectiveness of whole person learning pedagogy in skill acquisition. Academy of Management Learning & Education, 192- 203. Hoover, J.D., & Whitehead, C.J. (1975) An experiential- cognitive methodology in the first course in management: some preliminary results. Simulation Games and Experiential Learning in Action, Volume 2. . (Reprinted from Bernie Keys Library (11th ed.)) Hu, S., & Wolniak, G.C. (2013). College student engagement and early career earnings: Differences by gender, race/ethnicity, and academic preparation. The Review of Higher Education. 36, 211-233. Kahn, W.A. (1990). Psychological conditions of personal engagement and disengagement at work. Academy of Management Journal, 33, 692-724. Kolb, D.A. (1984). Experiential learning: Experience as the source of learning and development. Englewood Cliffs, NJ: Prentice Hall. Nelson Laird, T.F., Shoup, R., & Kuh, G.D. (2005). Measuring deep approaches to learning using the National Survey of Student Engagement. Paper presented at the Annual Meeting of the Association for Institutional Research. Chicago, IL. Rich, B.L., LePine, J.A., & Crawford, E.R. (2010) Job engagement: Antecedents and effects on job performance. Academy of Management Journal. 53, 617-635. Schaefeli, W., & Bakker, A. (2004). Ultrecht Work Engagement Scale: Preliminary manual. Ultrecht University.